The shortlist
Tracing, evaluation and adversarial testing solve different jobs. Choose tools that expose the evidence you need, then define what failure looks like for your actual application.
Documentation-based starting points, not hands-on benchmarks. Check each option's fit, setup and costs below. Compare them with the same worksheet.
Freemium
LangSmith combines application traces, datasets and evaluations. Cloud, hybrid and self-hosted entitlements differ; inspect trace retention, model calls and usage billing.
Capability source ↗— You want an end-user chatbot or assume all usage and self-hosting are included in the Developer allowance.
12k stars · Free · Python
Phoenix provides tracing, datasets and evaluator workflows. Its current server terms are ELv2, while infrastructure and evaluator/model calls carry separate costs and data routes.
Capability source ↗— You need an OSI open-source licence; Phoenix uses the Elastic License 2.0, which restricts offering it as a managed service.
Costs not checked
Promptfoo tests configured prompts and application targets. A locally running scanner can still use remote attack or target models; review authorization, model costs and report contents.
Capability source ↗— You need a guarantee that an application is secure or cannot authorize scans against the target.
Start with representative successes, failures and adversarial cases. Use explicit assertions and human review for consequential outcomes. Observing a trace is not blocking an action, and a passing test set is not a security certification. Red-team only targets you are authorized to test.
Check the linked product sources, supported languages, regional availability, permissions and current costs for your actual setup.
Google Analytics helps us understand visits. It stays off unless you allow it. Your choice is remembered for up to 180 days. Privacy policy
