What we still need to verify : 2 points in this profile are not yet confirmed against vendor documentation.
- Boundary between the open source CLI and the commercial offering: confirm
- Current red team plugin and strategy catalog: verify against vendor docs
Treat these points as unconfirmed. They are open items in the catalog's verification queue, and this note stays until each is checked against the vendor's documentation.
What it does
Promptfoo runs declarative test suites against prompts, models and whole LLM applications. You write a config listing providers (a hosted endpoint, a local model, or a hook into your own application), the prompts under test, test cases with variables, and assertions over the output. Assertions span deterministic checks such as regex, JSON schema validity and latency, embedding similarity against a reference answer, and model graded rubrics where a judge model scores the response. Results render as a matrix comparing providers and prompt versions on the same cases, and the run exits non zero in CI.
The security half is the red team mode. Instead of hand written cases, Promptfoo generates adversarial inputs from a plugin catalog covering prompt injection, jailbreaks, system prompt and PII leakage, excessive agency and unsafe tool invocation, authorization flaws in agents that act on a user's behalf, and harmful content. Strategies then wrap those payloads: encoding tricks, multi turn escalation, roleplay framing, and iterative search that mutates a failed probe until it lands. Findings are grouped by vulnerability class with the failing conversation retained. Execution is local by default, so prompts and responses stay in your environment unless you share a report.
Where it fits
Developer laptop first, pipeline second. The config lives in the repository next to the application, so a prompt or model change can be gated by the same suite under pull request review. Full red team sweeps are slower and noisier, and usually belong on a schedule or a pre release gate rather than every commit. You need a callable target and an honest description of the application's purpose, otherwise generated attacks test a system that does not exist.
Strengths
- Test suites are diffable files in the repo, which makes safety behavior reviewable rather than tribal knowledge.
- Deterministic and model graded assertions coexist, so a judge model is not required for what a schema check can decide.
- Splitting attack payloads from evasion strategies produces useful coverage: one injection class, many wrappers.
- Local execution matters for teams that cannot send production prompts to a third party.
Limitations
- Model graded assertions inherit the judge model's error rate, and red team findings need human triage.
- Adversarial sweeps consume many inference calls and real wall clock time, and coverage is probabilistic, so a clean report is evidence of effort rather than proof.
- A testing harness, not an inline control. It finds bad behavior, it does not block it at runtime.
Who it suits
Engineering teams that already treat prompts and agent configuration as code and want safety regressions caught in CI. Less suited to a security team wanting a managed assessment with no repository access, or to anyone needing a runtime control.
Used Promptfoo? Recommend it under your own name and title.
Recommend this tool