Promptfoo

Package

Local LLM evaluations and red-team tests with an official MCP server

Price: Free CLI; model calls billed separately
Access: No account; model credentials as needed

About

CLI and library for comparing prompts, models and RAG systems with datasets and assertions, plus adversarial testing of AI applications. Evaluations run locally or in CI; model calls can incur provider charges. Its official MCP server exposes evaluation and red-team tools to agents.

What you can do with it

Compare prompts and model outputs against datasets and assertions
Probe an AI application with adversarial red-team test cases
Let an agent validate configurations and run evaluations through local MCP

Get started

Install Node.js and configure prompts, providers and test cases
Provide credentials for model providers used in the evaluation
Run evaluations or expose tools with the local MCP command

Example

npx promptfoo@latest validate
# Requires a configured evaluation project and any provider credentials
npx promptfoo@latest eval

Details

Hosting: Runs locally, Self-hosted
Available in: Worldwide
Official SDKs: JavaScript/TypeScript
MCP server: Local