Promptfoo

Paket

Local LLM evaluations and red-team tests with an official MCP server

Fiyat: Free CLI; model calls billed separately
Erişim: No account; model credentials as needed

Hakkında

CLI and library for comparing prompts, models and RAG systems with datasets and assertions, plus adversarial testing of AI applications. Evaluations run locally or in CI; model calls can incur provider charges. Its official MCP server exposes evaluation and red-team tools to agents.

Neler yapabilirsin

Compare prompts and model outputs against datasets and assertions
Probe an AI application with adversarial red-team test cases
Let an agent validate configurations and run evaluations through local MCP

Başlarken

Install Node.js and configure prompts, providers and test cases
Provide credentials for model providers used in the evaluation
Run evaluations or expose tools with the local MCP command

Örnek kod

npx promptfoo@latest validate
# Requires a configured evaluation project and any provider credentials
npx promptfoo@latest eval

Ayrıntılar

Barındırma: Kendi bilgisayarında, Kendi sunucunda
Kullanılabildiği yerler: Tüm dünya
Resmi SDK'lar: JavaScript/TypeScript
MCP sunucusu: Yerel sunucu