10 posts tagged with "Evaluation"

Answer checks now catch an answer that drops its source's condition, and are much faster on long answers. Text from PDFs and web pages is read the way it was typed, and a 1,000-sentence answer is checked in under half a second.

npm install @devilsdev/rag-pipeline-utils@^3.9.0
# and, if you use the MCP server:
npx @devilsdev/rag-mcp-server@3.9.0 ./docs

If you install @devilsdev/rag-core, rag-guardrails, rag-connectors or rag-eval directly, upgrade them to 3.9.0 as well.

v3.7.0 adds Weaviate. It also makes Pinecone work on the indexes its console creates for you, with no embedder needed. The answer check now reads units, fractions and times, and was measured against the Claude benchmarks before release. Nothing breaks for existing code.

npm install @devilsdev/rag-pipeline-utils@^3.7.0

v3.6.0 widens the way in. npm create rag-pipeline gives you a working app that answers from your documents and checks every answer, with Next.js, Mastra or Cloudflare Workers. Pinecone joins pgvector and Qdrant. The answer check raises fewer false alarms, measured against the Claude benchmarks, including where it got worse. And if you deploy to Cloudflare Workers, this release fixes the main entry there. Nothing breaks for existing code.

npm create rag-pipeline@latest my-app -- --template next

v3.5.0 is about trusting the checks. The guards take a model of your choice for what patterns miss, and the patterns' accuracy is now published, misses included. Evaluation scores show up in OpenTelemetry and Langfuse, on the spans they judge. And when Claude or OpenAI file search cites its sources, verifyCitations() checks that each cited passage actually says what the sentence claims. Nothing breaks for existing code.

npm install @devilsdev/rag-pipeline-utils@^3.5.0

v3.4.0 closes the biggest gap in the answer check, paraphrases, without giving up what makes it trustworthy, and puts your documents one command away from Claude. Nothing breaks for existing code: without the new option, every check returns the same result as before.

npm install @devilsdev/rag-pipeline-utils@^3.4.0

v3.3.0 makes the grounding check catch answers that contradict their source, and brings the checks to the tools you already use: middleware for the Vercel AI SDK, matchers for Jest and Vitest, and scorers and a guardrails processor for Mastra. Nothing breaks for existing code, but some answers that used to pass the grounding check now fail it, on purpose.

npm install @devilsdev/rag-pipeline-utils@^3.3.0

v3.0.2 updates csv-parse to 7.x for a security advisory and adds a recorded demo of rag-pipeline eval to the README. There are no API changes.

npm install @devilsdev/rag-pipeline-utils@^3.0.2

v2.7 is about knowing when your RAG pipeline gets worse before your users do. You can draft a test set from your own documents, score it from code or the command line, compare two configurations, and fail a pull request when answers get less grounded, with a new GitHub Action. It also adds retrievers for PostgreSQL with pgvector and for Qdrant, a generated API reference with corrected TypeScript types, and published, reproducible benchmarks.

v2.6 makes the library easier to adopt alongside what you already run and cheaper to run in production. checkAnswer grounds answers from LangChain, LlamaIndex, Mastra or the Vercel AI SDK. A new /testing entry runs pipelines offline in tests, and opt-in caching skips the model calls for repeated questions. The CLI's query command can now show how it got its answer, and both the package and the CLI load several times faster. The current version is 2.6.2: 2.6.1 published the updated README to npm, and 2.6.2 fixes npm install on machines without husky (see the 2.6.2 post).

v2.5.0 is a consolidation release. The CLI now does real work end to end, every pipeline stage can be cancelled, grounding metrics measure what they claim to, and modules that were simulations are now clearly separated in an /experimental entry point. Several of these changes alter scores and defaults, so please read the upgrade notes before you update.