RAG Pipeline Utils v2.7: Evaluation in CI, pgvector and Qdrant, and Published Benchmarks

v2.7 is about knowing when your RAG pipeline gets worse before your users do. You can draft a test set from your own documents, score it from code or the command line, compare two configurations, and fail a pull request when answers get less grounded, with a new GitHub Action. It also adds retrievers for PostgreSQL with pgvector and for Qdrant, a generated API reference with corrected TypeScript types, and published, reproducible benchmarks.

npm install @devilsdev/rag-pipeline-utils@2.7.0

There are no runtime breaking changes from 2.6. Two things are worth knowing when you upgrade; see Upgrading.

A test set for your pipeline

Evaluation starts with questions you know the answers to. A test set is a JSON array of cases:

[
  {
    "id": "vacation",
    "prompt": "How many vacation days do I get?",
    "expected": "Full-time employees receive twenty days of paid vacation.",
    "sourceIds": ["handbook.md"]
  }
]

expected is the reference answer, and sourceIds names the documents retrieval should find. Writing the first few dozen cases is the slowest part, so generateSyntheticDataset(documents, { llm }) drafts them: an LLM writes question and answer pairs about one document at a time, and each case records the document it came from. Answers their own document doesn't support are dropped, along with repeated questions and malformed replies. Every generated case is tagged synthetic and marked reviewed: false, because a model's questions are a starting point, not ground truth: read them, fix them, and add the ones your users actually ask.

validateDataset(dataset) checks a test set before anything runs and reports every problem with its case and field, without throwing.

Score it, and compare two configurations

runEvaluation(dataset, pipeline) runs every case with citations and evaluation on and returns a plain-JSON report you can save:

import { runEvaluation } from "@devilsdev/rag-pipeline-utils";
import { createTestPipeline } from "@devilsdev/rag-pipeline-utils/testing";

const { pipeline } = await createTestPipeline();
const report = await runEvaluation(
  [
    {
      id: "vacation",
      prompt: "How many vacation days do I get?",
      expected: "Full-time employees receive twenty days of paid vacation.",
      sourceIds: ["vacation"],
    },
  ],
  pipeline,
);

console.log(report.summary.failed); // 0
console.log(report.summary.metrics.faithfulness.mean); // 1
console.log(report.cases[0].scores.sourceRecall); // 1

Each case gets faithfulness, relevance, context precision and recall, groundedness, citation coverage, and sourceRecall: the share of its sourceIds that were retrieved. A failed case is recorded, not thrown.

compareConfigurations(dataset, { baseline, candidate }) runs both sides and lists what got worse. Either side can be a pipeline or a saved report, so in CI you compare each pull request with the report from your main branch without running main's pipeline again:

import { compareConfigurations } from "@devilsdev/rag-pipeline-utils";
import {
  createTestPipeline,
  EchoLLM,
} from "@devilsdev/rag-pipeline-utils/testing";

const dataset = [
  { id: "vacation", prompt: "How many vacation days do I get?" },
];
const { pipeline: baseline } = await createTestPipeline();
const { pipeline: candidate } = await createTestPipeline({
  llm: new EchoLLM({ answer: "The office is in Paris." }),
});

const comparison = await compareConfigurations(dataset, {
  baseline,
  candidate,
});

console.log(comparison.regressed); // true
console.log(comparison.metrics.faithfulness.delta); // -1
console.log(comparison.regressions[0].id); // "vacation"

Reports record the library version that made them, and a comparison warns when the two differ, since scoring can change between releases. renderEvaluationReport turns a run into Markdown for a pull request comment or a standalone HTML page. See Evaluation.

The same from the command line

npx rag-pipeline generate-dataset --output tests.json --count 30
npx rag-pipeline eval tests.json --output eval-main.json
npx rag-pipeline eval tests.json --baseline eval-main.json \
  --min faithfulness=0.8 --min sourceRecall=0.9 --report eval-summary.md

eval uses the pipeline and index from your .ragrc.json, prints the scores and exits 1 when a case errors, a metric is below its --min, or the run regressed from the --baseline. A --report ending in .html writes a standalone page; any other name writes Markdown. See the CLI reference.

Fail a pull request when answers get worse

DevilsDev/rag-eval-action, now on GitHub Marketplace, runs rag-pipeline eval on every pull request. It compares the run with the report from your default branch, kept in the Actions cache, posts the summary as a comment that later runs update in place, and fails the check when scores drop:

- uses: DevilsDev/rag-eval-action@v1
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  with:
    ingest: docs
    dataset: eval/tests.json
    baseline-cache: true
    min: faithfulness=0.8

Its README covers the inputs, the permissions it needs and pull requests from forks.

pgvector and Qdrant

Two retrievers store and search your documents in a database you run:

import pg from "pg";
import {
  PgVectorRetriever,
  QdrantRetriever,
} from "@devilsdev/rag-pipeline-utils";

const postgres = new PgVectorRetriever({
  pool: new pg.Pool({ connectionString: process.env.DATABASE_URL }),
  table: "rag_documents",
});

const qdrant = new QdrantRetriever({
  url: process.env.QDRANT_URL,
  apiKey: process.env.QDRANT_API_KEY,
  collection: "rag_documents",
});

• PgVectorRetriever creates the extension, table and HNSW index on first use, stores each batch in one transaction, and keeps values in query parameters. pg is an optional dependency, loaded only when you use it. efSearch trades query speed for recall (see the benchmarks below).
• QdrantRetriever talks to Qdrant's REST API with Node's built-in fetch, so it needs no client package. Document ids are mapped to stable UUIDs, and results come back with your own ids.

Both follow the same retriever contract as the built-in stores, and the same contract tests run against real pgvector and Qdrant servers in CI. In .ragrc.json they are the pgvector and qdrant retrievers, reading their connection details from DATABASE_URL, QDRANT_URL and QDRANT_API_KEY, never from the file. Neither is cached unless you give it a cacheKey, because other processes can change the data. See vector stores.

Benchmarks you can reproduce

The Benchmarks page now publishes numbers. Every table is generated from a committed report, the benchmark scripts are in the repository, and CI fails if the page and the reports disagree.

Retrieval quality on BEIR SciFact (5,183 abstracts, 300 claims), with the retrievers that run offline:

System	nDCG@10	Recall@100
BM25Search	0.665	0.883
HybridRetriever (BM25 + TF-IDF)	0.564	0.887
LocalEmbedder (TF-IDF)	0.470	0.846

BM25Search matches the 0.665 the BEIR paper reports for BM25 on SciFact, which also confirms the metrics are computed the standard way. Fusing BM25 with the weaker TF-IDF retriever at equal weights scores below BM25 alone; hybrid search pays off when the vector side is a strong embedder.

Vector store speed, 10,000 vectors of 384 dimensions, measured in CI with pgvector 0.8.7 and Qdrant 1.20:

Store	Query p50	Recall@10
MemoryRetriever	9.9 ms	1.000
PgVectorRetriever, efSearch 40	1.9 ms	0.556
PgVectorRetriever, efSearch 200	3.6 ms	0.882
QdrantRetriever	5.3 ms	1.000

The pipeline itself adds about 4 µs per run() around its components. The vectors are synthetic, so measure recall on your own embeddings before tuning efSearch.

Building the benchmarks found three bugs, all fixed before release:

• QdrantRetriever was about 40 ms slow on every request: Node's fetch asks for gzip, and Qdrant's compressed responses stalled each one. It now asks for uncompressed responses, and queries take 1–5 ms.
• PgVectorRetriever returned at most 40 results, because an HNSW scan returns no more than hnsw.ef_search rows. A larger topK now raises it for that query.
• LocalEmbedder is described under Fixes.

A generated API reference, and types that match the code

There is now a page for every export, with its methods, options and types, generated from the published TypeScript declarations. Generating it showed that the hand-written declarations for the core API didn't match the code. A plugin written to them failed at runtime: the declared rerank had its arguments reversed, and an LLM that returned the declared LLMResponse object produced the answer "[object Object]". The declarations now come from the source wherever possible. The plugin contracts are corrected: retrieve takes one { query, queryVector, topK, signal } object, rerank takes (query, documents), and generate takes (query, documents, options) and returns the answer text. The build now compiles plugins typed by these contracts and runs them through a real pipeline, so they can't drift again.

Fixes

• LocalEmbedder ignored most words on a large corpus. With more distinct words than its 10,000-token vocabulary, it kept the rarest ones, mostly words found in a single document, so most queries embedded to all zeros. It now keeps the most common words; on SciFact its nDCG@10 went from 0.12 to 0.47. This is the CLI's default offline embedder.
• The CLI didn't wait for retrievers whose methods are async, as a remote store's are. Re-ingesting a file could keep its old chunks or delete the new ones.
• logger.info(), .warn(), .error() and .debug() now work. The logger export is the logger module, so these were undefined; logger.logger.info() still works.
• A bad option value such as --top-k 0 printed a stack trace; the CLI now prints a one-line error.

Upgrading

• TypeScript: code that compiled against the old, wrong declarations may not compile now, for example rerank(results, query) or await loadConfig({ configPath }). That code didn't work when run, so the compiler is now catching a real bug.
• LocalEmbedder indexes keep the vocabulary they were built with. Run rag-pipeline ingest again to get the fix.

What's next

3.0 is the next release. It removes the deprecated experimental classes from the main entry (they stay available from /experimental), drops Node 18, and moves to the next major version of the openai SDK. A Pinecone retriever follows pgvector and Qdrant.

The full list of changes is in the CHANGELOG.