RAG Pipeline Utils v3.1: Contract Tests for Plugins and Context Packing

v3.1.0 adds two things for people building on the library: contract tests that tell a plugin author whether their plugin does what the pipeline expects, and a token budget for the documents sent to the LLM. Nothing changes for existing code.

npm install @devilsdev/rag-pipeline-utils@^3.1.0

Contract tests for your plugins

A retriever, embedder, reranker or LLM plugin can have the right method names and still break a pipeline: a generate() that returns an object instead of the answer text, a retriever that ignores topK or doesn't sort by score, vectors whose length depends on the text. validatePlugin only checks the shape. runContractTests checks the behaviour:

// my-retriever.test.js
const { runContractTests } = require("@devilsdev/rag-pipeline-utils/testing");
const { MyRetriever } = require("./my-retriever");

runContractTests("retriever", () => new MyRetriever());

That registers one test per guarantee createRagPipeline relies on, read from what the pipeline actually calls. A retriever, for example, must return an array when empty and, after store(), return the closest documents first, sorted by a numeric score, at most topK of them, with content and metadata intact. Every plugin type also runs one query through a real pipeline, with stand-ins for the other stages.

The checks use node:assert, so they run in Jest, Mocha and Vitest with globals as written, and in node:test or Vitest without globals when you pass { describe, it }. To run them without a test framework, checkContract returns a report:

import { checkContract } from "@devilsdev/rag-pipeline-utils/testing";

const broken = await checkContract("llm", () => ({
  async generate() {
    return { text: "Twenty days." }; // an object, not the answer text
  },
}));
console.log(broken.results.find((r) => !r.passed).error);
// "generate() resolved to object; the pipeline needs the answer text"

Retrievers that read an existing index and have no store() take a setup(retriever, documents) option to load the test documents. The Testing guide lists every check.

Fit retrieved context into a token budget

Retrieving ten passages and sending them all to the model is the usual way a RAG app overflows its context window or its bill. maxContextTokens keeps the documents the LLM sees within a budget:

const result = await pipeline.run({
  query: "How many vacation days do employees get?",
  options: { topK: 10, maxContextTokens: 2000 },
});

console.log(result.packing); // { tokens, maxTokens: 2000, kept, dropped, truncated }

It runs after guardrail filtering and reranking, and keeps the best-ranked documents in order while they fit. A document that doesn't fit is skipped, and a shorter one after it may still fit. The top document is always included: when it alone is too long it is cut at a word boundary, so a small budget never leaves the model with nothing. result.results is then the list the model saw, so citations and evaluation score the right documents.

The default count is the library's estimate of about four characters per token; pass your model's tokenizer as countTokens for exact numbers. Cut documents are copies, so the cache keeps the full text, and two budgets never share a cached answer. The same logic is exported as packContext(documents, { maxTokens }) for use outside a run. See packContext.

Upgrading

There are no breaking changes. The full list is in the CHANGELOG.