RAG Pipeline Utils v2.6: Caching, Offline Testing and Grounding for Any Framework

v2.6 makes the library easier to adopt alongside what you already run and cheaper to run in production. checkAnswer grounds answers from LangChain, LlamaIndex, Mastra or the Vercel AI SDK. A new /testing entry runs pipelines offline in tests, and opt-in caching skips the model calls for repeated questions. The CLI's query command can now show how it got its answer, and both the package and the CLI load several times faster. The current version is 2.6.2: 2.6.1 published the updated README to npm, and 2.6.2 fixes npm install on machines without husky (see the 2.6.2 post).

npm install @devilsdev/rag-pipeline-utils@2.6.2

There are no breaking changes from 2.5.

Check answers from any framework

You don't have to rebuild your pipeline to get citations and grounding checks. checkAnswer takes the answer your chain produced and the sources it was given, in the shape your framework returns them: LangChain Documents, LlamaIndex nodes, strings or { id, content } objects.

import { checkAnswer } from "@devilsdev/rag-pipeline-utils";

const check = await checkAnswer({
  question: "How many vacation days do I get?",
  answer: "Full-time employees receive thirty days of paid vacation per year.",
  sources: [
    {
      pageContent:
        "Full-time employees receive twenty days of paid vacation per year.",
      metadata: { source: "handbook.md" },
    },
  ],
});

console.log(check.grounded); // false
console.log(check.unsupported.length); // 1

The result has grounded, groundedness (the share of supported sentences), the unsupported sentences, per-sentence citations and evaluation scores. The number check from 2.5 applies, so "thirty" against a source that says "twenty" is flagged.

Vercel AI SDK models in a pipeline

createAiSdkAdapter turns AI SDK models into an embedder and an llm for createRagPipeline(), so any AI SDK provider works with the pipeline, its citations, evaluation and guardrails:

import { embed, embedMany, generateText, streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { createAiSdkAdapter } from "@devilsdev/rag-pipeline-utils";

const { embedder, llm } = createAiSdkAdapter({
  ai: { embed, embedMany, generateText, streamText },
  embeddingModel: openai.embedding("text-embedding-3-small"),
  languageModel: openai("gpt-4o-mini"),
  settings: { temperature: 0 },
});

You pass in the AI SDK functions, so the AI SDK is not a dependency, and versions 5, 6 and 7 all work from ESM or CommonJS. Streaming, embedMany batching and cancellation go through to the AI SDK. The adapter is tested against AI SDK 6 and 7 with their mock models. See Use with other frameworks.

OpenAI-compatible providers

OpenAIConnector takes a baseURL for Groq, Together, the Hugging Face router, vLLM, LM Studio or Ollama's /v1 endpoint:

import { OpenAIConnector } from "@devilsdev/rag-pipeline-utils";

const groq = new OpenAIConnector({
  baseURL: "https://api.groq.com/openai/v1",
  apiKey: process.env.GROQ_API_KEY,
  model: "llama-3.1-8b-instant",
});

With a custom baseURL the connector never falls back to OPENAI_API_KEY, so your OpenAI key is never sent to another provider.

Tests that need no network

@devilsdev/rag-pipeline-utils/testing has deterministic stand-ins for the parts of a pipeline that call a model:

• HashEmbedder hashes words into vectors, so texts that share words rank as similar;
• EchoLLM answers with the top passage, or an answer you set, and records every call;
• fixtures.documents provides three short policy documents;
• createTestPipeline() wires them together.

import { createTestPipeline } from "@devilsdev/rag-pipeline-utils/testing";

const { pipeline, llm } = await createTestPipeline();
const result = await pipeline.run({
  query: "How many vacation days do I get?",
});

console.log(result.results[0].id); // "vacation"
console.log(llm.calls.length); // 1

Same input, same output, no API keys. The entry shares its classes with the main package, and importing the main package doesn't load it. Plugin authors also get validatePlugin(stage, plugin), which returns { valid, errors, warnings } instead of throwing, so plugin tests can assert on it. See Testing your RAG code.

Caching

Pass cache: true and the pipeline caches query embeddings, retrieval results and answers in memory:

import { createRagPipeline } from "@devilsdev/rag-pipeline-utils";
import {
  createTestPipeline,
  EchoLLM,
} from "@devilsdev/rag-pipeline-utils/testing";

const { embedder, retriever } = await createTestPipeline();
const llm = new EchoLLM();
const pipeline = createRagPipeline({ embedder, retriever, llm, cache: true });

await pipeline.run({ query: "How many vacation days do I get?" });
const again = await pipeline.run({ query: "How many vacation days do I get?" });

console.log(again.cache.generation); // "hit"
console.log(llm.calls.length); // 1

With simulated latencies (an 80 ms embedding call and a 600 ms LLM call), the first run took 704 ms and repeats took under 1 ms. Use RedisCache (node-redis or ioredis) to share the cache between processes, set a time to live per layer, or plug in any store with get and set.

A cache that serves a stale or wrong answer is worse than none, so the cache is strict:

• Retrieval is keyed by the index's state and the query vector, so storing or removing documents means a miss. Answers are keyed by the exact documents the model received and the generation options.
• Components must identify themselves. All built-in connectors, retrievers and the AI SDK adapter do, and their identity includes the model, endpoint, temperature and system prompt. A custom component without a cacheKey is not cached, and a warning says so once.
• Answers are cached only at temperature 0. The built-in connectors default to 0.7, so set temperature: 0 to cache answers.
• Access control still runs on every request. Retrieval is cached before filterResults and the reranker, so guardrail ACL filtering applies to each user. Users with different permissions never share an answer.
• Nothing is stored from a failed, aborted or timed-out run, or from a stream that didn't finish. A Redis outage doesn't fail the run; the pipeline warns once and carries on without the cache.

See Caching.

Semantic similarity

computeSemanticSimilarity(answer, reference, { embedder }) is the cosine similarity of two embeddings, from 0 to 1. Unlike BLEU and ROUGE, it doesn't penalise a correct answer that uses different words. checkAnswer reports it when you pass a referenceAnswer and an embedder.

Embeddings capture meaning, not facts: "twenty days" and "thirty days" come out very similar. Use it alongside the grounding checks, not instead of them.

A query command that shows its work

2.5.0 removed CLI options that printed output without doing anything. In 2.6 they come back, and this time they do real work:

npx rag-pipeline query "How many vacation days do I get?" --explain --trace

Option	What it shows
--explain	Each passage's rank and score, and which passages support each sentence; unsupported sentences are flagged
--trace	Time spent in embed, retrieve, rerank and generate
--stats	Latency, passage count, estimated tokens and, for priced models, estimated cost
--streaming	The answer as the model generates it
--timeout <ms>	Aborts the query after that many milliseconds
--quiet	Only the answer

With --json, the trace, stats and explanation are included in the output. Token counts are estimates (about four characters per token), not the provider's count.

Faster to load

Most of the package's dependencies now load only when the feature that needs them is first used:

Measured on Node 22	2.5.0	2.6
require('@devilsdev/rag-pipeline-utils')	about 515 ms, 54 packages	about 125 ms, 9 packages
Loading the CLI	1,281 modules, 93 packages	55 modules, 13 packages
rag-pipeline query	about 3.6 s	about 0.5 s

axios, jsonwebtoken, ajv and chokidar load when you first use the Anthropic, Cohere or Ollama connector, JWTValidator, plugin registration or config validation, or hot reload. The HTML, Markdown and CSV loaders load jsdom, marked and csv-parse when they first read a file.

Fixes

• ExecutionTracer.exportTraces('json') threw "circular structure" with the default captureOutputs: true. Traces now store JSON snapshots, so later changes to a result no longer alter its recorded trace either.
• ParallelEmbedder failed with the built-in connectors, because it always passed an array to embed(). A new mode option ('auto', 'batch' or 'single') controls this. In the default 'auto' mode it embeds one text at a time for those connectors, and uses embedBatch(texts) when an embedder has it.

Try it in your browser

The three examples run on StackBlitz in their offline modes, with no install and no API keys: basic pipeline, OpenAI with Pinecone and the Fastify service. If StackBlitz stays on "Cloning repo from GitHub" for more than a minute, reload the page.

What's next

2.7 focuses on evaluation in CI:

• a rag-eval GitHub Action that fails a pull request when groundedness, faithfulness or citation coverage drops;

• dataset validation and A/B comparison of pipeline configurations;

• synthetic test set generation from a corpus.

  3.0 will drop Node 18 and remove the deprecated experimental exports from the main entry. They stay available from /experimental.

The full list of changes is in the CHANGELOG.