Skip to main content

RAG Pipeline Utils v3.3: Grounding That Catches Negation, AI SDK Middleware, Test Matchers and Mastra

· 4 min read
Ali Kahwaji
Ali Kahwaji
Creator & Maintainer of RAG Pipeline Utils

v3.3.0 makes the grounding check catch answers that contradict their source, and brings the checks to the tools you already use: middleware for the Vercel AI SDK, matchers for Jest and Vitest, and scorers and a guardrails processor for Mastra. Nothing breaks for existing code, but some answers that used to pass the grounding check now fail it, on purpose.

npm install @devilsdev/rag-pipeline-utils@^3.3.0

Grounding catches negation​

The grounding check compares each sentence of an answer with the sources: its content words and its numbers must be there. Until now, a negation slipped through, because "not" or "never" is one word among many:

import { CitationTracker } from "@devilsdev/rag-pipeline-utils";

const sources = [
  {
    id: "policy",
    content:
      "Full-time employees receive twenty days of paid vacation per year.",
  },
];

new CitationTracker().track(
  "Full-time employees never receive paid vacation.",
  sources,
).groundednessScore; // 0 in 3.3.0; it was 1 in 3.2.0

Each sentence is now compared with the source sentence closest to it, and a sentence that is negated where that one isn't (or the reverse) is not grounded. The closest sentence matters: in a source that says employees get vacation and contractors don't, "contractors receive paid vacation" is judged against the contractor sentence. checkAnswer, the evaluation metrics, rag-pipeline eval and the guardrails all use the check, and checkPolarityConsistency exposes it.

It looks for negation words, so a source that negates in other words ("contractors are excluded") no longer supports a negated answer. That errs on the side of not grounding, and it means some answers score lower after upgrading. Paraphrases are still a limit of word-based checks.

Ground an AI SDK app with middleware​

If you already call generateText or streamText, groundingMiddleware adds retrieval, answer checks and guardrails without changing those calls:

import { generateText, wrapLanguageModel } from "ai";
import { openai } from "@ai-sdk/openai";
import { groundingMiddleware } from "@devilsdev/rag-pipeline-utils";

const model = wrapLanguageModel({
  model: openai("gpt-4o-mini"),
  middleware: groundingMiddleware({ retriever, embedder }),
});

const { text, sources, providerMetadata } = await generateText({
  model,
  prompt: "How many vacation days do I get?",
});
providerMetadata.ragPipeline.grounded; // false if a sentence is unsupported

Each call retrieves passages for the last user message, gives them to the model as untrusted reference data, checks the answer, and returns the passages as the AI SDK's own sources. Streams stay live; the report comes with the last part. Access control, injection detection and PII redaction are options, and per-request settings such as the user's permissions go in providerOptions.ragPipeline. It works with AI SDK 6 and 7 and doesn't make the AI SDK a dependency. See the middleware docs.

Test answers like any other value​

ragMatchers adds three matchers to Jest and Vitest:

import {
  createTestPipeline,
  ragMatchers,
} from "@devilsdev/rag-pipeline-utils/testing";

expect.extend(ragMatchers);

const { pipeline } = await createTestPipeline();
const result = await pipeline.run({
  query: "How many vacation days do I get?",
});

expect(result).toBeGroundedIn(); // every sentence supported by result.results
expect(result).toCite("vacation");
expect(result).toContainNoPII();

They also take answers from other frameworks: expect(answer).toBeGroundedIn(context) reads LangChain documents and LlamaIndex nodes. A failure lists the sentences that aren't supported. See Matchers for answers.

Mastra scorers and guardrails​

createMastraScorer turns groundedness, faithfulness, context precision or context recall into a Mastra scorer that makes no model calls, so it can run on every response for free. By default it scores against what the agent's tools returned. createMastraGuardrails is a processor that stops a run on prompt injection and redacts PII from the answer:

import { createScorer } from "@mastra/core/evals";
import {
  createMastraGuardrails,
  createMastraScorer,
} from "@devilsdev/rag-pipeline-utils";

const groundedness = createMastraScorer({
  createScorer,
  metric: "groundedness",
});
const guardrails = createMastraGuardrails();

const agent = new Agent({
  // ...
  scorers: { groundedness: { scorer: groundedness } },
  inputProcessors: [guardrails],
  outputProcessors: [guardrails],
});

See Mastra.

Coming from LlamaIndex.TS​

LlamaIndex.TS is deprecated and its repository archived. The comparison now covers LangChain.js, Mastra and the AI SDK, and says how to use the last two together with this library. A new guide, Coming from LlamaIndex.TS, maps documents, indexes, query engines, rerankers and evaluators, starting with a step that needs no rewrite: checkAnswer reads the sourceNodes your query engine already returns.

Also in 3.3.0​

  • Issue templates for bug reports, feature requests and plugin submissions, and a Community plugins section for plugins people publish.
  • Development dependencies updated: 88 of 99 Dependabot alerts on the repository are closed. None affected the published package, which still installs with no known vulnerabilities.

Upgrading​

There are no API changes. If you gate CI on grounding scores, expect answers that negate their sources to fail where they passed before; that is the fix. The full list is in the CHANGELOG.