RAG Pipeline Utils v3.8: Testing RAG Answers in TypeScript

v3.8.0 is about testing RAG in TypeScript. The answer matchers now compile with types, and a new guide walks through the whole setup with Vitest, from a unit test to a test set that fails CI when answers get worse. Nothing breaks for existing code.

npm install @devilsdev/rag-pipeline-utils@^3.8.0

The matchers, with types

@devilsdev/rag-pipeline-utils/testing has had three matchers for a while: toBeGroundedIn (every sentence of the answer is supported by its sources, numbers and negation included), toCite and toContainNoPII. They worked in JavaScript. In TypeScript, expect(result).toBeGroundedIn() didn't compile, because nothing told TypeScript the matchers existed. Worse, the declaration our own Testing page suggested failed under Vitest 5 with "All declarations of 'Assertion' must have identical type parameters".

3.8.0 exports their signatures as RagMatchers. Add them to Vitest's Matchers once, in a declaration file:

import "vitest";
import type { RagMatchers } from "@devilsdev/rag-pipeline-utils/testing";

declare module "vitest" {
  interface Matchers<R, T> extends RagMatchers<R> {}
}

Now the matchers type-check like any other assertion. A misspelled matcher is an error, and so is a wrong argument:

expect(answer).toCite(123)
Argument of type 'number' is not assignable to parameter of type 'string | string[]'.

Jest works the same way, with jest.Matchers<R>; the Testing page has both.

A guide: Testing RAG in TypeScript

RAG code rarely fails loudly: someone changes the chunk size, the prompt or the model, everything still runs, and an answer quietly says thirty days where the document says twenty. The new guide sets up a project that catches that, in three layers:

1. Unit tests with the typed matchers, on an offline test pipeline: deterministic, no API key.
2. Answer checks with checkAnswer, which return the exact sentence the sources don't support, for answers from any framework.
3. A test set scored by rag-pipeline eval, with minimums or a baseline from your main branch, failing the build when scores drop.

test("a wrong number is caught", async () => {
  const sources = [
    {
      pageContent:
        "Full-time employees receive twenty days of paid vacation per year.",
    },
  ];
  const answer =
    "Full-time employees receive thirty days of paid vacation per year.";

  expect(answer).not.toBeGroundedIn(sources);

  const check = await checkAnswer({ answer, sources });
  expect(check.grounded).toBe(false);
  expect(check.unsupported).toEqual([answer]);
});

The guide is built from a runnable project, examples/testing-typescript. A test keeps the two identical. Before each release, a check installs the packed package with the latest Vitest and TypeScript, then type-checks the project, runs its tests and runs its evaluation, so the code in the guide works as written. Read the guide.

Also

• Releases announce themselves. This post reached you through it: publishing a release now posts to GitHub Discussions, dev.to, Bluesky and Mastodon, built from the release notes and this blog post. Every post passes checks first: each number must appear in the files it came from, and each link must be live.
• Fewer development dependencies. Eight unused OpenTelemetry SDK packages and three modules nothing loaded are gone, 237 packages in all. npm audit findings in development tooling fell from 64 to 14, none high. What you install from npm had none and still has none.

The scoped packages, @devilsdev/rag-mcp-server and create-rag-pipeline are released at 3.8.0 too. Full details: CHANGELOG.