Skip to main content

RAG Pipeline Utils v3.4: An LLM Judge for Paraphrases and an MCP Server for Your Documents

· 4 min read
Ali Kahwaji
Ali Kahwaji
Creator & Maintainer of RAG Pipeline Utils

v3.4.0 closes the biggest gap in the answer check, paraphrases, without giving up what makes it trustworthy, and puts your documents one command away from Claude. Nothing breaks for existing code: without the new option, every check returns the same result as before.

npm install @devilsdev/rag-pipeline-utils@^3.4.0

An LLM judge for paraphrases​

The answer check compares words, numbers and negation with the sources. That makes it free, fast and the same on every run, but a correct paraphrase shares too few words with its source to pass: "Staff working full time get 20 days of paid leave annually" was reported unsupported against "Full-time employees receive twenty days of paid vacation per year."

Pass judge, any LLM with this library's generate() contract, and it rules on those sentences:

import { checkAnswer, OpenAIConnector } from "@devilsdev/rag-pipeline-utils";

const check = await checkAnswer({
  answer: "Staff working full time get 20 days of paid leave annually.",
  sources: [
    "Full-time employees receive twenty days of paid vacation per year.",
  ],
  judge: new OpenAIConnector({ apiKey: process.env.OPENAI_API_KEY }),
});

check.grounded; // true when the judge accepts the paraphrase
check.judge.verdicts; // [{ sentence, verdict, sourceId, reason, lexicalReason, decidedBy }]

The deterministic check stays in charge:

  • The judge only sees sentences that failed for lack of shared words. A sentence with a wrong number, or one negated where its source is not, never reaches it and stays unsupported.
  • A "supported" verdict has to name a source, and that source must pass the number and negation checks. So a judge that says yes to everything still can't accept "thirty days" against a source that says twenty.
  • It fails closed. Output that isn't the expected JSON, or a source id that doesn't exist, keeps the sentence unsupported.
  • One call per answer at most, and none when the word check settles everything. check.judge.calls and check.judge.usage show what it cost.

The AI SDK middleware takes the same option, so generateText and streamText get the verdicts in providerMetadata.ragPipeline.judge. There, a judge that throws, on a rate limit say, doesn't fail the call: the answer comes back and its sentences stay unsupported. See the LLM judge for the details.

Your documents in Claude, with one command​

@devilsdev/rag-mcp-server is an MCP server for a folder of documents. It indexes the folder when it starts, offline, and gives Claude, Cursor or any MCP client two tools: search_documents returns the best passages with their files, and check_answer checks an answer against them sentence by sentence.

claude mcp add docs -- npx -y @devilsdev/rag-mcp-server /absolute/path/to/docs

It reads Markdown, HTML, CSV and text files, needs no API key, database or network, and the MCP SDK stays out of the main package's install. See the MCP server docs, including the Claude Desktop config.

A shorter npm page​

The npm page showed the whole README, nearly 700 lines. It now has a page of its own, about 110 lines: what the library does, an answer check, an offline pipeline, what's included and the packages. The full README, with every use case, is still the repository's front page on GitHub.

Also​

  • Ten of the README's use cases run on StackBlitz with no install or API key, and one checks an answer you paste in.

The scoped packages (@devilsdev/rag-core, -connectors, -eval, -guardrails) are released at 3.4.0 too, and @devilsdev/rag-mcp-server starts at 3.4.0. The rest of the planned 3.4 work, measured guardrails, provider citations, score export, Pinecone and starters, moves to 3.5.0. Full details: CHANGELOG.