RAG Pipeline Utils v3.5: Checks You Can Measure, See in Your Traces, and Run on the Model's Own Citations

v3.5.0 is about trusting the checks. The guards take a model of your choice for what patterns miss, and the patterns' accuracy is now published, misses included. Evaluation scores show up in OpenTelemetry and Langfuse, on the spans they judge. And when Claude or OpenAI file search cites its sources, verifyCitations() checks that each cited passage actually says what the sentence claims. Nothing breaks for existing code.

npm install @devilsdev/rag-pipeline-utils@^3.5.0

Security fix

On 3.4.0 and earlier, an error from AnthropicConnector, CohereConnector or OllamaConnector carried the request headers, so logging it, or sending it to an error reporter, printed your API key. 3.5.0 fixes it. If your logs may hold such errors, rotate the key. OpenAIConnector was not affected.

Bring your own detector

The guards' injection and PII checks are patterns: fast, free and offline, but blind to what they have no rule for, such as an injection in plain words or a person's name. Each guard now takes detectors, an injection classifier or a PII service, alongside the patterns or, with builtIn: false, instead of them:

const assistant = new GuardrailsPipeline(pipeline, {
  preRetrieval: { detectors: [injectionClassifier] },
  retrieval: { detectInjection: true, detectors: [injectionClassifier] },
  postGeneration: { detectors: [presidio] },
});

A detector is { name, kind: "injection" | "pii", detect(text, context) }, sync or async. A detector that throws blocks by default, because a check skipped on a timeout is an attack let through, and what a PII detector finds is redacted everywhere the answer is copied. runContractTests("detector", ...) checks your own against what the guards rely on. Plug in your own detectors

Guardrail accuracy, published

How good are the patterns? The Benchmarks page now says, on public test splits nothing was tuned on, measured in CI:

• Prompt injection (deepset/prompt-injections, 116 prompts): the patterns flag nothing harmless (precision 1.000) and catch 4 of 60 injections (recall 0.067). A DeBERTa injection classifier run as a detector catches 22 (0.367), also with no false alarms.
• RAG attacks (40 fixtures written for this project, illustrative): the patterns catch 3 of 23, the classifier 15.
• PII (Gretel's synthetic financial documents, 2,891): the patterns find 95% of emails and 92% of SSNs, but almost no names, companies or street addresses, which make up most of the PII in real documents.

Those numbers are the case for a detector, which is why both shipped together. To measure your own on cases you label, and fail CI when recall drops:

rag-pipeline eval-guardrails cases.jsonl --detectors ./detectors.mjs \
  --min my-classifier.recall=0.6 --report guardrails.html

Measure your detectors

Scores in your traces

With telemetry: true and evaluate or citations on, each score becomes a gen_ai.evaluation.result event, as the OpenTelemetry GenAI conventions define it, parented to the span it judges: faithfulness on the chat span, context precision on retrieval. The run result carries its trace and span ids, and toLangfuseScores(result) turns its scores into langfuse.score.create() objects attached to the right observation:

const result = await pipeline.run({
  query,
  options: { evaluate: true, citations: true },
});
for (const score of toLangfuseScores(result)) langfuse.score.create(score);

Events use the logs API of @opentelemetry/api 1.10 or later, so there is no new dependency. One change follows the conventions: the retrieval span is now named retrieval (it was retrieval <collection> for pgvector and Qdrant). Evaluation scores as events

Check the model's own citations

A citation can point at a real document and still not say what the sentence claims. verifyCitations() checks each claim against the passages it cites, with the same word, number and negation checks as checkAnswer, and the optional judge for paraphrases:

const check = await verifyCitations(
  fromAnthropicCitations(message, { documents }),
);
check.verified; // every cited claim supported
check.claims; // [{ text, verdict, lexicalReason, citations: [{ sourceId, score }] }]
check.uncited; // sentences no citation covers

Claude's citations carry the cited text, so they are checked from the response alone. OpenAI's file citations don't; ask for include: ["file_search_call.results"] and fromOpenAIFileCitations(response) uses the search results. Verify the citations a model returns

Fixed: current Claude models

If you use AnthropicConnector, upgrade. On 3.4.0 the LLM judge failed with an HTTP 400 on Claude Haiku 5.5, Sonnet 5.5 (the connector's own default), Opus 5.5 and Fable 5.1, which accept no temperature but the default, and generate() returned undefined when a model thought before answering. 3.5.0 retries without temperature (with a warning), returns every text block of the reply, rejects a refusal with code: "MODEL_REFUSED" instead of a TypeError, and reports the judge's token usage. Every current Claude model now passes the live tests as an LLM and as the judge.

Tested the way you'll use it

Before this release, the package was installed from its release tarballs into empty projects and used the way a developer would, with the Claude API: the README quick start in ESM, CommonJS and strict TypeScript, the Anthropic SDK with checkAnswer and verifyCitations, the Vercel AI SDK middleware, guardrails, the CLI, telemetry and the MCP server. Besides the security fix above, that found three more things, all fixed in 3.5.0: the type declarations didn't compile with @types/node 25 and later, CostTracker under-counted input tokens, and a strict guardrails block didn't say why.

Also

• rag-pipeline eval-guardrails and evaluateDetectors() report precision, recall, F1, the confusion counts and a 95% interval per detector and per tag, and list case ids, never case text.
• RetrievalGuard's ACL filter is tested for correctness over every combination of restriction and permissions.
• Every feature that calls a model is now tested against the real Claude API (npm run test:live): the connector, routing, reranking, agentic planning, dataset generation, the judge, citations, the AI SDK adapter and middleware on AI SDK 6 and 7, and the CLI. It found that CostTracker had no price for Claude Haiku 5.5 and Fable 5.1, now fixed.

The scoped packages (@devilsdev/rag-core, -connectors, -eval, -guardrails) and @devilsdev/rag-mcp-server are released at 3.5.0 too. Next, in 3.6.0: a Pinecone retriever and npm create rag-pipeline starters. Full details: CHANGELOG.