RAG Pipeline Utils v3.9.0: Catching Answers That Drop a Condition

Answer checks now catch an answer that drops its source's condition, and are much faster on long answers. Text from PDFs and web pages is read the way it was typed, and a 1,000-sentence answer is checked in under half a second.

npm install @devilsdev/rag-pipeline-utils@^3.9.0
# and, if you use the MCP server:
npx @devilsdev/rag-mcp-server@3.9.0 ./docs

If you install @devilsdev/rag-core, rag-guardrails, rag-connectors or rag-eval directly, upgrade them to 3.9.0 as well.

Highlights

• A condition left out is caught. "Remote work is allowed two days per week" is no longer grounded by "Remote work is allowed two days per week with manager approval", and "Access is allowed" no longer by "Access is allowed, but not for guests". The check reports condition_mismatch. Turn it off with conditionCheck: false, or --no-condition-check on the CLI.
• Text is read as typed. Zero-width characters, soft hyphens, full-width digits and no-break spaces are normalised before text is compared, so a source copied from a PDF or a web page supports the sentence that quotes it. Sentences end where a reader would end them: "Dr. Lee" and "10 a.m." don't end one, and a list item does. Citation markers such as [1] and ¹ no longer fail a correct sentence.
• Long answers are checked in linear time. Each source and answer is prepared once per check, so a 1,000-sentence answer that took about 9 s is checked in under half a second. The MCP server counts its input limits the way the check does.
• Stable scores across source sets. A duplicated or unrelated source no longer changes a verdict, and computeFaithfulness, groundedness and PipelineEvaluator use the same word weights as checkAnswer.
• Fixes across the library. The chunkers keep all their text; FileVectorStore scores are true cosine similarities, and a save merges with what other processes saved; streams end with one complete done chunk and honour timeout; CostTracker checks the budget before a run; DAG node timeouts and concurrency take effect; and AuditLogger gains close() and a working webhook.

After upgrading

Some changes need action:

• Re-tune minRelevanceScore for FileVectorStore. Its scores are now cosine similarity (1 for an identical vector); they used to grow with the query vector's length.
• Re-ingest to pick up the chunker fixes. The recursive (default), fixed-size and sentence chunkers no longer drop or split text, so chunks can change.
• Raise timeout if you stream long answers. A run's timeout now covers a returned stream until it ends.
• Regenerate saved baselines that compare multi-source scores or hallucination classes. Scores move for answers checked against several sources, and hallucination classes change in many cases while rates and verdicts don't.

Benchmarks

On the condition-dropped benchmark, F1 on the test split is 85.9% lexical and 90.1% with a Claude judge on the released code. On the public RAGTruth and HaluEval sets, F1 moved by at most 0.01: the stricter condition check adds a few false alarms on RAGTruth, and HaluEval has three more misses. Every number, including those that got worse, is on the Benchmarks page.

Known gaps

The condition check is deliberately narrow. A negated contrast with no scope word ("but not guests", "but guests are not admitted") is not yet read as a restriction, and neither is a restriction attached to a different subject ("Staff may enter" against "Guests may enter, but staff need approval"). 3.10 plans restrictions that follow their subject, which covers these cases.

Thanks to the readers whose reports shaped this release.

The full list is in the CHANGELOG, and the release notes are on GitHub.