RAG Pipeline Utils v2.5.0: A Working CLI, Cancellation and Honest Metrics

v2.5.0 is a consolidation release. The CLI now does real work end to end, every pipeline stage can be cancelled, grounding metrics measure what they claim to, and modules that were simulations are now clearly separated in an /experimental entry point. Several of these changes alter scores and defaults, so please read the upgrade notes before you update.

A CLI that does real work

rag-pipeline init, ingest, query and run now work against a persistent local index:

npx rag-pipeline init --template local    # or openai | ollama | anthropic
npx rag-pipeline ingest ./docs
npx rag-pipeline query "What is the vacation policy?"

• init writes a .ragrc.json of built-in components. In a terminal it runs a setup wizard: it chooses the embedder, LLM, retriever and an optional reranker, checks custom plugin modules before accepting them, reports missing API key environment variables, can run a connection test, and previews the JSON before writing it.
• The index is a FileVectorStore, a JSON-file vector index with atomic writes, embedder/dimension tracking and per-source re-ingestion. It is also exported for use in code.
• .ragrc.json entries of the form { stage, name, config } resolve to built-in components, custom plugin modules listed under plugins, or plugins registered on pluginRegistry.
• rag-pipeline serve (and DevServer) exposes GET /health and POST /query on 127.0.0.1: JSON only, 64 KB body limit, no CORS.
• doctor and validate report real findings (unknown components, missing stages, missing API keys, broken custom plugins) and exit 1 on errors.

Commands and flags that printed success without doing anything have been removed. See the CLI reference.

Plugin install and certification

• rag-pipeline hub install installs a plugin from a local directory, a .tgz, a single .js file, npm:<package> or a registry (sha256-verified). Extraction is confined to .rag/plugins. A static security scan runs first, and high-risk findings block the install unless you pass --allow-risky. The stage contract is checked in a separate process, and the plugin is then registered in .ragrc.json. PluginHub.installPlugin() uses the same installer.
• rag-pipeline hub certify submit|check scores a local plugin from its files (contract, security scan, tests, documentation) against the BASIC/VERIFIED/ENTERPRISE automated thresholds. It previously issued certificates from fixed placeholder scores.
• rag-pipeline hub list lists the plugins registered in the config.
• There is no default registry. Registry operations need --registry <url> or the RAG_PLUGIN_REGISTRY environment variable.

rag-pipeline ai commands

The ai commands are rebuilt on the configured pipeline and index:

• ai train embedding refits the local TF-IDF embedder and re-embeds the index.
• ai benchmark reports measured p50/p95/mean latency per stage.
• ai feedback <question> <docId> --rating up|down stores votes in .rag/feedback.json. ai search applies them as a capped, explained score adjustment.
• ai multimodal process reports the type, character count and chunks of a text file. It exits 1 for images, because no built-in connector accepts image input.

The previous federated learning, fine-tuning, ai train status|deploy and ai dashboard commands printed hard-coded or simulated numbers and have been removed.

Cancellation everywhere

createRagPipeline now creates an AbortController per run and passes its signal to every component. The new options.signal lets you cancel a run yourself:

const controller = new AbortController();
const result = await pipeline.run({
  query: "Summarize the travel policy",
  options: { signal: controller.signal, timeout: 10000 },
});
// on abort: { success: false, error: "Execution aborted" }
// on timeout: { success: false, error: "Execution timeout" }

Components receive the signal as retriever.retrieve({ ..., signal }), embedder.embed/embedQuery(text, { signal }), reranker.rerank(q, docs, { signal }) and llm.generate(q, docs, { signal }). All built-in connectors and rerankers honour it, and AgenticPipeline and IterativeRetriever propagate it. Timeouts now cancel work, and no stage starts after an abort. The API reference describes the contract.

Other pipeline fixes:

• The embedder result was computed and discarded, so vector retrievers never received a queryVector. That is fixed.
• Without an llm, the pipeline runs retrieval only and returns answer: null instead of 'ok'.
• In stream mode with citations or evaluate, the stream ends with one { token: '', done: true, citations, evaluation } chunk.

Metric changes you need to know about

Scores from 2.5.0 are not comparable with earlier releases:

• Grounding (citations, faithfulness, context precision/recall, groundedness, PostGenerationGuard, SelfCritiqueChecker) now measures the share of a sentence's content words found in a document, optionally IDF-weighted. The old Jaccard similarity against the whole document shrank with document length: an answer copied verbatim from a ~70-word chunk scored 0. Expect noticeably higher scores for grounded answers.
• The default per-sentence similarity threshold rose from 0.3 to 0.5. This applies to CitationTracker.similarityThreshold, PipelineEvaluator.threshold and the faithfulness, recall and hallucination-detector thresholds. Aggregate thresholds are unchanged.
• Context recall needs a reference answer. Pass it as PipelineEvaluator.evaluate(result, { referenceAnswer }) or run({ options: { evaluate: true, referenceAnswer } }). Without one, the metric is skipped and listed in metadata.metricsSkipped, and groundedness becomes (0.4 * faithfulness + 0.3 * contextPrecision) / 0.7.
• Numbers are checked. A sentence whose numbers (digits, decimals, percentages, number words) are not in the source scores below 0.4 and is not grounded ("20" equals "twenty"). The helpers checkNumericConsistency and extractNumbers are exported.
• The tokenizer keeps non-ASCII letters and splits on newlines and hyphens. This also affects BM25, BLEU/ROUGE and the scoring reranker.

Grounding is still lexical plus a numeric check. Negation, units, ordinals and claims that combine several documents are not checked. See Evaluation.

Security fixes

• Guardrails: the retrieval guard (ACL, relevance, freshness) now runs before generation, through the new options.filterResults hook. Removed documents are reported by id only. Strict-mode blocks no longer return the raw answer. PII is redacted in citations and evaluation details too. RetrievalGuard with enforceACL fails closed when context.userPermissions is missing.
• Prompt injection: retrieved documents are no longer placed in the LLM system role. Built-in connectors pass context as delimited, untrusted reference data. Detection normalises homoglyphs, leetspeak and zero-width characters. detectInjection is exported, and RetrievalGuard can scan retrieved documents with detectInjection: true.
• AuditLogger:
  • Tenant, user and action filters never matched, so queries returned every tenant's records. They now filter correctly.
  • Encryption uses createCipheriv with a random IV per field.
  • The key comes from encryptionKey or RAG_AUDIT_ENCRYPTION_KEY; requireEncryption: true makes a missing key an error.
  • HMAC signatures are compared in constant time.
• InputSanitizer: a regex state bug let command-injection payloads through after a previously blocked input. It is fixed.
• Plugin signatures: Ed25519 verification works. A signed contentDigest binds plugin code and is always checked when present. The shared pluginRegistry verifies signatures by default only when NODE_ENV=production, and the trust store is empty by default.
• JWTValidator:
  • New singleUseTokens (default true) and maxTrackedJtis (default 100000) options; the replay store is bounded and expires entries.
  • An explicit clockTolerance or notBeforeTolerance of 0 is now honoured.
• Dependencies: npm audit --omit=dev reports no high or critical findings.
• .secrets is no longer tracked in git and is excluded from Docker builds. Rotate anything it held.

Experimental modules and deprecations

Some classes were simulations or incomplete: random training metrics, SSO that accepted any code outside production, random tenant quotas, GDPR erasure that deleted nothing, a non-functional plugin sandbox, fixed certification scores. They now live in a separate entry point:

import { TenantManager } from "@devilsdev/rag-pipeline-utils/experimental";

This covers ModelTrainingOrchestrator, FederatedLearningCoordinator, MultiModalProcessor, AdaptiveRetrievalEngine, SSOManager, SAMLProvider, OAuth2Provider, ActiveDirectoryProvider, OIDCProvider, TenantManager, ResourceMonitor, DataGovernanceManager / DataGovernance, PluginSandbox and PluginCertification.

The main-entry exports still work, but each emits a one-time DeprecationWarning (RAG_DEP_* codes). They are marked @deprecated in the types and will be removed from the main entry in the next major release. Both entries share the same classes, so instanceof keeps working. The Experimental modules page says what each one really does.

The experimental SSO also got security fixes:

• The development bypass requires devBypass: true and a non-production NODE_ENV.
• OIDC id_tokens are validated.
• OAuth2/OIDC use PKCE.
• SAMLProvider refuses to authenticate.

Packaging and TypeScript

• require() / import of 2.4.5 failed in every clean install with Cannot find module 'isolated-vm'. isolated-vm is now loaded only when PluginSandbox runs, and it is an optional peer dependency.
• A tarball smoke test runs in CI and before publish.
• 52 runtime exports had no TypeScript declarations. Declarations are now generated from JSDoc at build time, and the build fails if any export is undeclared.

Upgrade notes

npm install @devilsdev/rag-pipeline-utils@2.5.0

• Re-baseline evaluation thresholds. Grounding scores run higher, and the per-sentence threshold is now 0.5. Pass a referenceAnswer if you rely on context recall.
• Retrieval-only pipelines now return answer: null. Results always include answer when an llm is configured.
• DevServer.start() now listens on port 3000 and host 127.0.0.1 by default, so existing callers may see EADDRINUSE. Pass port: 0 for a free port. watchPaths now defaults to [].
• ACL filtering fails closed. With enforceACL, always pass context.userPermissions.
• PreRetrievalGuard returns the query unchanged and reports detections in injection. Strict mode still blocks.
• Answers may differ. Built-in connectors now put retrieved context in the user message.
• CostCalculator now prices gpt-4o-mini-* and claude-3-5-sonnet-* correctly. Built-in prices are marked unverified; see PRICING_METADATA.
• Plugin registry operations need --registry <url> or RAG_PLUGIN_REGISTRY.
• Removed CLI options: --trace, --stats, --parallel, --streaming, --max-memory, --preview, --validate, --explain, --abort-timeout and --quiet. Remove them from your scripts.
• docker compose now requires POSTGRES_PASSWORD, REDIS_PASSWORD and GRAFANA_ADMIN_PASSWORD (see .env.example) and publishes ports on 127.0.0.1 only.
• Experimental imports: move them to @devilsdev/rag-pipeline-utils/experimental.

The full list of changes is in the CHANGELOG.