Skip to main content

Security Capabilities

This page is the honest scorecard of what the framework actually implements vs. what it documents as best practice vs. what ships as example code. Use it for vendor-evaluation checklists and threat-model reviews.

For how to report a vulnerability, see SECURITY.md in the repo.

Legend​

MarkerMeaning
🟢 ImplementedCode in the public API with tests. Follows the SEMVER policy.
🟡 RecommendedThe framework provides primitives and documentation, but the user is responsible for the wiring. We don't ship a "turn it on" switch.
🟠 ExperimentalExported from @devilsdev/rag-pipeline-utils/experimental. Simulated or incomplete; not covered by SEMVER. Do not rely on it in production.
🔴 Not in scopeOutside the framework's responsibility — handled by surrounding infrastructure.

Pre-retrieval guardrails​

CapabilityStatusWhere it livesNotes
Prompt-injection detection🟢 ImplementedPreRetrievalGuard (preRetrieval.enableInjectionDetection), detectInjection()Pattern rules matched against normalised text (NFKC, zero-width characters removed, homoglyph and leetspeak folding). Extra patterns via injectionPatterns. The query is never rewritten; strict mode blocks. Heuristic, not a guarantee.
Topic / off-topic filtering🟢 ImplementedpreRetrieval.blockedTopics / preRetrieval.allowedTopicsKeyword-based topic lists (opt-in via enableTopicFiltering).
Query length limits🟢 ImplementedpreRetrieval.maxQueryLengthDefault 10000 characters.
Rate limiting per principal🟡 RecommendedWire your own middleware (Fastify, Express, Lambda authorizer)The framework does not own the request lifecycle.
Authentication / authorization🔴 Not in scopeYour edge / API gatewayThe framework receives an authenticated principal; it does not authenticate.

Retrieval-time guardrails​

CapabilityStatusWhere it livesNotes
Runs before generation🟢 ImplementedGuardrailsPipeline → options.filterResultsSince 2.5.0 the retrieval guard runs between retrieval and generation, so the LLM never sees removed documents. Removed documents are reported by id only.
Minimum relevance threshold🟢 Implementedretrieval.minRelevanceScoreDrops results whose score is below the threshold.
ACL filtering🟢 Implementedretrieval.enforceACL + context.userPermissionsChecks each document's requiredRole / acl (on the result or in its metadata) against the caller's roles. Fails closed since 2.5.0: without userPermissions, restricted documents are removed. Tenant isolation in your vector store is still your job.
Indirect prompt-injection scan🟢 Implementedretrieval.detectInjection, injectionActionOpt-in. Scans retrieved document text; removes (default) or flags matching documents.
Freshness filter🟢 Implementedretrieval.maxAgeDaysDisabled by default (0).
Retrieved text as untrusted🟢 ImplementedBuilt-in LLM connectorsSince 2.5.0 retrieved context goes in the user message as delimited reference data, never in the system role, with a system prompt telling the model to treat it as data.

Post-generation guardrails​

CapabilityStatusWhere it livesNotes
PII detection and redaction🟢 ImplementedpostGeneration.enablePIIDetection, piiPatternsRegex patterns for emails, phone numbers, SSNs and credit cards. Since 2.5.0 PII is also redacted in citations and evaluation details, and strict-mode blocks do not return the raw answer.
Groundedness scoring🟢 ImplementedpostGeneration.enableGroundednessCheck, CitationTracker, computeGroundedness()Lexical support per sentence plus a numeric-consistency check. Negation, units and multi-document claims are not checked.
Hallucination detection🟢 ImplementeddetectHallucinations(), computeFaithfulness()Lexical (not LLM-judged): sentences with low support in the retrieved documents are flagged.
Toxicity / safety classification🟡 RecommendedYour own check on result.answerNot shipped (model choice is policy-laden).

Supply-chain security​

CapabilityStatusWhere it livesNotes
Pinned GitHub Actions by SHA🟢 Implemented.github/workflows/*.ymlActions are pinned to a commit SHA, not a tag.
Dependabot enabled🟢 Implemented.github/dependabot.ymlWeekly npm scan.
Runtime npm audit gate🟢 Implemented.github/workflows/supply-chain.ymlnpm audit --audit-level=high --omit=dev. For 2.5.0 it reports no high or critical findings.
License allowlist enforcement🟢 Implementedsupply-chain.yml license-checker stepProduction dependencies are checked against an allowlist of licenses.
SBOM generation🟢 Implementednpm run sbomCycloneDX format.
Provenance attestation (npm publish --provenance)🟢 Implemented.github/workflows/release.ymlReleases are published from the release workflow with OIDC and --provenance.
Tarball smoke test🟢 Implementednpm run smoke:packInstalls the packed tarball in a clean project and imports it (CJS, ESM, TypeScript) in CI and before publish.
Signed git tags🟡 RecommendedUse git tag -sNot enforced by the project.

Plugin trust​

CapabilityStatusWhere it livesNotes
Ed25519 manifest signatures🟢 ImplementedPluginSignatureVerifier, pluginRegistry.register()On and fail-closed by default when NODE_ENV=production. The trust store is empty by default (2.5.0); add signers explicitly.
Code binding (contentDigest)🟢 ImplementedSigned manifest fieldA signed contentDigest binds the plugin files and is always checked when present (the shared registry verifies by default only when NODE_ENV=production). requireContentDigest / requireSignedPlugins are opt-in.
Install-time static scan🟢 Implementedrag-pipeline hub installHigh-risk findings block the install unless --allow-risky; the stage contract is checked in a separate process.
Plugin sandboxing🟠 ExperimentalPluginSandbox (/experimental)Non-functional; plugins run in-process with your privileges. See Experimental.

Authentication primitives​

CapabilityStatusWhere it livesNotes
JWT validation helper🟢 ImplementedJWTValidatorVerifies signature, expiry, audience, issuer. Replay protection via a bounded jti store (maxTrackedJtis); external tokens are single-use by default (singleUseTokens).
SSO (OAuth2 / OIDC / SAML / AD)🟠 Experimental@devilsdev/rag-pipeline-utils/experimentalOIDC validates id_tokens and uses PKCE, but the flows are untested against real identity providers; SAMLProvider refuses to authenticate. Use a vetted library such as openid-client in production.

Logging and audit​

CapabilityStatusWhere it livesNotes
Structured logging🟢 Implementedlogger export (pino-based)JSON output.
Secret redaction in logs🟢 ImplementedInternal secure-logging utilitiesPattern-based redaction of API keys and bearer tokens in log data.
Audit log🟢 ImplementedAuditLoggerEncrypted fields (createCipheriv, random IV) when a key is set via encryptionKey or RAG_AUDIT_ENCRYPTION_KEY; requireEncryption makes a missing key an error.
Tamper detection🟢 ImplementedAuditLogger.verifyIntegrity()HMAC signatures compared in constant time. Pass a stable signingKey, otherwise records written before a restart are flagged.
Centralized log shipping🔴 Not in scopeWire your own (Datadog, Splunk, Loki, OTEL)The framework writes structured logs; shipping is a platform concern.

Input validation​

CapabilityStatusWhere it livesNotes
Basic input validation🟢 ImplementedvalidateInput(input, { maxLength })Length limit and a small denylist of tokens; returns { valid, reasons, sanitized }.
Path traversal protection🟡 RecommendedYour application codeNo path sanitizer is exported; see the Security guide.
JSON schema validation for plugin contracts🟢 Implementedcontracts/*.jsonPlugins fail to register if they don't implement the stage contract.
ReDoS protection on user-supplied regex🟡 RecommendedYour application codeIf you accept regex from users, run it through a safe runtime (e.g. an RE2 binding).

Data governance​

CapabilityStatusWhere it livesNotes
Multi-tenancy / governance🟠 ExperimentalTenantManager, DataGovernanceManager in /experimentalSimulated: usage figures and data classification are random and GDPR erasure deletes nothing. Do not use for isolation or compliance.
Data retention policies🟡 RecommendedImplement on your storage backendThe framework reads/writes; lifecycle is the storage layer's job.
Encryption at rest🔴 Not in scopeStorage backend (Postgres TDE, S3 SSE, etc.)We don't manage your storage. FileVectorStore writes a plain JSON file.
Encryption in transit🟡 RecommendedBuilt-in cloud connectors use HTTPSVerify your custom plugins do too.

Security testing​

CapabilityStatusWhere it livesNotes
Unit tests for security utilities🟢 Implemented__tests__/unit/security/**JWT, audit logging, plugin signatures, command injection, SSO.
Guardrails tests🟢 Implemented__tests__/unit/guardrails/All three guardrail layers.
Static analysis (CodeQL)🟢 Implemented.github/workflows/security.ymlOn pushes and pull requests to main, plus weekly.
Penetration testing🔴 Not in scopeCustomer responsibilityWe publish disclosure-coordinated fixes.

What this framework does not protect against​

Calling this out so it's not assumed:

  • Compromised LLM provider — if your LLM API is hijacked, no framework guardrail will save you. Use trusted providers and treat outputs accordingly.
  • Embedding model bias — embedding models reflect their training data. We don't debias them. Use evaluation to detect regressions.
  • Side-channel timing attacks — guardrail evaluation takes measurable time; if you expose a public endpoint, a sufficiently motivated attacker can infer guardrail decisions from timing. Mitigate with constant-time response patterns at the HTTP layer.
  • Model exfiltration via inversion attacks — if your corpus contains secrets, an attacker with sufficient query budget can reconstruct fragments of it. The framework helps you set rate limits and ACLs but cannot eliminate this class of attack entirely.
  • Bring-your-own-plugin maliciousness — plugins run in your Node process with your privileges. Audit any third-party plugins you load.

See also​