Version: 2.3.1 (Latest)

Usage Guide

This comprehensive guide covers how to use @DevilsDev/rag-pipeline-utils programmatically via the JavaScript/TypeScript API and through the command-line interface. Whether you're building a simple Q&A system or a complex enterprise RAG application, this guide provides practical examples and best practices.

Installation

NPM Installation

# Install as a project dependency
npm install @DevilsDev/rag-pipeline-utils

# Or install globally for CLI usage
npm install -g @DevilsDev/rag-pipeline-utils

# Using Yarn
yarn add @DevilsDev/rag-pipeline-utils

# Using pnpm
pnpm add @DevilsDev/rag-pipeline-utils

System Requirements

Node.js: >= 18.0.0
TypeScript: >= 4.5.0 (for TypeScript projects)
Memory: Minimum 2GB RAM (4GB+ recommended for large documents)
Storage: Varies based on vector store and document size

Environment Setup

Create a .env file in your project root:

# OpenAI Configuration
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_ORG_ID=your_org_id_here

# Pinecone Configuration
PINCONE_API_KEY=your_pinecone_api_key
PINCONE_ENVIRONMENT=us-west1-gcp
PINCONE_INDEX_NAME=rag-documents

# Cohere Configuration (optional)
COHERE_API_KEY=your_cohere_api_key

# Anthropic Configuration (optional)
ANTHROPIC_API_KEY=your_anthropic_api_key

Programmatic API Usage

Basic Pipeline Setup

Here's a complete example showing how to create and use a RAG pipeline:

import {
  createRagPipeline,
  PipelineConfig,
} from "@DevilsDev/rag-pipeline-utils";
import dotenv from "dotenv";

// Load environment variables
dotenv.config();

// Define pipeline configuration
const config: PipelineConfig = {
  loader: "markdown",
  embedder: "openai",
  retriever: "pinecone",
  llm: "openai-gpt-4",
  useReranker: true,

  // Plugin-specific configuration
  config: {
    openai: {
      apiKey: process.env.OPENAI_API_KEY,
      model: "text-embedding-3-large",
      maxTokens: 4000,
      temperature: 0.1,
    },
    pinecone: {
      apiKey: process.env.PINECONE_API_KEY,
      environment: process.env.PINECONE_ENVIRONMENT,
      indexName: process.env.PINECONE_INDEX_NAME,
      topK: 5,
    },
  },

  // Pipeline-level configuration
  pipelineConfig: {
    chunkSize: 1000,
    chunkOverlap: 200,
    maxConcurrency: 3,
    enableCaching: true,
  },
};

// Create the pipeline
const pipeline = createRagPipeline(config);

// Use the pipeline
async function main() {
  try {
    // Run a query through the pipeline
    console.log("Querying pipeline...");
    const result = await pipeline.run({
      query: "How do I implement custom plugins in the RAG pipeline?",
      options: {
        topK: 5,
        timeout: 10000,
      },
    });

    if (result.success) {
      console.log("Query:", result.query);
      console.log("Results:", result.results);
    } else {
      console.error("Query failed:", result.error);
    }
  } catch (error) {
    console.error("Pipeline error:", error);
  } finally {
    // Clean up resources
    await pipeline.cleanup();
  }
}

main();

Advanced Configuration Examples

Multi-Model Setup

// Use different models for different purposes
const advancedConfig: PipelineConfig = {
  loader: "pdf",
  embedder: "openai",
  retriever: "chroma",
  llm: "anthropic-claude",
  useReranker: true,
  reranker: "cohere-rerank",

  config: {
    openai: {
      apiKey: process.env.OPENAI_API_KEY,
      model: "text-embedding-3-large",
    },
    anthropic: {
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: "claude-3-sonnet-20240229",
      maxTokens: 8000,
    },
    chroma: {
      host: "localhost",
      port: 8000,
      collection: "documents",
    },
    cohere: {
      apiKey: process.env.COHERE_API_KEY,
      model: "rerank-english-v2.0",
    },
  },
};

Local-First Setup

// Use local models for privacy-sensitive applications
const localConfig: PipelineConfig = {
  loader: "markdown",
  embedder: "sentence-transformers",
  retriever: "chroma",
  llm: "ollama",

  config: {
    sentenceTransformers: {
      model: "all-MiniLM-L6-v2",
      device: "cpu",
    },
    ollama: {
      baseUrl: "http://localhost:11434",
      model: "llama2:7b",
      temperature: 0.2,
    },
    chroma: {
      path: "./vector_store",
      collection: "local_docs",
    },
  },
};

Pipeline Methods & Operations

Document Ingestion

The base createRagPipeline returns a pipeline with run() method for queries. Document ingestion is handled by your retriever plugin's storage methods. Use your retriever directly to load and index documents before running queries.

Running Queries

// Basic query with the run() method
const result = await pipeline.run({
  query: "What is RAG?",
  options: { topK: 3 },
});

if (result.success) {
  console.log("Results:", result.results);
}

// Query with advanced options
const detailedResult = await pipeline.run({
  query: "Explain the plugin architecture",
  options: {
    topK: 5,
    timeout: 15000,
  },
});

if (detailedResult.success) {
  console.log("Query:", detailedResult.query);
  console.log("Retrieved documents:", detailedResult.results);
}

// Using pre-computed query vector
const queryVector = await embedder.embed("How does caching work?");
const vectorResult = await pipeline.run({
  queryVector,
  options: { topK: 3 },
});

// Streaming responses (if your LLM plugin supports generateStream)
const stream = await pipeline.run({
  query: "How do I optimize performance?",
  options: { stream: true },
});

for await (const chunk of stream) {
  if (!chunk.done) {
    process.stdout.write(chunk.token);
  }
}

Batch Processing with ParallelProcessor

const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils");

// Process multiple queries efficiently
const queries = [
  "What is the plugin system?",
  "How do I configure embeddings?",
  "What are the supported vector stores?",
];

const processor = new ParallelProcessor({
  concurrency: 3,
  retryAttempts: 2,
});

const results = await processor.process(
  queries,
  async (query) => await pipeline.run({ query, options: { topK: 3 } }),
);

results.forEach((result, index) => {
  if (result.success) {
    console.log(`Query ${index + 1}:`, queries[index]);
    console.log(`Results found:`, result.results.length);
    console.log("---");
  }
});

Pipeline Cleanup

// Clean up pipeline resources
await pipeline.cleanup();
console.log("Last updated:", stats.lastUpdated);

// Clear the vector store
await pipeline.clear();

// Rebuild the index
await pipeline.rebuild();

// Export pipeline data
const exportData = await pipeline.export();
fs.writeFileSync("./pipeline-backup.json", JSON.stringify(exportData));

// Import pipeline data
const importData = JSON.parse(
  fs.readFileSync("./pipeline-backup.json", "utf8"),
);
await pipeline.import(importData);

Command Line Interface (CLI)

Global CLI Installation

# Install globally
npm install -g @DevilsDev/rag-pipeline-utils

# Verify installation
rag-pipeline --version
rag-pipeline --help

Project Initialization

# Create a new RAG project
rag-pipeline init my-rag-project
cd my-rag-project

# Initialize in existing directory
rag-pipeline init .

# Initialize with template
rag-pipeline init my-project --template enterprise

Configuration Management

# Create default configuration
rag-pipeline config init

# Set configuration values
rag-pipeline config set openai.apiKey sk-your-key-here
rag-pipeline config set pinecone.environment us-west1-gcp

# View current configuration
rag-pipeline config show

# Validate configuration
rag-pipeline config validate

Document Ingestion via CLI

# Ingest single file
rag-pipeline ingest ./document.pdf --loader pdf

# Ingest directory with specific loader
rag-pipeline ingest ./docs --loader markdown --recursive

# Ingest with custom settings
rag-pipeline ingest ./docs \
  --loader markdown \
  --embedder openai \
  --chunk-size 1500 \
  --chunk-overlap 300

# Batch ingest with progress
rag-pipeline ingest ./large-docs \
  --loader pdf \
  --batch-size 10 \
  --progress

Querying via CLI

# Simple query
rag-pipeline query "What is the plugin architecture?"

# Query with specific LLM
rag-pipeline query "Explain embeddings" --llm openai-gpt-4

# Query with options
rag-pipeline query "How to optimize performance?" \
  --max-tokens 2000 \
  --temperature 0.3 \
  --include-sources

# Interactive query mode
rag-pipeline query --interactive

Evaluation and Testing

# Run evaluation on test set
rag-pipeline evaluate ./test-queries.json --output results.csv

# Evaluate with specific metrics
rag-pipeline evaluate ./queries.json \
  --metrics bleu,rouge,bertscore \
  --output detailed-results.json

# Start evaluation dashboard
rag-pipeline dashboard --port 3000 --host 0.0.0.0

Pipeline Management via CLI

# View pipeline status
rag-pipeline status

# Clear vector store
rag-pipeline clear --confirm

# Rebuild index
rag-pipeline rebuild --force

# Export pipeline data
rag-pipeline export --output backup.json

# Import pipeline data
rag-pipeline import backup.json

Plugin Management

# List available plugins
rag-pipeline plugins list

# Install custom plugin
rag-pipeline plugins install ./my-custom-plugin

# Create plugin template
rag-pipeline plugins create my-loader --type loader

# Validate plugin
rag-pipeline plugins validate ./my-plugin

Configuration File (.ragrc.json)

Create a .ragrc.json file in your project root for persistent configuration:

{
  "version": "2.1.8",
  "pipeline": {
    "loader": "markdown",
    "embedder": "openai",
    "retriever": "pinecone",
    "llm": "openai-gpt-4",
    "useReranker": true,
    "reranker": "cohere-rerank"
  },
  "config": {
    "openai": {
      "model": "text-embedding-3-large",
      "maxTokens": 4000,
      "temperature": 0.1
    },
    "pinecone": {
      "environment": "us-west1-gcp",
      "indexName": "rag-documents",
      "topK": 5
    }
  },
  "pipelineConfig": {
    "chunkSize": 1000,
    "chunkOverlap": 200,
    "maxConcurrency": 3,
    "enableCaching": true
  },
  "logging": {
    "level": "info",
    "format": "json",
    "outputs": ["console", "file"]
  },
  "plugins": {
    "custom": ["./plugins/my-custom-loader.js", "@company/rag-plugin-suite"]
  }
}

Advanced Usage Patterns

Error Handling & Resilience

import {
  RagError,
  PluginError,
  ConfigurationError,
} from "@DevilsDev/rag-pipeline-utils";

try {
  const response = await pipeline.query("Complex query");
} catch (error) {
  if (error instanceof PluginError) {
    console.error("Plugin error:", error.pluginName, error.message);
    // Handle plugin-specific errors
  } else if (error instanceof ConfigurationError) {
    console.error("Configuration error:", error.message);
    // Handle configuration issues
  } else if (error instanceof RagError) {
    console.error("RAG pipeline error:", error.message);
    // Handle general pipeline errors
  } else {
    console.error("Unexpected error:", error);
  }
}

Event Monitoring

// Listen to pipeline events
pipeline.on("document:loaded", (event) => {
  console.log(`Loaded ${event.chunks} chunks from ${event.source}`);
});

pipeline.on("embedding:completed", (event) => {
  console.log(
    `Generated ${event.embeddings} embeddings in ${event.duration}ms`,
  );
});

pipeline.on("generation:token", (event) => {
  process.stdout.write(event.token);
});

pipeline.on("error:handled", (event) => {
  console.warn(`Handled error in ${event.context}:`, event.error.message);
});

Custom Middleware

import { Middleware } from "@DevilsDev/rag-pipeline-utils";

// Create custom logging middleware
class CustomLoggingMiddleware implements Middleware {
  name = "custom-logger";
  priority = 100;

  async beforeQuery(context: QueryContext): Promise<QueryContext> {
    console.log(
      `[${new Date().toISOString()}] Query started: ${context.query}`,
    );
    context.startTime = Date.now();
    return context;
  }

  async afterQuery(
    context: QueryContext,
    result: QueryResult,
  ): Promise<QueryResult> {
    const duration = Date.now() - context.startTime;
    console.log(
      `[${new Date().toISOString()}] Query completed in ${duration}ms`,
    );
    return result;
  }
}

// Add middleware to pipeline
const config: PipelineConfig = {
  // ... other config
  middleware: [
    new CustomLoggingMiddleware(),
    // ... other middleware
  ],
};

This comprehensive usage guide covers the most common patterns and advanced features of @DevilsDev/rag-pipeline-utils. For more specific use cases, check out our Tutorials section, or explore the CLI Reference for detailed command documentation.

Plugin Configuration

You can customize the pipeline by registering plugins:

registry.register("loader", "custom", new MyCustomLoader());
registry.register("retriever", "opensearch", new MyOpenSearchRetriever());

Configuration via `.ragrc.json`

Create a JSON file at your root:

{
  "loader": "directory",
  "embedder": "openai",
  "retriever": "pinecone",
  "llm": "openai-gpt-4",
  "useReranker": true
}

Used automatically when no CLI args are passed.

Next → CLI

Installation​

NPM Installation​

System Requirements​

Environment Setup​

Programmatic API Usage​

Basic Pipeline Setup​

Advanced Configuration Examples​

Multi-Model Setup​

Local-First Setup​

Pipeline Methods & Operations​

Running Queries​

Batch Processing with ParallelProcessor​

Pipeline Cleanup​

Command Line Interface (CLI)​

Global CLI Installation​

Project Initialization​

Configuration Management​

Document Ingestion via CLI​

Querying via CLI​

Evaluation and Testing​

Pipeline Management via CLI​

Plugin Management​

Configuration File (.ragrc.json)​

Advanced Usage Patterns​

Error Handling & Resilience​

Event Monitoring​

Custom Middleware​

Plugin Configuration​

Configuration via .ragrc.json​