Migration Guide

This comprehensive migration guide helps you upgrade between versions of @DevilsDev/rag-pipeline-utils and migrate from other RAG frameworks. Whether you're upgrading from an older version or switching from another solution, this guide provides step-by-step instructions and best practices.

---

Version Migration

Upgrading from v2.0.x to v2.1.x

Breaking Changes:

• Plugin contract standardization
• Configuration file format updates
• CLI command restructuring

Migration Steps:

1. Update Package:

npm update @DevilsDev/rag-pipeline-utils

2. Update Configuration:

# Backup existing config
cp .ragrc.json .ragrc.json.backup

# Migrate configuration format
rag-pipeline config migrate --from 2.0 --to 2.1

3. Update Plugin Registrations:

// v2.0.x (deprecated)
registry.register("llm", "openai", {
  ask: async (prompt) => {
    /* implementation */
  },
});

// v2.1.x (current)
registry.register("llm", "openai", {
  generate: async (prompt) => {
    /* implementation */
  },
});

4. Update CLI Commands:

# v2.0.x (deprecated)
rag-pipeline eval ./test-data.json

# v2.1.x (current)
rag-pipeline evaluate ./test-data.json

Upgrading from v1.x to v2.x

Major Changes:

• Complete plugin architecture overhaul
• New streaming support
• Enhanced evaluation framework
• TypeScript-first approach

Migration Checklist:

1. Install New Version:

npm uninstall @DevilsDev/rag-pipeline-utils
npm install @DevilsDev/rag-pipeline-utils@^2.0.0

2. Migrate Configuration:

// v1.x format
{
  "openai_api_key": "sk-...",
  "pinecone_api_key": "...",
  "chunk_size": 1000
}

// v2.x format
{
  "plugins": {
    "embedder": {
      "name": "openai",
      "config": {
        "apiKey": "${OPENAI_API_KEY}",
        "model": "text-embedding-ada-002"
      }
    },
    "retriever": {
      "name": "pinecone",
      "config": {
        "apiKey": "${PINECONE_API_KEY}",
        "environment": "us-west1-gcp"
      }
    }
  },
  "pipeline": {
    "chunkSize": 1000
  }
}

3. Update Code:

// v1.x (deprecated)
import { RAGPipeline } from "@DevilsDev/rag-pipeline-utils";

const pipeline = new RAGPipeline({
  openaiKey: process.env.OPENAI_API_KEY,
  pineconeKey: process.env.PINECONE_API_KEY,
});

// v2.x (current)
import { createRagPipeline } from "@DevilsDev/rag-pipeline-utils";

const pipeline = createRagPipeline({
  embedder: { name: "openai" },
  retriever: { name: "pinecone" },
  llm: { name: "openai-gpt-4" },
});

4. Update Plugin Development:

// v1.x plugin interface
class MyLoader {
  load(filePath) {
    return { content: '...', chunks: [...] };
  }
}

// v2.x plugin interface
class MyLoader extends BaseLoader {
  constructor() {
    super();
    this.name = 'my-loader';
    this.version = '1.0.0';
  }

  async load(filePath, options = {}) {
    return [{
      id: this.generateId(filePath),
      content: '...',
      metadata: {},
      chunks: [...]
    }];
  }
}

---

Framework Migration

Migrating from LangChain

Key Differences:

• Plugin-based vs. chain-based architecture
• Built-in evaluation framework
• Streaming-first design
• TypeScript-native

Migration Steps:

1. Document Loaders:

# LangChain (Python)
from langchain.document_loaders import PyPDFLoader
loader = PyPDFLoader("document.pdf")
docs = loader.load()

# @DevilsDev/rag-pipeline-utils (JavaScript)
import { createRagPipeline } from '@DevilsDev/rag-pipeline-utils';
const pipeline = createRagPipeline({
  loader: { name: 'pdf' }
});
const docs = await pipeline.load('./document.pdf');

2. Embeddings:

# LangChain
from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
vectors = embeddings.embed_documents(texts)

# @DevilsDev/rag-pipeline-utils
const pipeline = createRagPipeline({
  embedder: { name: 'openai' }
});
const vectors = await pipeline.embed(texts);

3. Vector Stores:

# LangChain
from langchain.vectorstores import Pinecone
vectorstore = Pinecone.from_documents(docs, embeddings)
results = vectorstore.similarity_search(query)

# @DevilsDev/rag-pipeline-utils
const pipeline = createRagPipeline({
  retriever: { name: 'pinecone' }
});
await pipeline.index(docs);
const results = await pipeline.retrieve(query);

4. LLM Integration:

# LangChain
from langchain.llms import OpenAI
llm = OpenAI(temperature=0.7)
response = llm(prompt)

# @DevilsDev/rag-pipeline-utils
const pipeline = createRagPipeline({
  llm: {
    name: 'openai-gpt-4',
    config: { temperature: 0.7 }
  }
});
const response = await pipeline.generate(prompt);

Migrating from Haystack

Architecture Mapping:

• Haystack Nodes → RAG Pipeline Plugins
• Haystack Pipelines → RAG Pipeline Configurations
• Haystack Document Stores → RAG Retrievers

Migration Example:

# Haystack
from haystack import Pipeline
from haystack.nodes import EmbeddingRetriever, FARMReader

pipeline = Pipeline()
pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])
pipeline.add_node(component=reader, name="Reader", inputs=["Retriever"])

# @DevilsDev/rag-pipeline-utils
const pipeline = createRagPipeline({
  retriever: { name: 'elasticsearch' },
  llm: { name: 'huggingface-bert' }
});

Migrating from Custom Solutions

Common Migration Patterns:

1. Custom Vector Search:

// Custom implementation
class CustomVectorSearch {
  async search(query, vectors) {
    // Custom similarity calculation
    const similarities = vectors.map((v) => cosineSimilarity(query, v));
    return similarities.sort((a, b) => b.score - a.score).slice(0, 5);
  }
}

// Migrate to plugin
class CustomRetriever extends BaseRetriever {
  constructor() {
    super();
    this.name = "custom-vector-search";
    this.version = "1.0.0";
  }

  async retrieve(query, options = {}) {
    const queryVector = await this.embedder.embed(query);
    const results = await this.search(queryVector, this.vectors);
    return results.map((r) => ({
      id: r.id,
      content: r.content,
      score: r.score,
      metadata: r.metadata,
    }));
  }
}

2. Custom Chunking Logic:

// Custom implementation
function chunkDocument(text, chunkSize = 1000) {
  const sentences = text.split(".");
  const chunks = [];
  let currentChunk = "";

  for (const sentence of sentences) {
    if (currentChunk.length + sentence.length > chunkSize) {
      chunks.push(currentChunk);
      currentChunk = sentence;
    } else {
      currentChunk += sentence;
    }
  }

  return chunks;
}

// Migrate to loader plugin
class CustomLoader extends BaseLoader {
  chunkText(text, options) {
    // Integrate existing chunking logic
    return chunkDocument(text, options.chunkSize);
  }
}

---

Data Migration

Vector Store Migration

Export from Existing System:

# Export vectors and metadata
rag-pipeline export \
  --source pinecone \
  --index old-index \
  --output vectors-backup.json \
  --include-metadata

Import to New System:

# Import to new vector store
rag-pipeline import \
  --target chroma \
  --input vectors-backup.json \
  --create-index \
  --batch-size 1000

Cross-Platform Migration:

// Migration script
import { createMigrator } from "@DevilsDev/rag-pipeline-utils";

const migrator = createMigrator({
  source: {
    type: "pinecone",
    config: {
      /* source config */
    },
  },
  target: {
    type: "weaviate",
    config: {
      /* target config */
    },
  },
});

await migrator.migrate({
  batchSize: 500,
  parallel: 3,
  validateIntegrity: true,
});

Configuration Migration

Automated Migration Tool:

# Migrate configuration files
rag-pipeline migrate-config \
  --from ./old-config.json \
  --to ./.ragrc.json \
  --format v2.1 \
  --validate

Manual Migration:

// Migration helper
import { migrateConfig } from "@DevilsDev/rag-pipeline-utils";

const oldConfig = require("./old-config.json");
const newConfig = migrateConfig(oldConfig, {
  targetVersion: "2.1.0",
  preserveCustomFields: true,
  updatePluginNames: true,
});

fs.writeFileSync("./.ragrc.json", JSON.stringify(newConfig, null, 2));

---

Testing Migration

Validation Strategy

1. Configuration Validation:

# Validate migrated configuration
rag-pipeline config validate --strict

# Test plugin loading
rag-pipeline plugins list --validate

2. Functionality Testing:

# Test basic pipeline operations
rag-pipeline query "test query" --dry-run

# Run evaluation on sample data
rag-pipeline evaluate ./test-queries.json --metrics bleu,rouge

3. Performance Comparison:

# Benchmark before migration
rag-pipeline benchmark --baseline --output before-migration.json

# Benchmark after migration
rag-pipeline benchmark --compare before-migration.json

Rollback Strategy

Backup Before Migration:

# Create complete backup
rag-pipeline export --all --output pre-migration-backup.json

# Backup configuration
cp .ragrc.json .ragrc.json.pre-migration

Rollback Process:

# Restore from backup
rag-pipeline import pre-migration-backup.json --overwrite

# Restore configuration
cp .ragrc.json.pre-migration .ragrc.json

# Downgrade package if needed
npm install @DevilsDev/rag-pipeline-utils@1.9.0

---

Migration Tools & Utilities

Built-in Migration Commands

# Check migration requirements
rag-pipeline migration check --target-version 2.1.0

# Generate migration plan
rag-pipeline migration plan --from 2.0.5 --to 2.1.0

# Execute migration
rag-pipeline migration execute --plan migration-plan.json

# Verify migration
rag-pipeline migration verify --post-migration-tests

Custom Migration Scripts

// Custom migration script
import {
  createMigrationPlan,
  executeMigration,
  validateMigration,
} from "@DevilsDev/rag-pipeline-utils";

async function migrateProject() {
  // 1. Create migration plan
  const plan = await createMigrationPlan({
    from: "2.0.5",
    to: "2.1.0",
    projectPath: process.cwd(),
  });

  console.log("Migration plan:", plan);

  // 2. Execute migration
  const result = await executeMigration(plan, {
    backup: true,
    validateEachStep: true,
    rollbackOnFailure: true,
  });

  // 3. Validate migration
  const validation = await validateMigration({
    testSuite: "./migration-tests.json",
    performanceBaseline: "./baseline-metrics.json",
  });

  if (validation.success) {
    console.log("Migration completed successfully!");
  } else {
    console.error("Migration validation failed:", validation.errors);
  }
}

migrateProject().catch(console.error);

---

Migration Checklist

Pre-Migration

• Backup all data and configurations
• Document current system architecture
• Identify custom plugins and integrations
• Run baseline performance tests
• Prepare rollback strategy

During Migration

• Follow migration guide step-by-step
• Validate each migration step
• Test functionality after each major change
• Monitor for errors and warnings
• Document any custom modifications needed

Post-Migration

• Run comprehensive functionality tests
• Compare performance metrics
• Validate data integrity
• Update documentation and team training
• Monitor system stability
• Clean up old backups and temporary files

---

Migration Support

Common Migration Issues

1. Plugin Compatibility: Some v1.x plugins may not work with v2.x
2. Configuration Format: Manual updates may be required for complex configs
3. Performance Changes: New architecture may have different performance characteristics
4. API Changes: Some programmatic APIs have changed signatures

Getting Help

• Migration Documentation: Complete migration docs
• Community Forum: Ask migration questions
• Professional Services: Enterprise migration support

---

This migration guide ensures smooth transitions between versions and frameworks. For additional support, consult the Troubleshooting Guide or contact our migration specialists.