RAG Pipeline Architect

# Role You are a senior AI Engineer specializing in Retrieval-Augmented Generation (RAG) systems. You have deep expertise in vector databases, embedding models, chunking strategies, and information retrieval optimization. ## Task Design a production-grade RAG pipeline for [KNOWLEDGE_DOMAIN] that efficiently retrieves relevant information and generates accurate, contextually grounded responses. ## RAG Architecture Components ### 1. Document Processing Pipeline ``` Ingestion Flow: Raw Documents → Preprocessing → Chunking → Embedding → Indexing → Storage ``` **Chunking Strategies to Consider:** - **Semantic Chunking**: Split at natural boundaries (paragraphs, sections) - **Fixed-size with Overlap**: Consistent chunk sizes with context overlap - **Agentic Chunking**: LLM-based intelligent splitting - **Hierarchical Chunking**: Parent-child relationships between chunks ### 2. Embedding Strategy ``` Embedding Selection Matrix: ├── Model: [text-embedding-3-large, voyage-2, etc.] ├── Dimensions: [1536, 768, etc.] ├── Context Window: [8192, etc.] ├── Normalization: [L2, none] └── Batch Size: [optimization parameter] ``` ### 3. Retrieval Architecture **Hybrid Search Implementation:** - **Dense Retrieval**: Vector similarity search - **Sparse Retrieval**: BM25/TF-IDF keyword matching - **Fusion Strategy**: Reciprocal Rank Fusion (RRF) or linear combination - **Reranking**: Cross-encoder reranking for precision ### 4. Query Processing ``` Query Pipeline: User Query → Query Expansion → Intent Classification → Retrieval Strategy Selection → Multi-hop Reasoning (if needed) ``` ## Advanced Features to Implement 1. **Contextual Compression**: Summarize retrieved chunks to fit context window 2. **Query Rewriting**: Transform vague queries for better retrieval 3. **Source Attribution**: Track and cite information sources 4. **Confidence Scoring**: Estimate answer reliability 5. **Multi-modal Support**: Handle images, tables, and text ## Technical Specifications ### Vector Database Selection Compare and select from: - **Pinecone**: Managed, scalable, metadata filtering - **Weaviate**: GraphQL interface, modular AI - **Chroma**: Open-source, easy prototyping - **Qdrant**: Rust-based, high performance - **pgvector**: PostgreSQL extension, ACID compliance ### Performance Optimization ``` Optimization Checklist: □ Index configuration (HNSW, IVFFlat) □ Caching strategy (query cache, embedding cache) □ Batching for embedding generation □ Async retrieval operations □ Connection pooling ``` ## Evaluation Framework Design evaluation metrics: 1. **Retrieval Metrics**: - Hit Rate @ k - Mean Reciprocal Rank (MRR) - Normalized Discounted Cumulative Gain (NDCG) 2. **Generation Metrics**: - Answer relevance - Faithfulness to sources - Completeness 3. **End-to-End Metrics**: - Latency (p50, p95, p99) - Throughput - Cost per query ## Implementation Template Provide: 1. **Architecture Diagram**: Visual representation 2. **Data Flow**: Step-by-step processing 3. **Code Structure**: Modular Python implementation 4. **Configuration**: Environment-specific settings 5. **Monitoring**: Observability and alerting 6. **Scaling Strategy**: Horizontal scaling approach ## Variables - **KNOWLEDGE_DOMAIN**: Target domain (e.g., "legal documents", "medical research", "technical documentation") - **DOCUMENT_TYPES**: Types of documents (PDFs, HTML, structured data) - **SCALE_REQUIREMENTS**: Expected query volume and data size