#atom

Methods for providing external information to language models during inference

Core Idea: Different approaches exist for adding context to LLMs, each with trade-offs in efficiency, cost, and implementation complexity.

Key Elements

Context Stuffing

• Definition: Inserting documentation directly into the context window of an LLM
• Advantages:
  • Simple implementation
  • No additional infrastructure required
  • Direct access to all information
• Limitations:
  • Costly for token usage
  • Less efficient for large documentation sets
  • Limited by context window size
  • Potential retrieval problems depending on document structure
• Appropriate Use Cases:
  • Small documents
  • Single-query scenarios
  • When using models with strong in-context retrieval capabilities

Vector Indexing

• Definition: Splitting documents into chunks, embedding them, and storing in a vector database for retrieval
• Advantages:
  • Cost-effective for multiple queries
  • Highly scalable to large document collections
  • Fast retrieval speeds
• Limitations:
  • Complex configuration (chunk size, embedding model, similarity thresholds)
  • Requires upfront processing and infrastructure
  • May miss contextual relationships between chunks
• Technical Requirements:
  • Vector database or store
  • Embedding model
  • Chunking strategy
  • Query processing pipeline

LLMs.txt + Tool Calling

• Definition: Using LLMs.txt as a reference guide and allowing the model to retrieve specific URLs via tool calls
• Advantages:
  • Intuitive and human-like document navigation
  • Minimal setup compared to vector databases
  • Transparent decision-making process
  • Only retrieves documents relevant to specific queries
• Limitations:
  • Higher latency due to multiple tool calls
  • Requires initial LLMs.txt creation
  • Dependent on quality of document descriptions
• Implementation Components:
  • LLMs.txt file creation
  • URL loader tool
  • Tool calling capability

Connections

• Related Concepts: LLMs.txt Standard (implementation of reference approach), Tool Calling with LLMs (enables document retrieval)
• Broader Context: Document Reference Management for LLMs (overall approach to managing external knowledge)
• Applications: Using LLMs.txt with Development Tools (practical implementation in IDEs)
• Components: Model Context Protocol (MCP) (connects tools to applications)

References

1. RAG (Retrieval-Augmented Generation) literature
2. Vector database documentation (Pinecone, Chroma, etc.)
3. LLM context window optimization research

#context-loading #rag #vector-indexing #llm #knowledge-management

Connections:

Sources:

• From: LangChain - Vibe Coding LangGraph apps with llms.txt and MCP