Core RAG features for building lower-cost AI agents
The RAG is meant to plug into existing apps. It can split documents into smaller chunks using different methods for different document types, such as paragraph-based splitting or overlapping windows. For Markdown and HTML, each chunk keeps the heading path so the system knows where the text came from.
For -like files, each row-group chunk keeps the column headers so the data still makes sense. It supports many formats, including HTML, Markdown, PDF, text, JSON, XML, DOCX, XLSX, CSV, and PPTX. It can ingest files from a local computer, a web address, or S3.
Documents can carry metadata for . Search combines BM25 and with weighted RRF; the current default is 0.7 vector and 0.3 keyword, but that can be changed. It also supports document property filters, such as limiting results to PDFs or documents with a certain tag, and normal queries return chunks with information about the original documents.
Key points
- The is and designed to drop into existing apps.
- It supports different chunking methods for different document types.
- Markdown and HTML chunks keep heading paths, while chunks keep column headers.
- Search combines BM25 and , with adjustable weights.
- Metadata and document property filters can narrow search results before they reach the AI model.