AWS design for making company documents ready for AI use
The is meant to turn scattered company documents into a platform for search, RAG, and internal AI apps. The documents would come from places such as Drive, , email, and s. Amazon S3 would store both raw files and processed files.
Airbyte and custom connectors would bring documents into the system. Unstructured.io and OCR would extract and clean up document content, while DataHub would track metadata such as source and document details. Amazon S3 Vectors is being considered for , FastAPI would provide the API layer, and Trino may be added if broader querying is needed.
AWS IAM, Secrets Manager, and CloudWatch would handle , secrets, and monitoring. The work may be done by only one or two developers, so early design choices need to balance scale with low operating burden.
Key points
- The goal is to prepare company documents for search, RAG, and internal AI apps.
- Raw and processed documents would be stored in Amazon S3.
- Unstructured.io and OCR would extract usable text from messy documents.
- DataHub would manage metadata about documents and their sources.
- The small team size makes simplicity and important design concerns.