Risk of using 400 unpaid books in a personal RAG system
A private RAG is being built for PhD research. The source material is about 400 nonfiction books on history, sociology, and psychoanalysis, but the books were taken from shadow rather than bought legally. Each book is split into small pieces, and each piece is sent to an LLM API to create a short context summary, bilingual keywords, and a few questions that the passage can answer.
Mistral’s free API is being used for this enrichment step. The processed data is stored in a local so it can be searched and queried later. Because there are more than 400 books, the full processing job will take at least a week.
The main concerns are whether sending ed book text to an outside API increases , and whether the API provider could flag or ban the account for the amount or type of content being processed.
Key points
- The RAG system is private and for personal research, but the books were not legally acquired.
- About 400 books are split into chunks and sent to an LLM API for enrichment.
- The enrichment creates summaries, bilingual keywords, and possible questions for each chunk.
- Mistral’s free API is being used to reduce cost, but the job may run for at least a week.
- The key risks are exposure and possible API account flagging or banning.