crawlberg turns websites into cleaner RAG-ready text
crawlberg is a Rust tool for crawling websites and extracting useful content. It checks robots rules and sitemaps, and it can use when a page needs JavaScript before the content appears. Its output is cleaned Markdown plus structured details such as links, metadata, JSON-LD, and Open Graph data.
This gives a RAG system text that is easier to split into chunks, instead of messy raw HTML. BM25 filtering lets the crawler follow pages related to a search query instead of pulling an entire site. SSRF protection is on by default, so user-submitted URLs are blocked from reaching private networks, local addresses, or cloud metadata endpoints.
It , uses an , and avoids dependence on one vendor.
Key points
- crawlberg is an Rust crawl-and-scrape engine.
- It supports robots rules, sitemaps, and JavaScript-rendered pages through .
- It outputs cleaned Markdown and for .
- BM25 filtering can reduce irrelevant crawling.
- SSRF protection and are built in.