A small search app found 4 million duplicate videos
nsfwbase.com is an adult content search service that has been running and improving for more than eight months. Its database has 37 million videos, and about 4 million of them were identified as duplicates. Different methods, including checks, were used to find repeated videos, which greatly reduced duplicates in search results.
Some duplicates can still appear, so a future feature will let users mark duplicates themselves. The search system has three modes: text search, , and . Choosing the right was a major challenge, and around 10 models were tested on real data.
Reading papers was not enough because model quality depends on real use, including choices like vector size and whether to run on CPU/GPU hardware.
Key points
- The service has been developed for more than eight months.
- About 4 million duplicate videos were found inside a 37 million video database.
- checks helped detect repeated videos and reduce duplicate search results.
- The service supports text search, , and .
- Around 10 options were tested, and CPU/GPU choices affect hardware needs.