Python · 2025
ArQiv
A search engine for arXiv papers that layers boolean, BM25, TF-IDF, vector and BERT ranking over one index.
- index structures
- 3
- ranking modes
- 5
- boolean search
- < 5 ms
- index build
- 0.8 s
ArQiv indexes the arXiv corpus with three structures and lets you choose how results are ranked, from exact boolean matches to a transformer reading the abstracts. It runs as a terminal app and as a hosted web app.
Inside
- Inverted index with positions, for term lookups and phrases.
- Trie, for prefix autocomplete and fuzzy matching.
- Bitmap index, for vectorised boolean queries.
- Rankers: BM25, TF-IDF, nearest-neighbour vectors and an optional BERT re-ranker, with an in-memory cache for repeated queries.
Numbers
Measured on a Ryzen 3 with 8 GB of RAM, over 1,000 documents:
| Task | Time |
|---|---|
| Build the index | 0.8 s |
| Boolean search | under 5 ms |
| TF-IDF or vector ranking | under 5 ms |
| BM25 ranking | 50–100 ms |
| BERT ranking | about 200 ms |
The index costs about 50 MB per thousand documents, which is the wall a sharded version would have to climb.