ai · retrieval · benchmark
lanka-rag: which embedding model understands Sinhala?
Sinhala NLP resources are thin, so I measured. A reproducible retrieval benchmark of 7 local Ollama embedding models and 2 BM25 baselines on Sinhala, Tamil and Singlish (900 questions per task, with bootstrap confidence intervals), plus a local RAG assistant over 1,010 pages of Sri Lankan Acts.
- bge-m3 leads Sinhala at 0.913 nDCG@10, but plain BM25 over character n-grams gets 0.899.
- Python's
\wregex splits Sinhala words apart, which quietly breaks many BM25 baselines. - Singlish is the open problem: the best model reaches only 0.31.
Plot twist: a 30-year-old keyword algorithm nearly beat a 568M-parameter model. I re-ran it three times.
Sinhala retrieval quality
nDCG@10 on 900 questions · higher is better
Lighter bar: BM25 keyword search, no neural model.