AI · Extended learning
AI retrieval metric toy lab
Use a tiny original English corpus to show Recall@k changes with k; not an Azure AI Search, vector, or LLM-quality benchmark.
Run the local exercise
Key results
| Metric | Value | Unit |
|---|---|---|
| Average Recall@k | 100% | |
| Average MRR over full ranking | 1 | |
| Average MRR@k | 1 |
Returned ranking
| Metric | Value | Unit |
|---|---|---|
| private dns endpoint | Known relevant IDs: d1; Returned ranking: d1(3) > d2(0) > d3(0) > d4(0) > d5(0); Recall@k: 100%; Reciprocal rank (full): 1; Reciprocal rank@k: 1 | |
| restore replication backup | Known relevant IDs: d2; Returned ranking: d2(3) > d1(0) > d3(0) > d4(0) > d5(0); Recall@k: 100%; Reciprocal rank (full): 1; Reciprocal rank@k: 1 | |
| capacity cpu traffic | Known relevant IDs: d4; Returned ranking: d4(3) > d1(0) > d2(0) > d3(0) > d5(0); Recall@k: 100%; Reciprocal rank (full): 1; Reciprocal rank@k: 1 | |
| evaluation recall | Known relevant IDs: d3, d5; Returned ranking: d5(2) > d3(1) > d1(0) > d2(0) > d4(0); Recall@k: 100%; Reciprocal rank (full): 1; Reciprocal rank@k: 1 | |
| authorization evaluation | Known relevant IDs: d3, d5; Returned ranking: d3(2) > d5(1) > d1(0) > d2(0) > d4(0); Recall@k: 100%; Reciprocal rank (full): 1; Reciprocal rank@k: 1 |
- The same original English synthetic corpus is used in both languages; this is lexical token matching, not embeddings or an LLM benchmark.
Experiment steps
- Use five fixed short documents, five queries, and known relevant IDs.
- Inspect known relevant IDs, returned ranking, scores, Recall@k, and reciprocal rank for each query.
- Separate retrieval hits from answer generation quality, citations, and authorization filtering.
Reproducible input corpus and relevance labels
These five original English sentences and query relevance IDs are the complete teaching input. Scores count exact query-token matches; tied documents are ordered by ID.
- d1
- Private Endpoint design requires private DNS zones and public network review.
- d2
- Backup restore drills compare RPO, RTO, validation, and replication assumptions.
- d3
- RAG retrieval needs citations, authorization filtering, and answer evaluation.
- d4
- Container capacity depends on CPU service time, traffic, replicas, and headroom.
- d5
- Classification evaluation uses precision, recall, confusion matrices, and thresholds.
- private dns endpoint — relevant IDs: d1
- restore replication backup — relevant IDs: d2
- capacity cpu traffic — relevant IDs: d4
- evaluation recall — relevant IDs: d3, d5
- authorization evaluation — relevant IDs: d3, d5
Sources, formula, and limitations
Original English synthetic corpus, fixed queries, and known relevant IDs; computes Recall@k, MRR@k, and full-ranking MRR locally.
Both language pages use the same English corpus for reproduction; no embeddings, models, network calls, or validated human benchmark are used.
Lexical retrieval on a tiny corpus does not prove RAG reliability, grounding, safety, or production quality.