Jev RAG Benchmark
Free English benchmark of an open System One model as a reranking and decision layer in RAG.
Reranking (frozen top-20 candidates)
| Dataset | n | Method | nDCG@10 | Recall@5 | MRR@10 | Rerank p50 |
|---|---|---|---|---|---|---|
| scifact | 300 | A — no reranker (hybrid order) | 71.67% | 79.67% | 68.75% | 0 ms |
| scifact | 300 | T — TypeSafe Jev 1.13 batch noul | 79.29% | 85.67% | 77.03% | 4044 ms |
| scifact | 300 | N — NVIDIA cross-encoder reranker | 78.70% | 87.33% | 76.16% | 307 ms |
| xquad-en | 1190 | A — no reranker (hybrid order) | 98.11% | 99.58% | 97.60% | 0 ms |
| xquad-en | 1190 | T — TypeSafe Jev 1.13 batch noul | 98.93% | 99.66% | 98.67% | 4000 ms |
| xquad-en | 1190 | N — NVIDIA cross-encoder reranker | 99.37% | 99.66% | 99.27% | 409 ms |
Probability calibration (candidate-level relevance)
| Dataset | Model | ECE (10 bin) | Brier | Top-1 accuracy | Top-1 confidence (correct) | Top-1 confidence (wrong) |
|---|---|---|---|---|---|---|
| scifact | TypeSafe Jev 1.13 batch noul | 0.0625 | 0.0366 | 71.00% | 0.812 | 0.549 |
| xquad-en | TypeSafe Jev 1.13 batch noul | 0.0133 | 0.0045 | 97.90% | 0.960 | 0.740 |
Frozen-context answer generation
| Dataset | Generator | Token F1 | Exact match | F1 >= 0.5 | Abstention | Valid citations | |
|---|---|---|---|---|---|---|---|
| xquad-en | T | diffusiongemma-26b | 30.58% | 1.26% | 17.31% | 2.61% | 99.85% |
RAG optimization mode (confidence-partitioned Jev, fixed t = 0.50)
| Dataset | Model | Threshold | A baseline nDCG@10 | Always-on nDCG@10 | Partitioned nDCG@10 | Delta vs baseline | 95% CI |
|---|---|---|---|---|---|---|---|
| scifact | TypeSafe Jev 1.13 batch noul | 0.50 | 71.67% | 79.29% | 75.81% | +4.14 pts | +2.13 to +6.29 |
| xquad-en | TypeSafe Jev 1.13 batch noul | 0.50 | 98.11% | 98.93% | 99.00% | +0.89 pts | +0.45 to +1.36 |