Run2 report
| split |
method |
condition |
Hit@3 (95% CI over seeds) |
MRR@10 |
nDCG@10 |
Δ Hit@3 vs none [95% CI] |
Δ Hit@3 vs shuffled [95% CI] |
| within |
none |
none |
0.382 [0.353, 0.411] |
0.334 |
0.384 |
|
|
| within |
contextual |
genuine |
0.383 [0.350, 0.415] |
0.338 |
0.387 |
+0.001 [-0.003, +0.004] |
-0.001 [-0.005, +0.004] |
| within |
contextual |
shuffled |
0.383 [0.350, 0.416] |
0.332 |
0.383 |
+0.001 [-0.004, +0.006] |
|
| within |
contextual |
noisy |
0.384 [0.353, 0.415] |
0.336 |
0.385 |
+0.002 [-0.003, +0.006] |
+0.001 [-0.004, +0.005] |
| within |
contextual+ts |
genuine |
0.389 [0.362, 0.415] |
0.335 |
0.385 |
+0.007 [+0.002, +0.012] |
+0.004 [-0.001, +0.009] |
| within |
contextual+ts |
shuffled |
0.384 [0.356, 0.413] |
0.334 |
0.384 |
+0.002 [-0.002, +0.007] |
|
| within |
contextual+ts |
noisy |
0.384 [0.360, 0.409] |
0.335 |
0.385 |
+0.002 [-0.002, +0.006] |
+0.000 [-0.005, +0.005] |
| within |
global |
genuine |
0.323 [0.288, 0.357] |
0.273 |
0.332 |
-0.059 [-0.079, -0.039] |
+0.015 [-0.006, +0.035] |
| within |
global |
shuffled |
0.309 [0.290, 0.327] |
0.257 |
0.317 |
-0.074 [-0.093, -0.054] |
|
| within |
global |
noisy |
0.313 [0.265, 0.362] |
0.264 |
0.324 |
-0.069 [-0.089, -0.050] |
+0.004 [-0.016, +0.025] |
| within |
ranker |
genuine |
0.422 [0.394, 0.450] |
0.365 |
0.411 |
+0.040 [+0.025, +0.056] |
+0.038 [+0.023, +0.054] |
| within |
ranker |
shuffled |
0.384 [0.356, 0.413] |
0.334 |
0.384 |
+0.002 [-0.002, +0.007] |
|
| within |
ranker |
noisy |
0.390 [0.344, 0.436] |
0.341 |
0.389 |
+0.008 [-0.002, +0.017] |
+0.005 [-0.004, +0.016] |
| test |
none |
none |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
|
|
| test |
contextual |
genuine |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
contextual |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
contextual |
noisy |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
contextual+ts |
genuine |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
contextual+ts |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
contextual+ts |
noisy |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
global |
genuine |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
global |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
global |
noisy |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
ranker |
genuine |
0.414 [0.387, 0.442] |
0.361 |
0.404 |
+0.028 [+0.019, +0.039] |
+0.028 [+0.019, +0.039] |
| test |
ranker |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
ranker |
noisy |
0.397 [0.374, 0.420] |
0.348 |
0.389 |
+0.011 [+0.005, +0.017] |
+0.011 [+0.005, +0.017] |
| method |
feedback phase (s, mean) |
eval search p50 (ms) |
| none |
0 |
10.3 |
| global |
3.4 |
2.8 |
| contextual |
3.4 |
2.7 |
| contextual+ts |
3.5 |
2.8 |
| ranker |
3.6 |
2.2 |