Run1 report
| split |
method |
condition |
Hit@3 (95% CI over seeds) |
MRR@10 |
nDCG@10 |
Δ Hit@3 vs none [95% CI] |
Δ Hit@3 vs shuffled [95% CI] |
| within |
none |
none |
0.382 [0.353, 0.411] |
0.334 |
0.384 |
|
|
| within |
contextual |
genuine |
0.383 [0.350, 0.415] |
0.338 |
0.387 |
+0.001 [-0.003, +0.004] |
-0.001 [-0.005, +0.004] |
| within |
contextual |
shuffled |
0.383 [0.350, 0.416] |
0.332 |
0.383 |
+0.001 [-0.004, +0.006] |
|
| within |
contextual |
noisy |
0.384 [0.353, 0.415] |
0.336 |
0.385 |
+0.002 [-0.003, +0.006] |
+0.001 [-0.004, +0.005] |
| within |
contextual+ts |
genuine |
0.389 [0.362, 0.415] |
0.335 |
0.385 |
+0.007 [+0.002, +0.012] |
+0.004 [-0.001, +0.009] |
| within |
contextual+ts |
shuffled |
0.384 [0.356, 0.413] |
0.334 |
0.384 |
+0.002 [-0.002, +0.007] |
|
| within |
contextual+ts |
noisy |
0.384 [0.360, 0.409] |
0.335 |
0.385 |
+0.002 [-0.002, +0.006] |
+0.000 [-0.005, +0.005] |
| within |
global |
genuine |
0.323 [0.288, 0.357] |
0.273 |
0.332 |
-0.059 [-0.079, -0.039] |
+0.015 [-0.006, +0.035] |
| within |
global |
shuffled |
0.309 [0.290, 0.327] |
0.257 |
0.317 |
-0.074 [-0.093, -0.054] |
|
| within |
global |
noisy |
0.313 [0.265, 0.362] |
0.264 |
0.324 |
-0.069 [-0.089, -0.050] |
+0.004 [-0.016, +0.025] |
| within |
ranker |
genuine |
0.421 [0.394, 0.448] |
0.363 |
0.409 |
+0.039 [+0.024, +0.055] |
+0.141 [+0.120, +0.162] |
| within |
ranker |
shuffled |
0.280 [0.244, 0.315] |
0.234 |
0.293 |
-0.102 [-0.121, -0.083] |
|
| within |
ranker |
noisy |
0.330 [0.232, 0.428] |
0.287 |
0.342 |
-0.055 [-0.073, -0.036] |
+0.047 [+0.025, +0.069] |
| test |
none |
none |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
|
|
| test |
contextual |
genuine |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
contextual |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
contextual |
noisy |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
contextual+ts |
genuine |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
contextual+ts |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
contextual+ts |
noisy |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
global |
genuine |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
global |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
global |
noisy |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
+0.000 [+0.000, +0.000] |
| test |
ranker |
genuine |
0.414 [0.386, 0.442] |
0.361 |
0.404 |
+0.028 [+0.018, +0.038] |
+0.028 [+0.018, +0.038] |
| test |
ranker |
shuffled |
0.386 [0.361, 0.411] |
0.334 |
0.378 |
+0.000 [+0.000, +0.000] |
|
| test |
ranker |
noisy |
0.397 [0.374, 0.420] |
0.348 |
0.389 |
+0.011 [+0.005, +0.017] |
+0.011 [+0.005, +0.017] |
| method |
feedback phase (s, mean) |
eval search p50 (ms) |
| none |
0 |
10.3 |
| global |
3.4 |
2.8 |
| contextual |
3.5 |
2.8 |
| contextual+ts |
3.5 |
2.7 |
| ranker |
3.7 |
2.1 |