pub enum RefineKeep {
Last,
Best,
}Expand description
Which state of a refinement walk becomes the injected child.
Selectable because the choice is an empirical claim, and the same rule
applies here as to Acquisition: a rule chosen on evidence should stay
re-checkable, and a rule rejected on evidence doubly so. make climb and
search_health --budget-ab are where the comparison runs.
§Why this is a question at all
A refinement walk renders and featurizes ~40 candidates and injects one.
Which one is free to choose — the whole walk is already in the memo, and
every trace the kernel returns already carries its own log π_β — so the
choice costs nothing either way and has never been measured.
The tension is real in both directions. Self::Last is a draw from where
the chain ended up, which respects the target’s own weighting and is
robust: it cannot be fooled by a single point where the surrogate happens
to be over-optimistic. Self::Best takes the walk’s argmax, which is what
a shortlist wants — the pool is not a sample, it is a few dozen patches a
person will listen to — but argmax over a surrogate is the classic way to
find that surrogate’s errors rather than the user’s preferences.
§The A/B, run, and its result — a tie
make climb SEEDS=16 on both arms, same seed list, so the per-seed lines
pair directly:
Last Best
mean gain +1.927 ± 0.452 +1.774 ± 0.302
median gain +2.058 +1.819
10% trimmed +1.840 ± 0.383 +1.925 ± 0.190
climbed on 14/16 15/16
paired (Best − Last) mean −0.153 ± 0.384 (−0.40 se)
median −0.185
trimmed −0.113 ± 0.318
sign 8 better / 8 worse, p = 1.000Eight and eight is as exact a tie as sixteen seeds can produce. The
difference does not clear zero at 2 se on any of the three statistics, so
the default stays Self::Last — kept re-checkable rather than
deleted, the same way Acquisition::Thompson is kept after losing.
Two things worth reading off it rather than leaving in the table:
- The feared failure did not happen, and neither did the hoped-for win.
The worry was that argmax over a surrogate would find the surrogate’s
errors and deepen the catastrophic tail. Across the pair the tails are a
wash — the worst
Lastseed goes −0.74 → −1.80 underBest, and the next two worst go −0.64 → +0.52 and +0.12 → +1.29.Bestclimbs on one more seed and means marginally less. Bestis the lower-variance rule, not the better one. Its trimmed standard error is halfLast’s (0.190 against 0.383). Injecting the walk’s argmax is more consistent than injecting where it stopped; it just does not aim anywhere better on average. That is a coherent thing for argmax-over-a-noisy-surrogate to be, and it is the argument to re-run this on if the surrogate ever gets sharper.
Variants§
Last
Inject the state the walk ended on. The shipped behaviour, and the default — the A/B above ran and tied, so nothing moved it.
Best
Inject the highest-log π_β state the walk occupied, seed included —
so a walk that found nothing better than its seed injects nothing.
Trait Implementations§
Source§impl Clone for RefineKeep
impl Clone for RefineKeep
Source§fn clone(&self) -> RefineKeep
fn clone(&self) -> RefineKeep
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreSource§impl Debug for RefineKeep
impl Debug for RefineKeep
Source§impl Default for RefineKeep
impl Default for RefineKeep
Source§fn default() -> RefineKeep
fn default() -> RefineKeep
Source§impl PartialEq for RefineKeep
impl PartialEq for RefineKeep
Source§fn eq(&self, other: &RefineKeep) -> bool
fn eq(&self, other: &RefineKeep) -> bool
self and other values to be equal, and is used by ==.