Skip to main content

auracle_session/
engine.rs

1//! The two-loop session engine (the reference: *The two loops*).
2//!
3//! - **Patch loop** (machine-paced, silent): fill a pool with vetted prior
4//!   draws; once a posterior exists, *refine* — warm-start fugue-evo's typed
5//!   MH from the best pool members on the Boltzmann target
6//!   `π_β ∝ p_grammar · exp(β·E[u(x)])` and inject improved candidates. The
7//!   refinement run is a **short local MH walk** from each seed (a dozen
8//!   steps, final state kept), not a draw from `π_β`: it moves candidates
9//!   uphill on that target, which is what the pool needs, but nothing here
10//!   claims the pool is distributed as `π_β`.
11//! - **Taste loop** (human-paced, persistent): feedback events append to the
12//!   [`ObservationLog`] as **raw** φ; the posterior is re-fit from the log,
13//!   standardizing at fit time.
14//!
15//! Between them, **acquisition**: [`Engine::next_duel`] maximizes expected
16//! information about θ (BALD). See its docs for why the obvious alternative —
17//! dueling Thompson sampling — is the wrong objective for this product.
18//!
19//! **Locks** (partial evolution): any set of trace addresses can be frozen
20//! during refinement. The MH kernel still proposes over all sites; a proposal
21//! that touches a locked address is rejected outside the kernel. Because the
22//! underlying kernel satisfies detailed balance on the full space, rejecting
23//! locked-coordinate moves yields a valid Metropolis-within-Gibbs sampler on
24//! the *conditional* posterior given the locked values — locking is exact,
25//! not a heuristic. That exactness depends on the rejection region being
26//! *symmetric*: [`Engine::violates_locks`] therefore checks births as well as
27//! deaths and edits. Wasted proposals are compensated by scaling step counts.
28//!
29//! All UI modes are emitters into the same observation stream: the engine
30//! does not know which surface produced an event. Candidates carry stable
31//! `id`s — pool positions shift on eviction, ids never do.
32
33use std::collections::{HashMap, HashSet, VecDeque};
34use std::sync::Arc;
35
36use auracle_features::{
37    featurize_memo, render_playback, Audition, Features, PhraseSpec, RenderMemo,
38};
39use auracle_grammar::prior::N_OPS;
40use auracle_grammar::{tree_diff, DiffEntry, PatchGrammarPrior, PatchTree};
41use auracle_taste::{
42    Feedback, FitSet, Observation, ObservationLog, Provenance, Standardizer, TasteConfig,
43    TasteModel, TastePosterior,
44};
45use fugue::Trace;
46use fugue_evo::inference::mh::EvolutionChain;
47use fugue_evo::inference::model::EvolutionModel;
48use rand::rngs::StdRng;
49use rand::{Rng, SeedableRng};
50use serde::{Deserialize, Serialize};
51
52use crate::calib::{calibration, Calibration, Forecast};
53use crate::farm::{draw_seed, Draw, PreFeaturized};
54use crate::naming::{claim_name, NameScale};
55use crate::surrogate::SurrogateFitness;
56
57/// Ceiling on the step-count compensation for locked sites — the most a locked
58/// refinement walk may cost relative to an unlocked one.
59///
60/// 4× fully compensates a walk with three quarters of its sites pinned, which is
61/// already a heavier lock than the hand-build → pin → breed loop produces. Past
62/// that the walk is deliberately under-compensated, because `⚡ evolve from
63/// this` is a button press with a person waiting behind it and a 90%-locked
64/// patch would otherwise ask for ten times the budget. See the note at the use
65/// site in `refine_one` for what that costs.
66const LOCK_SCALE_CAP: f64 = 4.0;
67
68/// The φ coordinate names, as owned strings (what the log records).
69pub fn phi_names() -> Vec<String> {
70    Features::phi_names()
71        .into_iter()
72        .map(|n| n.to_string())
73        .collect()
74}
75
76/// Which rule picks the next duel.
77///
78/// Selectable because the choice is an empirical claim, and
79/// `learn_synthetic --compare` measures it. Both alternatives are kept so
80/// that comparison stays runnable — a rule chosen on evidence should stay
81/// re-checkable, and a rule rejected on evidence doubly so.
82///
83/// ## The measurement, and what it is a measurement *of*
84///
85/// `cargo run -p auracle-session --example learn_synthetic --release --
86/// --compare 20`, on the synthetic user: 20 seeds, 72 duels, refit every 12.
87/// **Common random numbers** — pool fill, the user's coin flip at duel *t*,
88/// MCMC seed at round *r*, and refinement seeds are all shared across arms,
89/// so only the acquisition draw differs. Both regimes are graded on one fixed
90/// held-out exam under a single reference scale, so arms that built different
91/// pools are still answering the same questions. `±` is two standard errors
92/// of the paired difference.
93///
94/// ### Static pool (i.i.d. prior draws, `refine_steps: 0`)
95///
96/// | | cos θ\* ↑ | rank r ↑ | excess nats ↓ |
97/// |---|---|---|---|
98/// | **random** | 0.460 | 0.731 | 0.211 |
99/// | thompson | 0.416 | 0.628 | 0.254 |
100/// | bald | 0.484 | 0.762 | 0.199 |
101/// | bald − thompson | **+0.068 ± 0.062** | **+0.134 ± 0.044** | **−0.055 ± 0.014** |
102/// | bald − random | +0.025 ± 0.058 | +0.031 ± 0.046 | −0.012 ± 0.013 |
103///
104/// Dueling Thompson sampling is the one clear loser, at t = 2.2 / 6.1 / −8.0.
105/// It is a best-arm rule: it converges on identifying the top patch, which is
106/// not what a duel is for here. BALD and uniform pairing are inside two
107/// standard errors of each other on every metric.
108///
109/// A static i.i.d. pool is also a weak regime to conclude from on its own:
110/// prior draws are spread over feature space *by construction*, which is
111/// exactly where uniform pairs already achieve near-optimal `‖φ_a − φ_b‖`
112/// coverage and an information-seeking rule has no redundancy to prune. The
113/// concern was that the shipped pool is not that pool — refinement injects
114/// children near the current best and `insert_candidate` evicts the worst —
115/// so `--compare` runs an **evolving** regime too, with real refinement
116/// between rounds (the `Regime` type in `learn_synthetic.rs` documents the
117/// design).
118///
119/// ### Evolving pool (`refine_steps: 12`, refinement between rounds)
120///
121/// | | cos θ\* ↑ | rank r ↑ | excess nats ↓ |
122/// |---|---|---|---|
123/// | **random** | 0.479 | 0.694 | 0.232 |
124/// | thompson | 0.459 | 0.583 | 0.276 |
125/// | bald | 0.465 | 0.707 | 0.232 |
126/// | bald − thompson | +0.006 ± 0.068 | **+0.124 ± 0.066** | **−0.044 ± 0.017** |
127/// | bald − random | −0.015 ± 0.055 | +0.013 ± 0.048 | −0.000 ± 0.014 |
128///
129/// Same answer: Thompson loses, BALD and uniform pairing tie on every metric.
130///
131/// The run's manipulation check is itself a finding. Final pool spread (mean
132/// pairwise `‖Δφ‖`, reference scale) was **7.7–7.9 evolving vs 7.2 static**:
133/// six generations over a 72-duel session did not concentrate the pool at
134/// all — frontier-biased injection plus worst-eviction *widened* it slightly,
135/// because mutation pushes children into feature-space extremes faster than
136/// eviction trims them. So the concentrated regime BALD was hypothesized to
137/// win never arises at session horizon, and the tie is not an artifact of a
138/// spread pool that only the static setup guaranteed — the product's own
139/// dynamics keep the pool spread.
140///
141/// ## Why `Random` is the default
142///
143/// Measured in both the regime the product starts in and the regime it
144/// evolves into, uniform pairing is indistinguishable from BALD — and a rule
145/// with four tuning constants that ties a rule with none should not ship on
146/// a tie. Two supporting justifications survived checking, one did not: the
147/// `info_gain` BALD reports had **zero** consumers in the frontend, and BALD's
148/// repeat avoidance, while real, is barely needed over a 48-candidate pool
149/// that uniform pairing already samples without repeating (measured in
150/// `duels_spread_over_candidates_not_just_pairs`). `Random` also makes
151/// **every** duel an unbiased calibration sample rather than one in ten —
152/// a virtue that holds regardless of which rule learns θ faster.
153///
154/// One earlier justification was retracted for a bad reason, and the record
155/// should say so. The "pool grows and concentrates" argument was dismissed on
156/// the grounds that `insert_candidate` caps the pool — but a capped *size* is
157/// not an unchanging *spread*, and evicting the worst member could in
158/// principle concentrate a pool. Dismissing the concentration argument
159/// *because it was unmeasured*, while treating a measurement from the other
160/// regime as decisive, had the burden of proof backwards. The evolving run
161/// above is that measurement; it happens to show the concentration never
162/// materializes, but the default rests on the measured tie, not on the
163/// dismissal.
164///
165/// ## What `Bald` is still for
166///
167/// It is not dead code and it is not a fallback. It decisively beats the
168/// best-arm rule, so it is the right thing to reach for if acquisition ever
169/// needs to *do* something uniform pairing cannot: bias duels toward patches
170/// the user will enjoy auditioning ([`SessionConfig::duel_utility_weight`]),
171/// bound how often one patch reappears ([`SessionConfig::duel_exposure_penalty`]),
172/// or report why a question was asked. Those levers exist and are measured;
173/// none of them is currently worth the tie.
174///
175/// ## A correction worth recording
176///
177/// An earlier version of this rule scored its enjoyment term on *unnormalized*
178/// utility and used an *absolute* softmax temperature of 0.05 nats. Both are
179/// scale bets, and both lost: the enjoyment term grew without bound as the
180/// posterior sharpened, and `exp(ΔJ/T)` ran to `e¹⁰`, so the "softmax" was an
181/// argmax. That version was measurably *worse* than random, and it is the
182/// version an independent replication measured. It is also what produced the
183/// duel repetition seen in the running app — the same defect, observed from
184/// two directions. Fixed, BALD ties random; the numbers above are the fixed
185/// rule.
186#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
187pub enum Acquisition {
188    /// Uniformly random pairs. The default — see the type doc.
189    #[default]
190    Random,
191    /// Dueling Thompson sampling: two posterior draws, duel their champions.
192    /// Best-arm identification — converges on the top patch, not on θ.
193    Thompson,
194    /// Expected information gain about θ, plus an enjoyment term and a
195    /// repeat penalty, sampled from a softmax. Beats [`Acquisition::Thompson`]
196    /// decisively and ties [`Acquisition::Random`]; see the type doc.
197    Bald,
198}
199
200/// Which state of a refinement walk becomes the injected child.
201///
202/// Selectable because the choice is an empirical claim, and the same rule
203/// applies here as to [`Acquisition`]: a rule chosen on evidence should stay
204/// re-checkable, and a rule rejected on evidence doubly so. `make climb` and
205/// `search_health --budget-ab` are where the comparison runs.
206///
207/// ## Why this is a question at all
208///
209/// A refinement walk renders and featurizes ~40 candidates and injects **one**.
210/// Which one is free to choose — the whole walk is already in the memo, and
211/// every trace the kernel returns already carries its own `log π_β` — so the
212/// choice costs nothing either way and has never been measured.
213///
214/// The tension is real in both directions. [`Self::Last`] is a draw from where
215/// the chain ended up, which respects the target's own weighting and is
216/// robust: it cannot be fooled by a single point where the surrogate happens
217/// to be over-optimistic. [`Self::Best`] takes the walk's argmax, which is what
218/// a *shortlist* wants — the pool is not a sample, it is a few dozen patches a
219/// person will listen to — but argmax over a surrogate is the classic way to
220/// find that surrogate's errors rather than the user's preferences.
221///
222/// ## The A/B, run, and its result — a tie
223///
224/// `make climb SEEDS=16` on both arms, same seed list, so the per-seed lines
225/// pair directly:
226///
227/// ```text
228///                        Last              Best
229/// mean gain        +1.927 ± 0.452    +1.774 ± 0.302
230/// median gain      +2.058            +1.819
231/// 10% trimmed      +1.840 ± 0.383    +1.925 ± 0.190
232/// climbed on       14/16             15/16
233///
234/// paired (Best − Last)   mean    −0.153 ± 0.384   (−0.40 se)
235///                        median  −0.185
236///                        trimmed −0.113 ± 0.318
237///                        sign     8 better / 8 worse, p = 1.000
238/// ```
239///
240/// Eight and eight is as exact a tie as sixteen seeds can produce. The
241/// difference does not clear zero at 2 se on any of the three statistics, so
242/// **the default stays [`Self::Last`]** — kept re-checkable rather than
243/// deleted, the same way [`Acquisition::Thompson`] is kept after losing.
244///
245/// Two things worth reading off it rather than leaving in the table:
246///
247/// - **The feared failure did not happen, and neither did the hoped-for win.**
248///   The worry was that argmax over a surrogate would find the surrogate's
249///   errors and deepen the catastrophic tail. Across the pair the tails are a
250///   wash — the worst `Last` seed goes −0.74 → −1.80 under `Best`, and the
251///   next two worst go −0.64 → +0.52 and +0.12 → +1.29. `Best` climbs on one
252///   more seed and means marginally less.
253/// - **`Best` is the lower-variance rule, not the better one.** Its trimmed
254///   standard error is half `Last`'s (0.190 against 0.383). Injecting the
255///   walk's argmax is more *consistent* than injecting where it stopped; it
256///   just does not aim anywhere better on average. That is a coherent thing
257///   for argmax-over-a-noisy-surrogate to be, and it is the argument to
258///   re-run this on if the surrogate ever gets sharper.
259#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
260pub enum RefineKeep {
261    /// Inject the state the walk ended on. The shipped behaviour, and the
262    /// default — the A/B above ran and tied, so nothing moved it.
263    #[default]
264    Last,
265    /// Inject the highest-`log π_β` state the walk occupied, seed included —
266    /// so a walk that found nothing better than its seed injects nothing.
267    Best,
268}
269
270/// What the pool does with audition audio.
271///
272/// Renders are the engine's only expensive artifact and its bulkiest one: at
273/// the default phrase a single audition buffer is ~565 KB of f32, and a full
274/// pool of them is tens of megabytes of wasm heap — resident forever, for
275/// audio the user will mostly never ask to hear. But φ, not audio, is what
276/// the pool exists to hold, and [`auracle_features::render_playback`] can
277/// reproduce any buffer bit-identically from the term. So retention is a
278/// policy, not a structural requirement.
279#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
280pub enum RenderPolicy {
281    /// Materialize every candidate's buffer at admission and keep it for the
282    /// lifetime of the candidate. Fastest audition, largest footprint.
283    Eager,
284    /// Materialize on [`Engine::render_of`], keeping the most recently
285    /// auditioned [`SessionConfig::audio_cache`] buffers and dropping the
286    /// rest. A cold audition costs one render.
287    Lazy,
288    /// Never keep audio. Headless callers (tests, `learn_synthetic`) never
289    /// audition anything, and this is what they should pay.
290    #[default]
291    None,
292}
293
294/// Engine configuration.
295#[derive(Clone, Debug)]
296pub struct SessionConfig {
297    /// Vetted candidates to maintain in the pool.
298    pub pool_size: usize,
299    /// Maximum prior draws attempted per `fill_pool` (vet failures burn
300    /// attempts).
301    pub max_draws: usize,
302    /// MH refinement steps per seed (scaled up when locks waste proposals).
303    ///
304    /// The default scales with [`N_OPS`]: a structural proposal picks a new
305    /// operator from a categorical that the v2 palette widened from six to
306    /// twenty, so a fixed budget would spend the same number of proposals
307    /// covering a far wider move set and land the pool's children in a
308    /// visibly thinner slice of it.
309    ///
310    /// ## The split is measured, not reasoned
311    ///
312    /// `2·N_OPS` steps from `N_OPS/2` seeds was an argument, and the argument
313    /// could have been wrong in either direction. `search_health --budget-ab`
314    /// exists to settle it; over 8 seeds, 6 generations, graded against the
315    /// synthetic user's true utility:
316    ///
317    /// | steps | seeds | proposals | mean u | max u | |
318    /// |---|---|---|---|---|---|
319    /// | 40 | 10 | 400 | **1.714** | **8.154** | shipped |
320    /// | 40 |  3 | 120 | 1.241 | 6.178 | same depth, fewer seeds |
321    /// | 66 |  3 | 198 | 0.774 | 6.281 | same total, fewer seeds |
322    /// | 20 | 20 | 400 | 0.568 | 6.790 | half depth, double breadth |
323    ///
324    /// The shipped split wins on both metrics, and it is a genuine optimum
325    /// rather than the top of a slope: moving off it in *either* direction is
326    /// worse. Two rows are worth more than the headline.
327    ///
328    /// **Depth from few seeds is actively harmful.** 66×3 runs 65% more
329    /// proposals than 40×3 and scores *lower* (0.774 against 1.241) — a long
330    /// chain from a bad starting point converges confidently on somewhere you
331    /// did not want to be, and the extra steps are what get it there.
332    ///
333    /// **Breadth is not free either.** 20×20 spends the shipped budget and is
334    /// the worst row of the four. Twenty steps is not enough for a chain to
335    /// leave its seed, so the generation is twenty barely-moved copies of the
336    /// current top — which is also why it has the second-best `max`: it
337    /// preserves the frontier by never straying from it.
338    ///
339    /// Re-run this before changing either number.
340    pub refine_steps: usize,
341    /// How many top candidates to refine from. Also scaled with [`N_OPS`] —
342    /// more seeds is more *starting points*, which is what actually buys
343    /// coverage of a wider palette, whereas more steps per seed buys depth
344    /// around one. See [`SessionConfig::refine_steps`] for the measurement
345    /// that fixes the ratio between them.
346    pub refine_seeds: usize,
347    /// Boltzmann sharpness β of the refinement target.
348    pub beta: f64,
349    /// Which state of a refinement walk becomes the injected child.
350    pub refine_keep: RefineKeep,
351    /// Maximum style components in the taste mixture
352    /// (max-of-linear-experts); the fitted K grows with evidence up to this
353    /// cap.
354    ///
355    /// K is also the fit's dominant cost driver, because single-site MH
356    /// rebuilds the whole program every step and the site count is
357    /// `d·K + n_sessions + 5` — at today's d = 40, that is 46 at K = 1 and
358    /// **206 at K = 5** (printed by `fit_bench`, so it moves with φ). Two
359    /// consequences, both measured by `auracle-taste/examples/fit_bench.rs`:
360    /// the fit is ~4× slower at the cap than at the first fit, and the step
361    /// budget is *fixed*, so a mature fit gets ~4× fewer sweeps per site than
362    /// an early one — growing K makes the fit both slower and statistically
363    /// thinner.
364    ///
365    /// **Open option, deliberately not taken here: cap this at 3** (sites
366    /// 206 → 126, a ~1.6× mature-fit win at no engineering cost). It is left
367    /// open because unlike the address hoist and the budget cut it is not a
368    /// pure efficiency change — it removes model *capacity*, and capacity is
369    /// the whole point of the mixture (a user with four islands of taste
370    /// cannot be represented by three lenses). Take it only on evidence:
371    /// [`TastePosterior::style_share`](auracle_taste::TastePosterior::style_share)
372    /// reports what fraction of the pool each lens claims, and if lenses 4
373    /// and 5 sit near zero share across real sessions they are paying 54
374    /// sites per step for nothing. `learn_synthetic --compare` is the A/B.
375    pub k_styles: usize,
376    /// The audition stimulus.
377    pub phrase: PhraseSpec,
378    /// How the pool retains audition audio.
379    pub render_policy: RenderPolicy,
380    /// Audition buffers kept resident under [`RenderPolicy::Lazy`], most
381    /// recently auditioned first. Sized for the current duel pair, the bench
382    /// subject, and enough recent history that stepping back through the bank
383    /// is free.
384    pub audio_cache: usize,
385    /// Post-warmup MH steps per posterior fit.
386    ///
387    /// This is the one knob in this struct that buys wall time with
388    /// *statistics*, so it is set from a measurement rather than a guess.
389    /// Only 500 draws survive thinning at any budget, so the budget does not
390    /// buy draws — it buys **sweeps per site**, and at K = 5 (206 sites) even
391    /// 10 000 steps is only ~49 sweeps.
392    ///
393    /// Recovery vs budget at the mature operating point (K = 5, n_obs = 100,
394    /// 12 seeds, `cargo run --release -p auracle-taste --example fit_bench
395    /// -- sweep 12`): held-out duel agreement with the noiseless ground-truth
396    /// ordering, and the cosine of the best lens against θ\*.
397    ///
398    /// | steps | held-out acc | best-lens cos | native fit |
399    /// |---|---|---|---|
400    /// | 30 000 | 0.767 | 0.724 | 1.79 s |
401    /// | 20 000 | 0.757 | 0.717 | 1.16 s |
402    /// | **10 000** | **0.746** | **0.686** | **0.60 s** |
403    /// | 8 000 | 0.738 | 0.690 | 0.49 s |
404    /// | 6 000 | 0.737 | 0.653 | 0.38 s |
405    /// | 5 000 | 0.729 | 0.655 | 0.33 s |
406    /// | 3 000 | 0.713 | 0.599 | 0.20 s |
407    ///
408    /// That curve is smooth, so it says where the trade *stops paying*. The
409    /// second instrument is the end-to-end M4 gate
410    /// (`closed_loop_learns_synthetic_taste`, which runs at exactly this
411    /// budget through the real render → vet → feature pipeline). One run of
412    /// it is a **single draw** — over the pool lottery, the duel answers and
413    /// the chain — so it is replicated over 13 seeds here (`cargo run
414    /// --release -p auracle-session --example closed_loop_sweep`). Its
415    /// pool/truth correlation `r` against the 0.6 gate, plus the other two
416    /// metrics the test asserts:
417    ///
418    /// | steps | mean r | min r | seeds with r ≤ 0.6 | mean top-5 | mean cos |
419    /// |---|---|---|---|---|---|
420    /// | 30 000 | 0.736 | 0.575 | 1/13 | 3.14 | 0.528 |
421    /// | 20 000 | 0.722 | 0.576 | 2/13 | 2.88 | 0.497 |
422    /// | **10 000** | **0.726** | **0.551** | **2/13** | **2.78** | **0.475** |
423    /// | 8 000 | 0.715 | 0.503 | 1/13 | 3.07 | 0.456 |
424    /// | 6 000 | 0.747 | 0.600 | 1/13 | 3.39 | 0.497 |
425    /// | 5 000 | 0.689 | 0.476 | 2/13 | 3.15 | 0.392 |
426    ///
427    /// Read that as a noisy measurement, because it is one. Within a single
428    /// budget the seed-to-seed spread of `r` is sd ≈ 0.07–0.10 over a range
429    /// of ≈ 0.25; between budgets from 6 000 up the means sit in
430    /// 0.715–0.747, i.e. inside one standard error (≈ 0.02) of each other —
431    /// and 6 000 posts the *highest* mean of the six, which is the plainest
432    /// sign that this instrument's ranking of the upper budgets is noise.
433    /// **From 6 000 to 30 000 it cannot tell them apart.** Only 5 000
434    /// separates at all — lowest on mean `r`, on min `r` and on cos — and
435    /// even that gap to 30 000 (0.047) is barely over one standard error of
436    /// the difference.
437    ///
438    /// So the argument for 10 000 is *not* that it passes where 5 000 fails.
439    /// Every budget here fails the 0.6 gate on some seed, including the old
440    /// 30 000 (1 of 13), and 5 000 clears it on 11 of 13. The argument is:
441    /// 10 000 is 3× cheaper than 30 000 and gives up 0.010 of mean `r`, which
442    /// is inside the noise; the `fit_bench` sweep above — 12 seeds on a
443    /// metric with far less variance — prices the same cut at 0.021 of
444    /// held-out accuracy and 0.038 of cos; and cutting further to 5 000 saves
445    /// only another 0.27 s per fit while costing 0.017 more held-out
446    /// accuracy, 0.031 more cos and 0.037 of mean `r`, the one budget *both*
447    /// instruments mark down. 10 000 is where the two instruments agree, not
448    /// where a threshold was crossed.
449    ///
450    /// (An earlier revision of this table read the M4 gate at a single seed,
451    /// `0xE05`, and concluded that 5 000 "fails outright" at r = 0.565 while
452    /// 10 000 held "the widest margin of any budget tried". Both are
453    /// artifacts of that one draw: 0xE05 sits ~1.2 sd low at 5 000 and right
454    /// on the mean at 10 000. The per-seed numbers reproduce exactly — the
455    /// inference from one of them did not.)
456    ///
457    /// The earlier 30 000 also predated the address hoist in
458    /// [`auracle_taste::model`], which made every step ~1.7× cheaper on its
459    /// own; the two together take a mature fit from ~1.86 s to ~0.60 s
460    /// natively (~13 s → ~4 s in the browser).
461    pub mcmc_samples: usize,
462    /// Warmup (adaptation) steps per fit, held at ~30 % of
463    /// [`Self::mcmc_samples`]. Warmup only tunes the per-site proposal
464    /// scales; it produces no draws, so it is pure overhead beyond the point
465    /// the scales converge.
466    pub mcmc_warmup: usize,
467    /// Recency half-life for the taste likelihood, in observations
468    /// (`None` = no forgetting). Tastes drift; old votes should fade.
469    pub recency_half_life: Option<f64>,
470    /// Strength of the taste→grammar proposal tilt (0 disables): structural
471    /// θ components multiply the grammar's kind weights by
472    /// `exp(η·θ)` during refinement.
473    pub proposal_tilt: f64,
474    /// λ in the duel objective: how much the *pleasantness* of a duel counts
475    /// against its informativeness, applied to **pool-standardized** utility.
476    /// The user's enjoyment is a resource too — two mud patches are a cheap
477    /// question and an expensive answer.
478    ///
479    /// Keep it small. Information gain is bounded by `ln 2 ≈ 0.693` nats, so
480    /// a λ near 0.3 lets the ±2σ enjoyment term swing the objective by ±0.6 —
481    /// as much as the entire information range — and the acquisition function
482    /// quietly reverts to "duel the two best patches", which is the best-arm
483    /// behaviour BALD was adopted to escape. Measured on the synthetic user
484    /// (`learn_synthetic --compare`), λ = 0.3 cost 0.15 of pool-ranking
485    /// correlation against λ = 0; 0.1 leaves it a tie-breaker.
486    pub duel_utility_weight: f64,
487    /// γ in the duel objective: penalty per previous showing of the same
488    /// pair. Without it the acquisition function re-asks its favourite
489    /// question until the next refit.
490    pub duel_repeat_penalty: f64,
491    /// Penalty per previous *appearance of either candidate*, regardless of
492    /// who it was paired against.
493    ///
494    /// The pair penalty alone does not stop degeneracy, and the shipped app
495    /// proved it: over twelve consecutive duels one candidate appeared in
496    /// six. Every pairing `#1 vs #7`, `#1 vs #15`, `#1 vs #22` is a *distinct*
497    /// pair and pays no pair penalty at all, while the enjoyment term keeps
498    /// nominating the highest-utility candidate. The user does not experience
499    /// "distinct pairs"; they experience hearing the same patch over and over.
500    /// This term is what makes the *candidate* budget finite.
501    pub duel_exposure_penalty: f64,
502    /// Softmax temperature over the duel objective, as a **fraction of the
503    /// objective's own spread** across the candidate pairs.
504    ///
505    /// Scale-free for the same reason the enjoyment term is standardized: an
506    /// absolute temperature is a bet on how far apart the scores happen to
507    /// be. Shipped at an absolute 0.05 nats it was a bad bet — the objective
508    /// spans several tenths of a nat once the enjoyment term is in it, so
509    /// `exp(ΔJ/T)` ran to `e¹⁰` and the "softmax" was an argmax with extra
510    /// steps. Expressed as a fraction of the observed SD, 0.6 means the same
511    /// softness whatever the spread.
512    pub duel_temperature: f64,
513    /// Show one uniformly-random "check" duel every N duels. An
514    /// information-seeking acquisition deliberately picks pairs near p = 0.5,
515    /// so calibration measured on acquisition-chosen duels is
516    /// selection-biased; these are the unbiased subsample.
517    ///
518    /// Redundant under [`Acquisition::Random`], where every duel is already
519    /// uniform and is tagged as a check — the setting is kept because it is
520    /// exactly what [`Acquisition::Bald`] would need, and because one in ten
521    /// was measured to be underpowered anyway (a few forecasts out of fifty
522    /// cannot fill a five-bin reliability diagram). 0 disables.
523    pub duel_check_every: usize,
524    /// Which rule picks the next duel.
525    pub acquisition: Acquisition,
526    /// Fold each new observation into the posterior weights by importance
527    /// sampling between full refits. Off makes the posterior frozen between
528    /// fits, which is what the A/B compares against.
529    pub sis_between_fits: bool,
530}
531
532impl Default for SessionConfig {
533    fn default() -> Self {
534        Self {
535            pool_size: 48,
536            max_draws: 400,
537            // 12 and 3 were tuned against the six-operator v1 palette; both
538            // ride N_OPS so the same tuning survives a palette change.
539            refine_steps: 2 * N_OPS,
540            refine_seeds: N_OPS.div_ceil(2),
541            beta: 2.0,
542            refine_keep: RefineKeep::default(),
543            k_styles: 5,
544            phrase: PhraseSpec::default(),
545            render_policy: RenderPolicy::None,
546            audio_cache: auracle_features::DEFAULT_AUDIO_CAP,
547            mcmc_samples: 10_000,
548            mcmc_warmup: 3_000,
549            recency_half_life: Some(150.0),
550            proposal_tilt: 0.6,
551            duel_utility_weight: 0.1,
552            duel_repeat_penalty: 0.5,
553            duel_exposure_penalty: 0.25,
554            duel_temperature: 0.6,
555            duel_check_every: 10,
556            acquisition: Acquisition::default(),
557            sis_between_fits: true,
558        }
559    }
560}
561
562/// Where a candidate came from.
563#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
564#[serde(rename_all = "snake_case")]
565pub enum Origin {
566    /// Drawn from the grammar prior.
567    Prior,
568    /// Produced by taste-guided MH refinement.
569    Refined,
570    /// Hand-edited on the panel and committed.
571    Edited,
572    /// Loaded from the built-in preset bank.
573    Preset,
574}
575
576/// Give a tree its node identities on the way into the pool.
577///
578/// The pool is where a term stops being a search intermediate and becomes a
579/// patch someone can open, lock, lay out and breed from, so it is exactly where
580/// identities are worth minting — and the only place. Search itself hands
581/// through thousands of anonymous trees per generation; a prior draw carries
582/// none, and a tree restored from a save written before uids existed carries
583/// none either, which is the whole of that migration: old saves deserialize
584/// with every `uid` defaulted to unset and are settled here on the way in.
585fn settled(mut tree: PatchTree) -> PatchTree {
586    tree.ensure_uids();
587    tree
588}
589
590/// A vetted pool member.
591pub struct Candidate {
592    /// Stable id (unique for the lifetime of the engine; survives pool
593    /// reordering and eviction of *other* members).
594    pub id: u64,
595    /// The term.
596    pub tree: PatchTree,
597    /// Its extracted features.
598    pub features: Features,
599    /// Standardized feature vector (empty until the standardizer exists).
600    pub phi_std: Vec<f64>,
601    /// Content address of this candidate's `(term, spec)` featurization.
602    /// Carried rather than recomputed because hashing the term is the one
603    /// thing every cache path needs and the term never changes.
604    pub key: String,
605    /// The audition buffer, when resident. Governed by
606    /// [`SessionConfig::render_policy`] — under [`RenderPolicy::Lazy`] this is
607    /// `None` until [`Engine::render_of`] materializes it, and may go back to
608    /// `None` when a newer audition evicts it. Never a signal that the
609    /// candidate is unplayable; ask [`Engine::render_of`] for that.
610    ///
611    /// Shared with the memo (and with whoever last asked for it) through an
612    /// [`Arc`] — one allocation per audition, however many holders it has.
613    pub render: Option<Arc<Audition>>,
614    /// Provenance.
615    pub origin: Origin,
616    /// User-given name (frontends fall back to `tree.signature()`).
617    pub name: Option<String>,
618    /// The user asked to keep this one: [`Engine::insert_candidate`] will never
619    /// evict it.
620    ///
621    /// Deliberately **not** derived from the star rating. A star is an
622    /// observation that enters the log and moves θ; if a rating also decided
623    /// what survives, users would rate strategically to protect patches, and
624    /// every protective over-rating is a preference they never held — under
625    /// exactly the pressure where they care most. So the two channels stay
626    /// separate: stars are what you think, pins are what you keep.
627    ///
628    /// Capped by [`Engine::pin_cap`]; see there for why the pool cannot be
629    /// pinned solid.
630    pub pinned: bool,
631}
632
633/// One recorded evolution/edit step, for the lineage display.
634#[derive(Clone, Debug, Serialize, Deserialize)]
635pub struct LineageEvent {
636    /// Generation counter at the time of the event (increments per
637    /// `refine`/`refine_from` call).
638    pub generation: usize,
639    /// `"refine"` or `"edit"`.
640    pub kind: String,
641    /// Parent candidate id.
642    pub parent_id: u64,
643    /// Child candidate id.
644    pub child_id: u64,
645    /// What changed, in trace-address terms.
646    pub diff: Vec<DiffEntry>,
647    /// Parent posterior-mean utility at event time (0 with no posterior).
648    pub parent_utility: f64,
649    /// Child posterior-mean utility at event time.
650    pub child_utility: f64,
651}
652
653/// A portable taste profile: the observation log **plus the standardizer its
654/// φ vectors were standardized under**. θ is only meaningful relative to its
655/// standardizer, so the two persist together.
656#[derive(Clone, Debug, Serialize, Deserialize)]
657pub struct Profile {
658    /// The observation log (source of truth).
659    pub log: ObservationLog,
660    /// The standardizer under which every φ in the log was recorded.
661    pub standardizer: Option<Standardizer>,
662}
663
664/// Tilt categorical proposal weights by taste: `w'_i ∝ w_i · exp(η·t_i)`,
665/// with each multiplier clamped to `[1/4, 4]` so no kind is ever starved or
666/// monopolized, and the result renormalized. Pure, so the taste→grammar
667/// mapping is testable without an MCMC fit.
668/// Shrink a posterior mean toward zero by its own uncertainty:
669/// `θ·|θ|/(|θ| + σ)`.
670///
671/// The factor is 1 when the coefficient is many standard deviations from
672/// zero, ½ when `σ = |θ|`, and →0 when the posterior is mostly prior. It is
673/// the same shape as a signal-to-noise weighting, chosen over a hard
674/// significance cut because a cut makes the proposal distribution jump
675/// discontinuously as evidence accumulates, and users hear that as the
676/// instrument changing its mind.
677fn shrink(mean: f64, std: f64) -> f64 {
678    let m = mean.abs();
679    if m <= 0.0 {
680        return 0.0;
681    }
682    mean * m / (m + std.max(0.0))
683}
684
685pub fn tilt_weights(base: &[f64], tilts: &[f64], eta: f64) -> Vec<f64> {
686    let mut out: Vec<f64> = base
687        .iter()
688        .zip(tilts)
689        .map(|(w, t)| w * (eta * t).exp().clamp(0.25, 4.0))
690        .collect();
691    let sum: f64 = out.iter().sum();
692    if sum > 0.0 {
693        for w in &mut out {
694            *w /= sum;
695        }
696    }
697    out
698}
699
700/// One bank entry of a saved session (renders and features are re-derived
701/// on import — trees are the source of truth).
702#[derive(Clone, Debug, Serialize, Deserialize)]
703pub struct BankEntry {
704    /// The candidate's stable id (preserved so lineage references stay
705    /// meaningful).
706    pub id: u64,
707    /// The patch term.
708    pub tree: PatchTree,
709    /// Provenance.
710    pub origin: Origin,
711    /// User-given name.
712    pub name: Option<String>,
713    /// Whether the user pinned this patch against eviction.
714    ///
715    /// `#[serde(default)]` is what makes this change safe for sessions saved
716    /// before pins existed: the record is one IndexedDB key with no schema
717    /// version, so compatibility has to be by construction. An old session
718    /// loads with nothing pinned, which is exactly what it meant.
719    #[serde(default)]
720    pub pinned: bool,
721}
722
723/// An implicit preference signal, logged but (for now) not modeled: promote
724/// events, hand-edit commits, per-patch play counts. Un-logged signal is
725/// gone forever; modeling can come later.
726///
727/// The three optional fields carry the editor's stream (WS-8 §3). The single
728/// most informative row in it is a **revert**: the player made an edit, heard
729/// it, sat with it for a few seconds, and took it back. That is a preference
730/// statement about a pair of patches neither of which is in the bank, at edit
731/// granularity — far denser than the duel stream and the natural training set
732/// for an edit-level model. It is unbuildable without a year of this log, and
733/// the log is unbuildable retroactively, which is the whole argument for
734/// writing it before anything reads it.
735///
736/// Deliberately **not** in the likelihood, exactly as the play counts already
737/// here are not: a revert is confounded with curiosity, and the honest place
738/// for it is a v2 fit that can be validated, not a silent term in the model
739/// the player is being shown a number from.
740#[derive(Clone, Debug, Serialize, Deserialize)]
741pub struct ImplicitEvent {
742    /// `"promote"`, `"play"`, `"edit"`, `"revert"`, `"commit"`, …
743    pub kind: String,
744    /// Candidate id the event is about (0 when it is about the bench, which
745    /// is not a candidate until it is committed).
746    pub id: u64,
747    /// Magnitude (play counts, dwell in ms, 1 for point events).
748    pub value: f64,
749    /// Session index when it happened.
750    pub session: usize,
751    /// Free-form JSON detail: the `StructOp` and module kind for an edit, the
752    /// query string for a link-drag search, the outcome of a commit. A string
753    /// rather than a typed field per event kind, because the point of this log
754    /// is to be *written* now and interpreted later — a schema fixed today is
755    /// a schema that stops the next event kind from being logged at all.
756    #[serde(default, skip_serializing_if = "String::is_empty")]
757    pub detail: String,
758    /// Raw φ before the event, where the event is a transition (revert).
759    #[serde(default, skip_serializing_if = "Vec::is_empty")]
760    pub phi_before: Vec<f64>,
761    /// Raw φ after it.
762    #[serde(default, skip_serializing_if = "Vec::is_empty")]
763    pub phi_after: Vec<f64>,
764}
765
766/// What one posterior fit claimed for each style lens.
767///
768/// Recorded per fit and persisted, because the question it answers is about
769/// **real sessions over time** and cannot be answered from one of them.
770/// [`SessionConfig::k_styles`] documents an option that is deliberately not
771/// taken — cap K at 3, taking the fit from 206 sites to 126 for a ~1.6×
772/// mature-fit win — and gates it explicitly on whether lenses 4 and 5 sit near
773/// zero share across real sessions. Nothing collected that, so the decision
774/// could not be made either way; this is the collection.
775///
776/// It is deliberately not a judgment. A row says what the shares *were* at a
777/// given evidence count, and how many lenses the fit was even allowed (`k`
778/// grows with the log and is capped by config, so an early row with two lenses
779/// is not evidence that lenses 3–5 are idle — it is evidence they did not
780/// exist yet). Reading rows where `k == k_styles` is what the option needs.
781#[derive(Clone, Debug, Serialize, Deserialize)]
782pub struct StyleShareRecord {
783    /// Observations in the log at the time of the fit.
784    pub observations: usize,
785    /// Lenses this fit was allowed — `min(1 + log/20, k_styles)`.
786    pub k: usize,
787    /// Share of the pool claimed by each lens, aligned, summing to ~1.
788    pub shares: Vec<f64>,
789}
790
791/// A full saved session: everything needed to restore the app across a
792/// reload — the portable profile plus the bank and its history.
793#[derive(Clone, Debug, Serialize, Deserialize)]
794pub struct SessionState {
795    /// Log + standardizer.
796    pub profile: Profile,
797    /// The patch bank (trees, origins, names).
798    pub bank: Vec<BankEntry>,
799    /// Evolution/edit history.
800    pub lineage: Vec<LineageEvent>,
801    /// Generation counter.
802    pub generation: usize,
803    /// User-given style names (index = aligned style index).
804    #[serde(default)]
805    pub style_names: Vec<String>,
806    /// Implicit preference events.
807    #[serde(default)]
808    pub events: Vec<ImplicitEvent>,
809    /// Out-of-sample duel forecasts (calibration survives a reload).
810    #[serde(default)]
811    pub forecasts: Vec<Forecast>,
812    /// Per-fit style shares — the evidence [`SessionConfig::k_styles`]' open
813    /// option is gated on.
814    #[serde(default)]
815    pub style_shares: Vec<StyleShareRecord>,
816}
817
818/// A chosen duel, with the reasoning that produced it.
819#[derive(Clone, Copy, Debug, Serialize, Deserialize)]
820pub struct DuelChoice {
821    /// Pool index of candidate A.
822    pub a: usize,
823    /// Pool index of candidate B.
824    pub b: usize,
825    /// Expected information gain about θ, in nats (0 for random pairs).
826    /// Bounded above by `ln 2 ≈ 0.693`, the entropy of a coin flip.
827    pub info_gain: f64,
828    /// True when this pair was drawn uniformly at random as a calibration
829    /// check rather than chosen by the acquisition function.
830    pub random_check: bool,
831    /// `"random"` (no posterior), `"check"`, or `"bald"`.
832    pub method: &'static str,
833}
834
835/// What a hand edit's commit reported about the edit against the original.
836///
837/// The type exists because the old `as_improvement: bool` could not say the
838/// most informative thing a player can say. "I edited this, listened to both,
839/// and the original was better" is a duel with a known answer — and under a
840/// boolean it was **unrepresentable**: `false` meant "said nothing", so the
841/// loss was silently discarded and the log only ever saw edits that won. A
842/// preference log that records only successes is a biased sample of exactly
843/// the kind the model has no defence against, and hand editing is the richest
844/// signal in the app.
845#[derive(Clone, Copy, Debug, PartialEq, Eq)]
846pub enum EditOutcome {
847    /// Nothing was claimed. Lineage still links the two; no observation.
848    Untold,
849    /// The player heard both and picked. `edited_won: false` is the losing
850    /// direction — the half that used to be inexpressible.
851    Heard {
852        /// True when the edit beat the original.
853        edited_won: bool,
854    },
855    /// The player asserted the edit is better without hearing them back to
856    /// back (the express "my edit is better" checkbox). Same claim, weaker
857    /// evidence, tagged so it can be scored separately.
858    SelfReported,
859}
860
861impl EditOutcome {
862    /// `(edited_won, provenance)` when this outcome makes a claim.
863    pub fn told(&self) -> Option<(bool, Provenance)> {
864        match self {
865            EditOutcome::Untold => None,
866            EditOutcome::Heard { edited_won } => Some((*edited_won, Provenance::HeardEdit)),
867            EditOutcome::SelfReported => Some((true, Provenance::SelfReport)),
868        }
869    }
870}
871
872/// One feature's exact share of a candidate's utility.
873#[derive(Clone, Debug, Serialize, Deserialize)]
874pub struct Contribution {
875    /// φ coordinate name.
876    pub name: String,
877    /// The lens's weight on it.
878    pub theta: f64,
879    /// The candidate's standardized value on it.
880    pub phi_std: f64,
881    /// `theta · phi_std` — this feature's signed share of the utility.
882    pub contribution: f64,
883}
884
885/// Why the model scores one candidate the way it does.
886///
887/// Utility is **exactly linear within a style lens**, so this decomposition
888/// is exact rather than a local surrogate: `Σ contribution = utility`. No
889/// SHAP, no LIME, no approximation error to caveat.
890#[derive(Clone, Debug, Serialize, Deserialize)]
891pub struct Explanation {
892    /// Candidate id.
893    pub id: u64,
894    /// Aligned index of the lens that claims this candidate.
895    pub style: usize,
896    /// That lens's user-given name (`""` if unnamed).
897    pub style_name: String,
898    /// Posterior-mean utility **under that lens** — exactly the sum of the
899    /// contributions. This is the quantity the decomposition explains.
900    pub utility: f64,
901    /// Posterior std of that same lens utility — how sure the model is about
902    /// this score.
903    pub utility_std: f64,
904    /// Posterior-mean **mixture** utility `E[max_k u_k]` — the number the
905    /// bank is ranked by, and the one to show as *the score*.
906    ///
907    /// It is not the same number as `utility`, and it is never smaller: the
908    /// ranking takes the max over lenses inside the expectation, while the
909    /// decomposition necessarily fixes one lens first. Jensen's inequality
910    /// does the rest. Showing `utility` next to a bank ordered by
911    /// `mix_utility` would render a systematically lower number beside the
912    /// row it is supposed to explain.
913    pub mix_utility: f64,
914    /// Posterior probability that `style` really is this candidate's best
915    /// lens. Near 1, `utility ≈ mix_utility` and the explanation is the whole
916    /// story; well below 1, the candidate sits between islands and the gap is
917    /// worth surfacing rather than hiding.
918    pub responsibility: f64,
919    /// Every feature's contribution, sorted by descending magnitude.
920    pub contributions: Vec<Contribution>,
921}
922
923fn sigmoid(x: f64) -> f64 {
924    1.0 / (1.0 + (-x).exp())
925}
926
927/// Binary entropy in nats, guarded at the ends.
928fn binary_entropy(p: f64) -> f64 {
929    let p = p.clamp(1e-12, 1.0 - 1e-12);
930    -p * p.ln() - (1.0 - p) * (1.0 - p).ln()
931}
932
933/// Dueling Thompson sampling, kept for the acquisition A/B (see
934/// [`Acquisition`]). Draw two posterior samples and duel each one's champion;
935/// if they agree, duel the champion against the runner-up.
936fn thompson_pair<R: Rng>(
937    posterior: &TastePosterior,
938    pool: &[Candidate],
939    cands: &[usize],
940    rng: &mut R,
941) -> (usize, usize) {
942    let n = posterior.samples.len();
943    if n == 0 {
944        return (cands[0], cands[1]);
945    }
946    let champion = |s: &auracle_taste::TasteSample, skip: Option<usize>| -> usize {
947        cands
948            .iter()
949            .copied()
950            .filter(|i| Some(*i) != skip)
951            .max_by(|x, y| {
952                s.utility_mix(&pool[*x].phi_std)
953                    .total_cmp(&s.utility_mix(&pool[*y].phi_std))
954            })
955            .unwrap_or(cands[0])
956    };
957    let s1 = &posterior.samples[rng.gen_range(0..n)];
958    let s2 = &posterior.samples[rng.gen_range(0..n)];
959    let a = champion(s1, None);
960    let b = champion(s2, None);
961    if a == b {
962        (a, champion(s2, Some(a)))
963    } else {
964        (a, b)
965    }
966}
967
968/// A hashable fingerprint of a feature vector, for de-duplicating the
969/// standardizer's reference sample. Bit patterns of the raw values: identical
970/// candidates featurize deterministically, so exact equality is the right
971/// test and rounding would only invent collisions.
972fn quantize(row: &[f64]) -> Vec<u64> {
973    row.iter().map(|x| x.to_bits()).collect()
974}
975
976/// Unordered key for a candidate pair.
977fn pair_key(a: u64, b: u64) -> (u64, u64) {
978    if a <= b {
979        (a, b)
980    } else {
981        (b, a)
982    }
983}
984
985/// The session engine.
986pub struct Engine {
987    /// Configuration.
988    pub cfg: SessionConfig,
989    /// The patch prior.
990    pub prior: PatchGrammarPrior,
991    /// Standardizer fit on the first pool fill; persisted with the profile.
992    pub standardizer: Option<Arc<Standardizer>>,
993    /// The observation log (source of truth).
994    pub log: ObservationLog,
995    /// The current posterior, if fit (label-aligned).
996    pub posterior: Option<Arc<TastePosterior>>,
997    /// Current session index.
998    pub session: usize,
999    /// The candidate pool.
1000    pub pool: Vec<Candidate>,
1001    /// Evolution/edit history.
1002    pub lineage: Vec<LineageEvent>,
1003    /// Generation counter (one per refinement call).
1004    pub generation: usize,
1005    /// User-given style names (index = aligned style index; empty = unnamed).
1006    pub style_names: Vec<String>,
1007    /// Implicit preference events (logged, not yet modeled).
1008    pub events: Vec<ImplicitEvent>,
1009    /// Out-of-sample duel forecasts, scored before each answer was known.
1010    pub forecasts: Vec<Forecast>,
1011    /// Style shares recorded at each posterior fit. See [`StyleShareRecord`].
1012    style_shares: Vec<StyleShareRecord>,
1013    /// How many times each (unordered) candidate pair has been shown, keyed
1014    /// by stable id. Drives the repeat penalty in [`Engine::next_duel`].
1015    shown_pairs: HashMap<(u64, u64), u32>,
1016    /// How many times each candidate has been offered, by any pairing.
1017    shown_candidates: HashMap<u64, u32>,
1018    /// Duels offered this run (not the same as observations recorded — the
1019    /// user may skip). Paces the random check duels.
1020    duels_shown: usize,
1021    /// The most recently offered *check* pair, so the forecast it produces can
1022    /// be tagged as unbiased even though the frontend records it like any
1023    /// other duel.
1024    last_check_pair: Option<(u64, u64)>,
1025    /// How many times the importance weights have collapsed and been
1026    /// resampled since the last full MCMC fit — the staleness signal behind
1027    /// [`Engine::needs_refit`].
1028    resamples_since_fit: usize,
1029    /// The featurization memo every featurize in this engine consults.
1030    memo: RenderMemo,
1031    /// Ids whose audition buffer is resident under [`RenderPolicy::Lazy`],
1032    /// least recently used first.
1033    audio_lru: VecDeque<u64>,
1034    /// Base seed of the indexed pool-draw stream ([`crate::farm`]). Taken from
1035    /// the caller's RNG on the first fill, so [`Engine::fill_pool_step`]'s
1036    /// signature — and the amount of that RNG's stream a fill consumes — stay
1037    /// what they were.
1038    fill_seed: Option<u64>,
1039    /// Next index of that stream the *fold* will consume. Advances only on
1040    /// absorption, which is what makes it independent of how many draws are in
1041    /// flight.
1042    draw_cursor: u64,
1043    /// Next index handed out by [`Engine::fill_draw`]. Runs ahead of
1044    /// `draw_cursor` by whatever is in flight; speculative distance between
1045    /// them is free, because an index that is never absorbed never happened.
1046    issue_cursor: u64,
1047    next_id: u64,
1048    /// What the last session restore had to mend — see
1049    /// [`Engine::repair_report`]. Saved terms whose knobs were out of range,
1050    /// log cells clamped, and observations dropped as uninterpretable.
1051    repaired_terms: usize,
1052    repaired_cells: usize,
1053    dropped_observations: usize,
1054}
1055
1056impl Engine {
1057    /// Create an engine over the given prior.
1058    pub fn new(prior: PatchGrammarPrior, cfg: SessionConfig) -> Self {
1059        Self {
1060            cfg,
1061            prior,
1062            standardizer: None,
1063            log: ObservationLog::new(),
1064            posterior: None,
1065            session: 0,
1066            pool: Vec::new(),
1067            lineage: Vec::new(),
1068            generation: 0,
1069            style_names: Vec::new(),
1070            events: Vec::new(),
1071            forecasts: Vec::new(),
1072            style_shares: Vec::new(),
1073            shown_pairs: HashMap::new(),
1074            shown_candidates: HashMap::new(),
1075            duels_shown: 0,
1076            last_check_pair: None,
1077            resamples_since_fit: 0,
1078            memo: RenderMemo::default(),
1079            audio_lru: VecDeque::new(),
1080            fill_seed: None,
1081            draw_cursor: 0,
1082            issue_cursor: 0,
1083            next_id: 1,
1084            repaired_terms: 0,
1085            repaired_cells: 0,
1086            dropped_observations: 0,
1087        }
1088    }
1089
1090    /// Replace the featurization memo every featurize in this engine consults
1091    /// — fill, insert, restore, and the refinement surrogate.
1092    ///
1093    /// Shared rather than owned so a frontend can pre-load one and read back
1094    /// what the engine learned. Refinement captures it by clone at
1095    /// [`Engine::refine_one`] time, so swapping it mid-generation is not
1096    /// something to do.
1097    pub fn set_memo(&mut self, memo: RenderMemo) {
1098        self.memo = memo;
1099    }
1100
1101    /// The featurization memo.
1102    pub fn memo(&self) -> &RenderMemo {
1103        &self.memo
1104    }
1105
1106    /// Content address of candidate `id`'s featurization.
1107    pub fn key_of(&self, id: u64) -> Option<&str> {
1108        self.find(id).map(|i| self.pool[i].key.as_str())
1109    }
1110
1111    /// The audition buffer of candidate `id`, materializing it if
1112    /// [`RenderPolicy::Lazy`] deferred it.
1113    ///
1114    /// `None` for an unknown id, for [`RenderPolicy::None`], or for a term
1115    /// that no longer renders — a restored bank outlives the DSP that made it,
1116    /// and a caller that cannot distinguish "not yet" from "never" will wait
1117    /// forever. This is the *only* honest source of that answer.
1118    ///
1119    /// Bit-identical to the buffer the candidate's features were measured on
1120    /// ([`auracle_features::render_playback`]).
1121    ///
1122    /// Shared rather than copied: the pool, the memo and the caller all hold
1123    /// the same ~565 KB allocation through an [`Arc`], so a repeat request is
1124    /// a refcount bump. Callers that must own samples clone the inner value at
1125    /// their own call site, where the cost is visible.
1126    pub fn render_of(&mut self, id: u64) -> Option<Arc<Audition>> {
1127        let i = self.find(id)?;
1128        if let Some(a) = self.pool[i].render.clone() {
1129            if self.cfg.render_policy == RenderPolicy::Lazy {
1130                self.touch_audition(id);
1131            }
1132            return Some(a);
1133        }
1134        if self.cfg.render_policy != RenderPolicy::Lazy {
1135            // Eager already stored one at admission; None keeps nothing.
1136            return None;
1137        }
1138        let key = self.pool[i].key.clone();
1139        let audio = match self.memo.get_audio(&key) {
1140            Some(a) => a,
1141            None => Arc::new(
1142                render_playback(
1143                    &self.pool[i].tree,
1144                    &self.cfg.phrase,
1145                    self.pool[i].features.gain_db,
1146                )
1147                .ok()?,
1148            ),
1149        };
1150        self.pool[i].render = Some(Arc::clone(&audio));
1151        // `touch_audition` may evict other members but never `id`, which it
1152        // marks most-recently-used; the returned handle is valid regardless.
1153        self.touch_audition(id);
1154        Some(audio)
1155    }
1156
1157    /// Mark `id`'s buffer as most recently used and drop whatever falls out of
1158    /// [`SessionConfig::audio_cache`].
1159    fn touch_audition(&mut self, id: u64) {
1160        // Evicted candidates leave their ids behind; dropping them here keeps
1161        // the cache from being consumed by ghosts and holding live buffers
1162        // past the cap.
1163        let live: HashSet<u64> = self.pool.iter().map(|c| c.id).collect();
1164        self.audio_lru.retain(|x| *x != id && live.contains(x));
1165        self.audio_lru.push_back(id);
1166        let cap = self.cfg.audio_cache.max(1);
1167        while self.audio_lru.len() > cap {
1168            let Some(evicted) = self.audio_lru.pop_front() else {
1169                break;
1170            };
1171            if let Some(i) = self.find(evicted) {
1172                self.pool[i].render = None;
1173            }
1174        }
1175    }
1176
1177    /// Whether an admitting featurize should bother producing samples.
1178    ///
1179    /// Only [`RenderPolicy::Eager`] keeps a buffer, so under the other two
1180    /// policies asking for one would convert 141 k samples straight into a
1181    /// `drop`. This is the flag every `featurize_memo` call in the engine
1182    /// passes, and the reason the pool fill under `Lazy` costs φ only.
1183    fn wants_admitted_audio(&self) -> bool {
1184        self.cfg.render_policy == RenderPolicy::Eager
1185    }
1186
1187    /// The audition buffer a freshly-admitted candidate should carry, per
1188    /// policy. `fresh` is the buffer the admitting featurize produced, if it
1189    /// rendered rather than hitting the memo.
1190    fn admitted_render(
1191        &self,
1192        tree: &PatchTree,
1193        features: &Features,
1194        fresh: Option<Arc<Audition>>,
1195    ) -> Option<Arc<Audition>> {
1196        match self.cfg.render_policy {
1197            RenderPolicy::Eager => fresh.or_else(|| {
1198                render_playback(tree, &self.cfg.phrase, features.gain_db)
1199                    .ok()
1200                    .map(Arc::new)
1201            }),
1202            RenderPolicy::Lazy | RenderPolicy::None => None,
1203        }
1204    }
1205
1206    fn alloc_id(&mut self) -> u64 {
1207        let id = self.next_id;
1208        self.next_id += 1;
1209        id
1210    }
1211
1212    /// Pool index of a candidate id.
1213    pub fn find(&self, id: u64) -> Option<usize> {
1214        self.pool.iter().position(|c| c.id == id)
1215    }
1216
1217    /// Start a new session (its own τ latent). Returns its index.
1218    pub fn begin_session(&mut self) -> usize {
1219        if !self.log.is_empty() {
1220            self.session = self.log.n_sessions();
1221        }
1222        self.session
1223    }
1224
1225    /// Fill the pool with vetted prior draws (up to `pool_size`). Fits the
1226    /// standardizer on the first successful fill.
1227    pub fn fill_pool<R: Rng>(&mut self, rng: &mut R) {
1228        let target = self.cfg.pool_size;
1229        while self.pool.len() < target {
1230            if self.fill_pool_step(rng, target - self.pool.len()) == 0 {
1231                break;
1232            }
1233        }
1234        // Fill fell short (vet failures exhausted the draw budget): fit the
1235        // standardizer on what we have rather than leaving φ un-standardized.
1236        if self.standardizer.is_none() && !self.pool.is_empty() {
1237            let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1238            self.standardizer = Some(Arc::new(Standardizer::fit(&rows)));
1239            for c in &mut self.pool {
1240                c.phi_std = self
1241                    .standardizer
1242                    .as_ref()
1243                    .unwrap()
1244                    .transform(&c.features.phi());
1245            }
1246        }
1247    }
1248
1249    /// Add up to `max_new` vetted candidates (bounded by `max_draws`
1250    /// attempts). Returns how many were added — the incremental unit that
1251    /// lets a frontend post progress between batches. Standardization runs
1252    /// once the pool first reaches `pool_size` (or on any later addition).
1253    ///
1254    /// This is the serial fold of the indexed draw stream ([`crate::farm`]):
1255    /// index `i` is consumed whatever its outcome, and dedupe / vetting decide
1256    /// only whether it *lands*. The farm path ([`Engine::fill_draw`] +
1257    /// [`Engine::absorb_prior`]) is the same fold with the render moved
1258    /// off-engine, so the two produce the same pool from the same
1259    /// `fill_seed` — and so does any chunking of `max_new`, because the cursor
1260    /// lives in the engine rather than in a loop variable.
1261    pub fn fill_pool_step<R: Rng>(&mut self, rng: &mut R, max_new: usize) -> usize {
1262        self.ensure_fill_seed(rng);
1263        let mut added = 0;
1264        while added < max_new
1265            && self.pool.len() < self.cfg.pool_size
1266            && self.draw_cursor < self.cfg.max_draws as u64
1267        {
1268            let index = self.draw_cursor;
1269            let Some(tree) = self.draw_at(index) else {
1270                break;
1271            };
1272            self.consume_draw(index);
1273            if self.pool.iter().any(|c| c.tree == tree) {
1274                continue;
1275            }
1276            let want_audio = self.wants_admitted_audio();
1277            if let Ok((cached, audition)) =
1278                featurize_memo(&tree, &self.cfg.phrase, &self.memo, want_audio)
1279            {
1280                self.push_prior(PreFeaturized {
1281                    tree,
1282                    cached,
1283                    audition,
1284                });
1285                added += 1;
1286            }
1287        }
1288        self.standardize_pool();
1289        added
1290    }
1291
1292    // ------------------------------------------------------------------
1293    // The indexed draw stream and its off-engine fold (see `crate::farm`)
1294    // ------------------------------------------------------------------
1295
1296    /// Base seed of this engine's pool-draw stream, taking one from `rng` if
1297    /// the stream has not started yet.
1298    ///
1299    /// Exactly one `u64` is drawn from the caller's RNG per engine, on the
1300    /// first fill. That is deliberate: the serial and farm paths consume the
1301    /// same amount of the caller's stream, so everything downstream of the
1302    /// fill that shares that RNG (duel selection, MCMC) stays aligned between
1303    /// them.
1304    pub fn ensure_fill_seed<R: Rng>(&mut self, rng: &mut R) -> u64 {
1305        match self.fill_seed {
1306            Some(s) => s,
1307            None => {
1308                let s = rng.gen::<u64>();
1309                self.fill_seed = Some(s);
1310                s
1311            }
1312        }
1313    }
1314
1315    /// Base seed of the pool-draw stream, if it has started.
1316    pub fn fill_seed(&self) -> Option<u64> {
1317        self.fill_seed
1318    }
1319
1320    /// Pin the pool-draw stream to an explicit base seed. Only meaningful
1321    /// before the first draw; a fill in progress keeps the seed it started on.
1322    pub fn set_fill_seed(&mut self, seed: u64) {
1323        if self.draw_cursor == 0 && self.issue_cursor == 0 {
1324            self.fill_seed = Some(seed);
1325        }
1326    }
1327
1328    /// Next index of the draw stream the fold will consume.
1329    pub fn draw_cursor(&self) -> u64 {
1330        self.draw_cursor
1331    }
1332
1333    /// The term at `index` of this engine's draw stream — a pure function of
1334    /// `(fill_seed, index)` and the prior, costing microseconds and no render.
1335    ///
1336    /// This is what makes a lost farm job re-issuable with no retained state:
1337    /// the job *is* its index.
1338    pub fn draw_at(&self, index: u64) -> Option<PatchTree> {
1339        let base = self.fill_seed?;
1340        let mut sub = StdRng::seed_from_u64(draw_seed(base, index));
1341        Some(self.prior.sample_with_rng(&mut sub))
1342    }
1343
1344    /// Hand out up to `n` unrendered draws for off-engine featurization.
1345    ///
1346    /// Returns fewer than `n` — or nothing — when the pool has as much work
1347    /// outstanding as it can still use, or the `max_draws` budget is spent.
1348    /// An empty return is *not* by itself a stop signal: it may simply mean
1349    /// every slot the pool can still fill is already in flight. The caller
1350    /// stops when the pool reaches its target, or when an empty return
1351    /// coincides with nothing outstanding.
1352    ///
1353    /// Requires a started stream ([`Engine::ensure_fill_seed`] or
1354    /// [`Engine::set_fill_seed`]); yields nothing otherwise.
1355    pub fn fill_draw(&mut self, n: usize) -> Vec<Draw> {
1356        let mut out = Vec::new();
1357        if self.fill_seed.is_none() {
1358            return out;
1359        }
1360        let need = self.cfg.pool_size.saturating_sub(self.pool.len());
1361        if need == 0 {
1362            return out;
1363        }
1364        // Over-issue by a quarter for vet failures and duplicates, plus one so
1365        // a single remaining slot still gets a second attempt in flight. More
1366        // than that is not wrong — over-issue is discardable by construction —
1367        // just wasted work on a machine that could have been rendering
1368        // something the pool will keep.
1369        let ceiling = (need + need / 4 + 1) as u64;
1370        let outstanding = self.issue_cursor.saturating_sub(self.draw_cursor);
1371        let room = ceiling.saturating_sub(outstanding);
1372        for _ in 0..(n as u64).min(room) {
1373            if self.issue_cursor >= self.cfg.max_draws as u64 {
1374                break;
1375            }
1376            let index = self.issue_cursor;
1377            let Some(tree) = self.draw_at(index) else {
1378                break;
1379            };
1380            let dup = self.pool.iter().any(|c| c.tree == tree);
1381            self.issue_cursor = index + 1;
1382            out.push(Draw { index, tree, dup });
1383        }
1384        out
1385    }
1386
1387    /// Fold one off-engine result into the pool.
1388    ///
1389    /// `index` **must** be [`Engine::draw_cursor`] — results are absorbed in
1390    /// index order, and that ordering is the entire determinism argument: the
1391    /// pool at index `i` is a pure function of indices `< i`, so it cannot
1392    /// depend on how many renders were running. Anything else is refused
1393    /// (returns `None` without consuming), because silently absorbing out of
1394    /// order would produce a pool no width reproduces.
1395    ///
1396    /// `pre` is `None` for a draw the farm rejected — a vet failure, a
1397    /// compile failure, or a result that failed to survive transport. The
1398    /// index is consumed either way, exactly as a failed draw burns an attempt
1399    /// in the serial loop.
1400    ///
1401    /// Returns the new candidate id, or `None` when the draw did not land
1402    /// (rejected, duplicate, or the pool was already full).
1403    pub fn absorb_prior(&mut self, index: u64, pre: Option<PreFeaturized>) -> Option<u64> {
1404        if index != self.draw_cursor
1405            || self.pool.len() >= self.cfg.pool_size
1406            || index >= self.cfg.max_draws as u64
1407        {
1408            return None;
1409        }
1410        self.consume_draw(index);
1411        let mut id = None;
1412        if let Some(pre) = pre {
1413            if !self.pool.iter().any(|c| c.tree == pre.tree) {
1414                id = Some(self.push_prior(pre));
1415            }
1416        }
1417        self.standardize_pool();
1418        id
1419    }
1420
1421    /// Mark index `index` as folded in, whatever its outcome.
1422    fn consume_draw(&mut self, index: u64) {
1423        self.draw_cursor = index + 1;
1424        self.issue_cursor = self.issue_cursor.max(self.draw_cursor);
1425    }
1426
1427    /// Admit a prior draw whose featurization is already done. The single push
1428    /// site for [`Origin::Prior`], shared by the serial and farm paths so
1429    /// there is no second copy of the admission rules to drift.
1430    fn push_prior(&mut self, pre: PreFeaturized) -> u64 {
1431        let PreFeaturized {
1432            tree,
1433            cached,
1434            audition,
1435        } = pre;
1436        // Fold the off-engine work into this engine's memo: a farm render is
1437        // exactly the artifact a later audition or refinement would otherwise
1438        // recompute, and the memo is where every other path looks for it.
1439        self.memo.put(cached.clone(), audition.clone());
1440        let id = self.alloc_id();
1441        let render = self.admitted_render(&tree, &cached.features, audition);
1442        self.pool.push(Candidate {
1443            id,
1444            tree: settled(tree),
1445            phi_std: Vec::new(),
1446            key: cached.key,
1447            render,
1448            features: cached.features,
1449            origin: Origin::Prior,
1450            name: None,
1451            pinned: false,
1452        });
1453        id
1454    }
1455
1456    /// Fit the standardizer once the pool first reaches `pool_size`, then give
1457    /// every un-standardized member its φ_std. The tail of a fill step, lifted
1458    /// so the serial and farm paths run the identical bookkeeping.
1459    fn standardize_pool(&mut self) {
1460        if self.standardizer.is_none() && self.pool.len() >= self.cfg.pool_size {
1461            let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1462            self.standardizer = Some(Arc::new(Standardizer::fit(&rows)));
1463        }
1464        let Some(sz) = self.standardizer.clone() else {
1465            return;
1466        };
1467        for c in &mut self.pool {
1468            if c.phi_std.is_empty() {
1469                c.phi_std = sz.transform(&c.features.phi());
1470            }
1471        }
1472    }
1473
1474    /// Give every pool member a φ_std **now**, fitting a standardizer from
1475    /// the current pool if none exists yet — so a *partially filled* pool is
1476    /// already duel-able.
1477    ///
1478    /// [`Engine::fill_pool_step`] only fits once the pool reaches
1479    /// `pool_size`, and that single condition is what forces a frontend to sit
1480    /// out the entire fill before it can ask its first question:
1481    /// [`Engine::next_duel_full`] skips candidates whose `phi_std` is empty,
1482    /// so a half-filled pool contains no legal pair at all. This is the
1483    /// escape hatch a progressive boot needs — it costs no renders, only the
1484    /// mean/variance of what has already been drawn.
1485    ///
1486    /// It never *replaces* an existing standardizer. θ is only meaningful
1487    /// relative to the standardization its φ were measured under, so an
1488    /// imported profile's geometry has to survive a boot that tops the pool
1489    /// up ([`Engine::import_profile`]). Re-fitting is
1490    /// [`Engine::restandardize_if_untaught`]'s job, and it is only safe
1491    /// before a posterior exists.
1492    pub fn standardize_now(&mut self) {
1493        if self.pool.is_empty() {
1494            return;
1495        }
1496        if self.standardizer.is_none() {
1497            let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1498            self.standardizer = Some(Arc::new(Standardizer::fit(&rows)));
1499        }
1500        let sz = self.standardizer.clone().expect("just fit above");
1501        for c in &mut self.pool {
1502            if c.phi_std.is_empty() {
1503                c.phi_std = sz.transform(&c.features.phi());
1504            }
1505        }
1506    }
1507
1508    /// Re-fit the standardizer over the finished pool — a no-op the moment a
1509    /// posterior exists.
1510    ///
1511    /// A progressive boot fits a *provisional* standardizer over the first
1512    /// handful of draws ([`Engine::standardize_now`]) so the user can start
1513    /// voting; the completed pool is a better reference population, and
1514    /// re-expressing φ on it is lossless because the log stores **raw**
1515    /// values (`refit_standardizer`'s rationale). But once θ has
1516    /// been fit, its coordinates are denominated in the standardizer that was
1517    /// live at fit time, and moving the scale under a live posterior would
1518    /// silently rescale every utility in the app. So this refuses in exactly
1519    /// that case: the next [`Engine::fit_posterior`] re-fits both together,
1520    /// in the order that keeps them consistent.
1521    pub fn restandardize_if_untaught(&mut self) {
1522        if self.posterior.is_none() {
1523            self.refit_standardizer();
1524        }
1525    }
1526
1527    /// Re-fit the standardizer over everything the model is about to see: the
1528    /// raw φ in the log **and** the live pool.
1529    ///
1530    /// Fitting it once on the first 40 prior draws and freezing it meant that
1531    /// as the pool drifted toward refined candidates the z-scores drifted with
1532    /// it, and the linear model ended up extrapolating well outside the range
1533    /// it was calibrated on. Because the log now stores raw values, re-fitting
1534    /// is free and lossless — it re-expresses the same evidence on a scale
1535    /// that still matches where the data actually is.
1536    fn refit_standardizer(&mut self) {
1537        let names = phi_names();
1538        // The reference population is *the patches the user has encountered*,
1539        // each counted once — the live pool plus anything in the log that has
1540        // since been evicted. Deliberately not the multiset of comparisons:
1541        // acquisition decides which candidates get dueled repeatedly, and
1542        // letting that decide the coordinate system closes a feedback loop
1543        // between the question-asker and the units the answers are measured
1544        // in. Same reason the standardizer exists at all.
1545        let mut rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1546        let mut seen: HashSet<Vec<u64>> = rows.iter().map(|r| quantize(r)).collect();
1547        for row in self.log.raw_rows(&names) {
1548            // Width guard, belt-and-braces: a ragged row reaches an assertion
1549            // inside `Standardizer::fit` and panics the whole engine. A log
1550            // that survived a bad migration should cost us that vote, not the
1551            // session.
1552            if row.len() == names.len() && seen.insert(quantize(&row)) {
1553                rows.push(row);
1554            }
1555        }
1556        if rows.is_empty() {
1557            return;
1558        }
1559        let sz = Arc::new(Standardizer::fit(&rows));
1560        for c in &mut self.pool {
1561            c.phi_std = sz.transform(&c.features.phi());
1562        }
1563        self.standardizer = Some(sz);
1564    }
1565
1566    /// Fit (or re-fit) the taste posterior from the observation log. The
1567    /// stored posterior is label-aligned (safe for per-style summaries) and
1568    /// its importance weights are reset to uniform.
1569    pub fn fit_posterior<R: Rng>(&mut self, rng: &mut R) {
1570        if self.log.is_empty() {
1571            return;
1572        }
1573        self.refit_standardizer();
1574        let Some(sz) = self.standardizer.clone() else {
1575            return;
1576        };
1577        let names = phi_names();
1578        let d = names.len();
1579        // Style capacity grows with evidence: one lens per ~20 observations,
1580        // capped by config. Idle lenses collapse to ~0 share on their own,
1581        // so K is an upper bound the data may or may not use.
1582        let k = (1 + self.log.len() / 20).min(self.cfg.k_styles).max(1);
1583        let mut taste_cfg = TasteConfig::mixture(d, k);
1584        taste_cfg.recency_half_life = self.cfg.recency_half_life;
1585        // The brightness cluster shares a latent mean per style. Resolved by
1586        // *name* here because this is the layer that knows them; the taste
1587        // crate is handed indices and never learns what they mean. A name that
1588        // is not in φ simply does not join the group, so a stimulus-tag bump
1589        // or a dropped column degrades to the flat prior rather than panicking
1590        // or silently fusing the wrong coordinate.
1591        let bright: Vec<usize> = ["rolloff_mean", "zcr_mean", "centroid_mean"]
1592            .iter()
1593            .filter_map(|want| names.iter().position(|n| n.split(':').next() == Some(want)))
1594            .collect();
1595        if bright.len() > 1 {
1596            taste_cfg.fused = vec![bright];
1597        }
1598        let model = TasteModel::new(taste_cfg);
1599        let data = FitSet::build(&self.log, &names, &sz);
1600        let posterior = model.fit(rng, &data, self.cfg.mcmc_samples, self.cfg.mcmc_warmup);
1601        let posterior = Arc::new(posterior.aligned());
1602        // Measured against the pool the fit is about to be used on, which is
1603        // the population the shares are a statement about — not against the
1604        // log, whose φ are the things already judged.
1605        let pool_phis: Vec<Vec<f64>> = self
1606            .pool
1607            .iter()
1608            .filter(|c| !c.phi_std.is_empty())
1609            .map(|c| c.phi_std.clone())
1610            .collect();
1611        if !pool_phis.is_empty() {
1612            self.style_shares.push(StyleShareRecord {
1613                observations: self.log.len(),
1614                k,
1615                shares: posterior.style_share(&pool_phis),
1616            });
1617        }
1618        self.posterior = Some(posterior);
1619        self.resamples_since_fit = 0;
1620    }
1621
1622    /// Style shares recorded at each fit, oldest first. See
1623    /// [`StyleShareRecord`].
1624    pub fn style_shares(&self) -> &[StyleShareRecord] {
1625        &self.style_shares
1626    }
1627
1628    /// Effective sample size of the current posterior's importance weights —
1629    /// how much of the draw set still carries information after the
1630    /// observations folded in since the last full fit. `None` before the
1631    /// first fit.
1632    pub fn posterior_ess(&self) -> Option<f64> {
1633        self.posterior.as_ref().map(|p| p.ess())
1634    }
1635
1636    /// True when the cheap between-fit updates have run out of road and a
1637    /// full MCMC refit is worth its seconds: the weights have collapsed
1638    /// (ESS below half the draws) at least once since the last fit, or the
1639    /// log has evidence no posterior has seen. A frontend can drive refits
1640    /// off this instead of a fixed vote count.
1641    pub fn needs_refit(&self) -> bool {
1642        match &self.posterior {
1643            Some(_) => self.resamples_since_fit > 0,
1644            None => !self.log.is_empty(),
1645        }
1646    }
1647
1648    /// Posterior-mean mixture utility of a standardized φ (0 with no
1649    /// posterior).
1650    pub fn utility_of(&self, phi_std: &[f64]) -> f64 {
1651        match &self.posterior {
1652            Some(p) if !phi_std.is_empty() => p.utility_mix(phi_std).0,
1653            _ => 0.0,
1654        }
1655    }
1656
1657    /// Did the step from `prev` to `next` touch any locked address?
1658    /// "Touch" = change its value, delete it, **or create it** (structure
1659    /// moves that would rewrite a locked module's path are rejected too —
1660    /// locked means *don't touch*).
1661    ///
1662    /// Both directions are checked, and that is not pedantry. Scanning only
1663    /// `prev` lets a *birth* at a locked address through while rejecting the
1664    /// death that would undo it. The constraint region is then asymmetric —
1665    /// x → x′ allowed, x′ → x rejected — which breaks detailed balance and
1666    /// makes the Metropolis-within-Gibbs argument for locking being exact
1667    /// simply false. The chain would drift into locked structure it can never
1668    /// leave.
1669    ///
1670    /// **What this does and does not guarantee.** `locked` is a set of exact
1671    /// address strings, typically snapshotted from the UI. Every address in
1672    /// it is frozen, in both directions, and *that* is exact. It is not the
1673    /// same as freezing a module: a structural move that grows a brand-new
1674    /// address inside a locked module — one that was in neither trace when
1675    /// the set was taken, so it cannot be in the set — is not caught. That
1676    /// case is symmetric (unmatched by construction in both directions), so
1677    /// it costs nothing in detailed balance; it just means "locked" is a
1678    /// guarantee about *addresses*, not about subtrees.
1679    pub fn violates_locks(prev: &Trace, next: &Trace, locked: &HashSet<String>) -> bool {
1680        if locked.is_empty() {
1681            return false;
1682        }
1683        for (addr, c) in &prev.choices {
1684            if locked.contains(&**addr) {
1685                match next.choices.get(addr) {
1686                    Some(n) if n.value == c.value => {}
1687                    _ => return true,
1688                }
1689            }
1690        }
1691        for addr in next.choices.keys() {
1692            if locked.contains(&**addr) && !prev.choices.contains_key(addr) {
1693                return true;
1694            }
1695        }
1696        false
1697    }
1698
1699    /// Grammar prior with kind-weights tilted toward the fitted taste: each
1700    /// structural θ component (share-weighted across styles) multiplies its
1701    /// kind's proposal weight by `exp(η·θ)`. This is θ_struct → grammar
1702    /// feedback — refinement *proposes* toward the user instead of merely
1703    /// filtering, which is where visible directionality comes from.
1704    ///
1705    /// Two things make the mapping from φ names to grammar weights less than
1706    /// a lookup, and both are consequences of φ carrying **families**
1707    /// (`auracle_features::StructFeatures`):
1708    ///
1709    /// - Several kinds share one coefficient. `n_drive` speaks for the
1710    ///   wavefolder, the distortion, the bitcrusher and the ring modulator;
1711    ///   `n_mod_fx` for the chorus, phaser, flanger, tremolo and vibrato;
1712    ///   `n_time` for the delay, the granulator and the pitch shifter;
1713    ///   `n_filter` for the filter, the EQ and the vocoder; `n_dynamics` for
1714    ///   the compressor, the ducker and the gate. They each get the family's
1715    ///   tilt, which is the honest reading: the evidence never distinguished
1716    ///   them, so the proposal should not pretend it did. Their *base* weights
1717    ///   still differ, so the tilt shifts the family without flattening it.
1718    /// - `n_mix` is not in φ at all — it is determined by the source count and
1719    ///   the other five binary counts under the exact identity that removed it
1720    ///   — so its tilt comes from the sources: wanting more sources is wanting
1721    ///   more binary nodes to combine them, and that is the only sense in
1722    ///   which the taste model has an opinion here. The other five binaries
1723    ///   take the tilt of whichever family they are counted under.
1724    ///
1725    /// Each coefficient is also **shrunk by its own posterior uncertainty**,
1726    /// `θ·|θ|/(|θ| + σ)`, before it tilts anything. The new palette's
1727    /// coefficients are the least identified ones in the model — a pool of 48
1728    /// draws contains a handful of bitcrushers — so a raw posterior mean is
1729    /// as likely to be sampling noise as signal, and feeding noise into the
1730    /// *proposal* distribution compounds it: the pool drifts toward the
1731    /// spurious kind, which produces more evidence about it, which is not the
1732    /// same as producing more evidence *for* it. A coefficient whose σ equals
1733    /// its mean tilts half as hard; one that is mostly noise tilts not at all.
1734    fn biased_prior(&self) -> PatchGrammarPrior {
1735        let mut prior = self.prior.clone();
1736        let eta = self.cfg.proposal_tilt;
1737        let Some(p) = &self.posterior else {
1738            return prior;
1739        };
1740        if eta <= 0.0 {
1741            return prior;
1742        }
1743        let names = Features::phi_names();
1744        let pool_phis: Vec<Vec<f64>> = self
1745            .pool
1746            .iter()
1747            .filter(|c| !c.phi_std.is_empty())
1748            .map(|c| c.phi_std.clone())
1749            .collect();
1750        let shares = p.style_share(&pool_phis);
1751        let (mut theta, mut sd) = (vec![0.0; names.len()], vec![0.0; names.len()]);
1752        for k in 0..p.k_styles() {
1753            let w = shares.get(k).copied().unwrap_or(0.0);
1754            for (t, mi) in theta.iter_mut().zip(p.theta_mean(k)) {
1755                *t += w * mi;
1756            }
1757            for (s, si) in sd.iter_mut().zip(p.theta_std(k)) {
1758                *s += w * si;
1759            }
1760        }
1761        let g = |name: &str| {
1762            names
1763                .iter()
1764                .position(|n| *n == name)
1765                .map(|i| shrink(theta[i], sd[i]))
1766                .unwrap_or(0.0)
1767        };
1768        let sources = [
1769            g("n_vco"),
1770            g("n_supersaw"),
1771            g("n_noise"),
1772            g("n_wavetable"),
1773            g("n_pluck"),
1774            g("n_formant"),
1775            // `Silence` is deliberately **not** tilted by taste, and this zero
1776            // is the whole of that decision. The tilt exists to move proposals
1777            // toward source kinds the listener is enjoying; a hole is not a
1778            // timbre anyone can enjoy, it is the absence of one, and its
1779            // prevalence is meant to come from a player unplugging a socket
1780            // rather than from a fitted coefficient.
1781            //
1782            // It is also the column where a tilt would be least trustworthy.
1783            // At a 0.5% prior rate `n_silence` is zero in nearly every row a
1784            // fit sees — the near-indicator shape that kept `n_ringmod` out of
1785            // φ as a column of its own. `shrink` would damp a spurious
1786            // coefficient, but the multiplier it feeds is exponential, and
1787            // amplifying holes into the pool is a failure a listener notices
1788            // immediately.
1789            0.0,
1790        ];
1791        let src = tilt_weights(&prior.source_weights, &sources, eta);
1792        prior.source_weights = src.try_into().expect("source weight arity");
1793        // Mix inherits the sources' average tilt — it is the node that exists
1794        // to combine them, and it is the one column the identity removed.
1795        //
1796        // Wave 3 tried to replace this proxy with a measurement: a
1797        // `branch_width_max` φ coordinate, so that "I like parallel routing"
1798        // would be a thing a user could say and this line could hand back. The
1799        // VIF sweep threw the column out (10.4, and it took every source count
1800        // with it), and the reason is the same identity that removed `n_mix`
1801        // in the first place — the leaf count is `1 + Σ binaries` exactly, so
1802        // a patch cannot gain a mixer without gaining a source. Which means
1803        // this proxy was never a proxy. Wanting more sources *is* wanting more
1804        // binaries, as an equation, and the average below is reading the
1805        // evidence for both. The wave-3 coordinates that did survive
1806        // (`chain_balance`, `frac_sidechained`, `mod_at_source`) describe how
1807        // a patch is arranged rather than how wide it is, and none of them
1808        // maps onto a single production's weight, so none of them belongs
1809        // here: a tilt is a claim about one categorical outcome, and
1810        // "asymmetric" is not an outcome any one production produces.
1811        let binary_tilt = sources.iter().sum::<f64>() / sources.len() as f64;
1812        let (drive, mod_fx) = (g("n_drive"), g("n_mod_fx"));
1813        // `n_filter` and `n_time` are families now too — the eq and the
1814        // vocoder are counted under the first, the granulator and the pitch
1815        // shifter under the second — so every member of each takes the same
1816        // tilt, exactly as the drive and movement families already did.
1817        let (spectral, time) = (g("n_filter"), g("n_time"));
1818        // Wave 2B's three level-shapers share one coefficient for the same
1819        // reason: the evidence never distinguished a compressor from a ducker
1820        // from a gate, so the proposal must not pretend it did.
1821        let dynamics = g("n_dynamics");
1822        let op = tilt_weights(
1823            &prior.op_weights,
1824            &[
1825                binary_tilt,   // mix
1826                spectral,      // filter
1827                drive,         // fold
1828                time,          // delay
1829                mod_fx,        // chorus
1830                g("n_reverb"), // reverb
1831                drive,         // distortion
1832                drive,         // bitcrush
1833                mod_fx,        // phaser
1834                drive,         // ring mod — counted inside n_drive
1835                mod_fx,        // flanger
1836                mod_fx,        // tremolo
1837                mod_fx,        // vibrato
1838                spectral,      // eq — counted inside n_filter
1839                time,          // granular — counted inside n_time
1840                time,          // pitch shift — counted inside n_time
1841                dynamics,      // compressor
1842                dynamics,      // ducker
1843                dynamics,      // gate
1844                spectral,      // vocoder — counted inside n_filter
1845            ],
1846            eta,
1847        );
1848        prior.op_weights = op.try_into().expect("op weight arity");
1849        // "no modulation" carries no tilt — only the filled kinds compete.
1850        // Wave 2C's three take their family's coefficient, on the same rule as
1851        // the op table: the euclidean generator and the two recursive
1852        // productions are all counted inside `n_mod_logic` or `n_mod_shape`,
1853        // and the evidence never separated a quantizer from a slew limiter.
1854        //
1855        // `Op` reads `n_mod_shape` and `Pair` reads `n_mod_logic`, but the
1856        // euclid — which is a *leaf* — reads `n_mod_logic` too, because that
1857        // is the column it is counted in. Tilting it by anything else would
1858        // move the prior in a direction no observation supports.
1859        let (shape, logic) = (g("n_mod_shape"), g("n_mod_logic"));
1860        let md = tilt_weights(
1861            &prior.mod_weights,
1862            &[
1863                0.0,
1864                g("n_lfo"),
1865                g("n_env"),
1866                g("n_rand"),
1867                g("n_follow"),
1868                logic, // euclid — counted inside n_mod_logic
1869                shape, // op
1870                logic, // pair
1871            ],
1872            eta,
1873        );
1874        prior.mod_weights = md.try_into().expect("mod weight arity");
1875        prior
1876    }
1877
1878    /// Posterior probability that pool member `a` beats `b` in a duel
1879    /// (`None` before the first fit).
1880    pub fn predict_duel(&self, a: usize, b: usize) -> Option<f64> {
1881        let p = self.posterior.as_ref()?;
1882        let (pa, pb) = (&self.pool[a].phi_std, &self.pool[b].phi_std);
1883        if pa.is_empty() || pb.is_empty() {
1884            return None;
1885        }
1886        Some(p.prob_prefers(pa, pb))
1887    }
1888
1889    /// Log an implicit preference event (promote, play time, …). Logged
1890    /// only — not yet part of the likelihood.
1891    pub fn log_event(&mut self, kind: &str, id: u64, value: f64) {
1892        self.log_event_detail(kind, id, value, "", Vec::new(), Vec::new());
1893    }
1894
1895    /// The same, carrying the editor's detail and (for a transition) the raw
1896    /// φ on both sides of it. See [`ImplicitEvent`] for why the detail is an
1897    /// opaque string and why none of this reaches the likelihood.
1898    pub fn log_event_detail(
1899        &mut self,
1900        kind: &str,
1901        id: u64,
1902        value: f64,
1903        detail: &str,
1904        phi_before: Vec<f64>,
1905        phi_after: Vec<f64>,
1906    ) {
1907        self.events.push(ImplicitEvent {
1908            kind: kind.into(),
1909            id,
1910            value,
1911            session: self.session,
1912            detail: detail.into(),
1913            phi_before,
1914            phi_after,
1915        });
1916    }
1917
1918    /// Name (or rename; empty clears) an aligned style index.
1919    pub fn set_style_name(&mut self, k: usize, name: &str) {
1920        if k >= 16 {
1921            return;
1922        }
1923        if self.style_names.len() <= k {
1924            self.style_names.resize(k + 1, String::new());
1925        }
1926        self.style_names[k] = name.trim().chars().take(24).collect();
1927    }
1928
1929    /// Run locked MH refinement from one seed. Returns the end state if it
1930    /// differs from the seed.
1931    fn refine_one<R: Rng>(
1932        &self,
1933        rng: &mut R,
1934        seed: &PatchTree,
1935        locked: &HashSet<String>,
1936        steps: usize,
1937    ) -> Option<PatchTree> {
1938        let (posterior, standardizer) = match (&self.posterior, &self.standardizer) {
1939            (Some(p), Some(s)) => (Arc::clone(p), Arc::clone(s)),
1940            _ => return None,
1941        };
1942        let fitness = SurrogateFitness {
1943            posterior,
1944            standardizer,
1945            phrase: self.cfg.phrase.clone(),
1946            memo: self.memo.clone(),
1947        };
1948        let model = EvolutionModel::new(self.biased_prior(), fitness).with_beta(self.cfg.beta);
1949        let mut chain = EvolutionChain::new(model);
1950        let mut trace = chain.init_from(seed)?;
1951
1952        // Scale steps for proposals wasted on locked sites. The kernel picks a
1953        // target site uniformly over all of them, so with a fraction `f` free
1954        // only `f` of the proposals can be accepted and the walk needs `1/f`
1955        // times as many steps to travel as far.
1956        //
1957        // **The cap is a cost bound, not a correction**, and it is stated
1958        // rather than left silent. Past 75% of sites locked, `LOCK_SCALE_CAP`
1959        // stops the compensation short — a patch with 90% of its sites pinned
1960        // would otherwise ask for ten times the budget, and a `⚡ evolve from
1961        // this` on a heavily-pinned patch is a button press with a person
1962        // waiting behind it. So a very heavily locked walk *does* explore less
1963        // than the config nominally buys. That is the intended trade; the thing
1964        // to avoid is believing otherwise.
1965        let total_sites = trace.choices.len().max(1);
1966        let locked_present = trace
1967            .choices
1968            .keys()
1969            .filter(|a| locked.contains(&***a))
1970            .count();
1971        let free = total_sites.saturating_sub(locked_present).max(1);
1972        let factor = (total_sites as f64 / free as f64).min(LOCK_SCALE_CAP);
1973        let steps = ((steps as f64) * factor).ceil() as usize;
1974
1975        let mut current = seed.clone();
1976        // The elite archive, and it is **free**.
1977        //
1978        // Every trace the kernel hands back is already scored under the target
1979        // program, so `total_log_weight()` *is* `log π_β = log p_grammar +
1980        // β·E[u]` for the state it accompanies — no extra model execution, no
1981        // extra featurization, one f64 compare per step.
1982        //
1983        // Scored on the target rather than on fitness alone, which is the
1984        // choice worth stating. Taking the argmax of `E[u]` would discard the
1985        // parsimony half of the very distribution the walk is sampling, and it
1986        // would do so with a bias: a bigger term has more modules to score
1987        // well with, so fitness-argmax systematically returns the largest tree
1988        // the walk touched. `log π_β` is what the walk is climbing, so it is
1989        // what "the best point this walk found" has to mean.
1990        //
1991        // The seed is in the archive. A walk that never improves on where it
1992        // started therefore returns the seed and is filtered to `None` below,
1993        // instead of injecting whatever it happened to be standing on at step
1994        // 40 — which is what `Last` does, and is the thing being A/B'd.
1995        let mut best: Option<(f64, PatchTree)> = match self.cfg.refine_keep {
1996            RefineKeep::Last => None,
1997            RefineKeep::Best => Some((trace.total_log_weight(), seed.clone())),
1998        };
1999        for _ in 0..steps {
2000            let (g, t) = chain.step(rng, &trace);
2001            if Self::violates_locks(&trace, &t, locked) {
2002                continue; // reject outside the kernel; stay at `trace`
2003            }
2004            if let Some((best_w, best_tree)) = &mut best {
2005                let w = t.total_log_weight();
2006                if w > *best_w {
2007                    *best_w = w;
2008                    *best_tree = g.clone();
2009                }
2010            }
2011            current = g;
2012            trace = t;
2013        }
2014        if let Some((_, best_tree)) = best {
2015            current = best_tree;
2016        }
2017        // The mutation boundary, and the reason the clamp is *here* rather than
2018        // at the knob that draws the number: everything downstream of this line
2019        // — φ, the observation log, the faceplate, the exported PNG — takes the
2020        // term as given, so a value that leaves this function wrong is wrong in
2021        // six places by the time anyone can see it.
2022        //
2023        // The kernel should never produce one. Every continuous site is
2024        // `Uniform(0,1)`, whose `log_prob` is −∞ outside the unit interval, so
2025        // a proposal that escapes scores `log α = −∞` and is rejected — and
2026        // that is measured, not assumed: `auracle-grammar --example
2027        // mh_escape` runs 8 chains × 20 000 single-site transitions through
2028        // this exact kernel and observes zero escapes. So this is a belt on a
2029        // proven brace, costing one trace walk per accepted child, and its real
2030        // job is to be the line that has to be deleted before the invariant can
2031        // be broken again.
2032        debug_assert_eq!(
2033            current.domain_violations().len(),
2034            0,
2035            "MH seated an out-of-domain site: {:?}",
2036            current.domain_violations()
2037        );
2038        current.clamp_domains();
2039        (current != *seed).then_some(current)
2040    }
2041
2042    /// Insert a candidate (evicting the worst if full, never `protect`).
2043    /// Returns the new id, or `None` if the newcomer ranks below the evictee.
2044    fn insert_candidate(
2045        &mut self,
2046        tree: PatchTree,
2047        origin: Origin,
2048        protect: Option<u64>,
2049    ) -> Option<u64> {
2050        let standardizer = self.standardizer.as_ref()?;
2051        // Memoized: refinement and the edit bench both featurize the tree they
2052        // hand here, so on every one of those paths this is a hit.
2053        let want_audio = self.wants_admitted_audio();
2054        let (cf, fresh) = featurize_memo(&tree, &self.cfg.phrase, &self.memo, want_audio).ok()?;
2055        let phi_std = standardizer.transform(&cf.features.phi());
2056        let mean_new = self.utility_of(&phi_std);
2057        if self.pool.len() >= self.cfg.pool_size {
2058            // Rank un-standardized members as *worst*, explicitly, rather
2059            // than letting `utility_of` score them 0.0 and land them
2060            // mid-pack above genuinely-disliked patches. Today this cannot
2061            // happen — the `?` above means a standardizer exists, and every
2062            // path that admits a candidate under one also transforms its φ —
2063            // but that is an invariant three functions away, and the same
2064            // "empty φ scores exactly zero" reasoning already produced one
2065            // live bug in duel selection. Cheaper to be unconditionally right
2066            // here than to rely on the invariant holding after the next edit.
2067            let rank = |c: &Candidate| (!c.phi_std.is_empty(), self.utility_of(&c.phi_std));
2068            let worst = self
2069                .pool
2070                .iter()
2071                .enumerate()
2072                .filter(|(_, c)| Some(c.id) != protect && !c.pinned)
2073                .min_by(|(_, x), (_, y)| {
2074                    let (sx, ux) = rank(x);
2075                    let (sy, uy) = rank(y);
2076                    sx.cmp(&sy).then(ux.total_cmp(&uy))
2077                })
2078                .map(|(i, c)| (i, self.utility_of(&c.phi_std)));
2079            match worst {
2080                Some((worst_idx, worst_mean)) => {
2081                    // Hand edits always land (the user asked for them);
2082                    // refined candidates must earn their slot.
2083                    if origin == Origin::Refined && mean_new <= worst_mean {
2084                        return None;
2085                    }
2086                    self.pool.swap_remove(worst_idx);
2087                }
2088                None => return None,
2089            }
2090        }
2091        let id = self.alloc_id();
2092        let render = self.admitted_render(&tree, &cf.features, fresh);
2093        self.pool.push(Candidate {
2094            id,
2095            tree: settled(tree),
2096            phi_std,
2097            key: cf.key,
2098            render,
2099            features: cf.features,
2100            origin,
2101            name: None,
2102            pinned: false,
2103        });
2104        Some(id)
2105    }
2106
2107    /// Taste-guided refinement: run fugue-evo typed MH on the Boltzmann
2108    /// target from each of the top seeds, and add improved, vetted, novel
2109    /// candidates to the pool (evicting the worst if full). Each injection
2110    /// is recorded as a lineage event.
2111    pub fn refine<R: Rng>(&mut self, rng: &mut R) {
2112        for parent_id in self.refine_begin() {
2113            self.refine_seed(rng, parent_id);
2114        }
2115    }
2116
2117    /// Open a generation and return the parent ids it will refine from, best
2118    /// first. Empty if there is nothing to refine toward yet (no posterior),
2119    /// in which case the generation counter is **not** advanced.
2120    ///
2121    /// This exists so a caller can drive refinement one seed at a time and
2122    /// report progress between seeds. A whole generation is tens of seconds of
2123    /// render-bound work — running it as one opaque call is what made the app
2124    /// look hung.
2125    pub fn refine_begin(&mut self) -> Vec<u64> {
2126        if self.posterior.is_none() || self.standardizer.is_none() {
2127            return Vec::new();
2128        }
2129        self.generation += 1;
2130        self.ranked()
2131            .iter()
2132            .take(self.cfg.refine_seeds)
2133            .map(|&(i, _, _)| self.pool[i].id)
2134            .collect()
2135    }
2136
2137    /// Refine from one seed of the open generation. Returns the injected child
2138    /// id, or `None` if the walk was rejected or landed on a patch the pool
2139    /// already holds.
2140    pub fn refine_seed<R: Rng>(&mut self, rng: &mut R, parent_id: u64) -> Option<u64> {
2141        let seed = self.pool[self.find(parent_id)?].tree.clone();
2142        let no_locks = HashSet::new();
2143        let end = self.refine_one(rng, &seed, &no_locks, self.cfg.refine_steps)?;
2144        if self.pool.iter().any(|c| c.tree == end) {
2145            return None;
2146        }
2147        self.record_child(parent_id, &seed, end, "refine", None)
2148    }
2149
2150    /// Locked refinement from one explicit seed candidate: evolve everything
2151    /// *except* the locked addresses. Returns the injected child id.
2152    pub fn refine_from<R: Rng>(
2153        &mut self,
2154        rng: &mut R,
2155        seed_id: u64,
2156        locked: &[String],
2157    ) -> Option<u64> {
2158        let seed = self.pool[self.find(seed_id)?].tree.clone();
2159        let locked: HashSet<String> = locked.iter().cloned().collect();
2160        self.generation += 1;
2161        let end = self.refine_one(rng, &seed, &locked, self.cfg.refine_steps)?;
2162        if self.pool.iter().any(|c| c.tree == end) {
2163            return None;
2164        }
2165        self.record_child(seed_id, &seed, end, "refine", Some(seed_id))
2166    }
2167
2168    /// Commit a hand-edited tree as a new candidate. If `original_id` is
2169    /// given, a lineage event links them; `outcome` says what the player
2170    /// reported about the pair, and only a *told* outcome writes an
2171    /// observation.
2172    pub fn commit_edit(
2173        &mut self,
2174        original_id: Option<u64>,
2175        tree: PatchTree,
2176        outcome: EditOutcome,
2177    ) -> Option<u64> {
2178        if let Some(i) = self.pool.iter().position(|c| c.tree == tree) {
2179            // The edit landed on a patch the bank already holds, so there is
2180            // no new candidate to insert. There *was* still a comparison: the
2181            // player heard two patches and picked one, and discarding that
2182            // answer because the winner happened to already exist would throw
2183            // away a real vote on the grounds of a bookkeeping collision. The
2184            // pair is scored against the twin instead.
2185            let (existing, told) = (self.pool[i].id, outcome.told());
2186            if let (Some(oid), Some((edited_won, provenance))) = (original_id, told) {
2187                if let Some(pi) = self.find(oid) {
2188                    if existing != oid {
2189                        self.record_duel_as(i, pi, edited_won, provenance);
2190                    }
2191                }
2192            }
2193            return None;
2194        }
2195        let original = original_id.and_then(|id| self.find(id)).map(|i| {
2196            (
2197                self.pool[i].id,
2198                self.pool[i].tree.clone(),
2199                self.pool[i].phi_std.clone(),
2200            )
2201        });
2202        let child_id = self.insert_candidate(tree, Origin::Edited, original_id)?;
2203        if let Some((pid, ptree, pphi)) = original {
2204            let ci = self.find(child_id).expect("just inserted");
2205            let (ctree, cphi) = (self.pool[ci].tree.clone(), self.pool[ci].phi_std.clone());
2206            self.lineage.push(LineageEvent {
2207                generation: self.generation,
2208                kind: "edit".into(),
2209                parent_id: pid,
2210                child_id,
2211                diff: tree_diff(&ptree, &ctree),
2212                parent_utility: self.utility_of(&pphi),
2213                child_utility: self.utility_of(&cphi),
2214            });
2215            // A committed edit is a genuine one-step-ahead question — the
2216            // model has never seen this tree — so it goes through the same
2217            // forecast-then-observe path a dealt duel does, in the same
2218            // (edit, original) order, and carries the tag that lets a
2219            // self-report be scored against a heard comparison rather than
2220            // averaged into it.
2221            if let Some((edited_won, provenance)) = outcome.told() {
2222                if let (Some(ci), Some(pi)) = (self.find(child_id), self.find(pid)) {
2223                    self.record_duel_as(ci, pi, edited_won, provenance);
2224                }
2225            }
2226        }
2227        Some(child_id)
2228    }
2229
2230    fn record_child(
2231        &mut self,
2232        parent_id: u64,
2233        seed: &PatchTree,
2234        mut end: PatchTree,
2235        kind: &str,
2236        protect: Option<u64>,
2237    ) -> Option<u64> {
2238        // The one place a refined child meets its seed, and therefore the one
2239        // place its node identities can be recovered.
2240        //
2241        // `refine_one` runs typed MH over the *trace*, and every accepted step
2242        // rebuilds the whole genome through `crate::genome`'s decoder — a trace
2243        // is a map from address to value and has no room for a uid, so what
2244        // comes back is structurally almost the seed and completely anonymous.
2245        // Without this line every ⚡ would look to the panel like a brand-new
2246        // patch: locks gone, hand-placed positions gone, selection gone, on the
2247        // one action the whole instrument is built around. Positions and locks
2248        // are the point of uids, and evolution is the point of auracle.
2249        end.inherit_uids(seed);
2250        let parent_phi = self
2251            .find(parent_id)
2252            .map(|i| self.pool[i].phi_std.clone())
2253            .unwrap_or_default();
2254        let child_id = self.insert_candidate(end, Origin::Refined, protect)?;
2255        let ci = self.find(child_id).expect("just inserted");
2256        let (ctree, cphi) = (self.pool[ci].tree.clone(), self.pool[ci].phi_std.clone());
2257        self.lineage.push(LineageEvent {
2258            generation: self.generation,
2259            kind: kind.into(),
2260            parent_id,
2261            child_id,
2262            diff: tree_diff(seed, &ctree),
2263            parent_utility: self.utility_of(&parent_phi),
2264            child_utility: self.utility_of(&cphi),
2265        });
2266        Some(child_id)
2267    }
2268
2269    /// Pool indices ranked by posterior-mean mixture utility (descending);
2270    /// with no posterior, arbitrary order with zero scores.
2271    pub fn ranked(&self) -> Vec<(usize, f64, f64)> {
2272        let mut rows: Vec<(usize, f64, f64)> = self
2273            .pool
2274            .iter()
2275            .enumerate()
2276            .map(|(i, c)| match &self.posterior {
2277                Some(p) if !c.phi_std.is_empty() => {
2278                    let (m, s) = p.utility_mix(&c.phi_std);
2279                    (i, m, s)
2280                }
2281                _ => (i, 0.0, 0.0),
2282            })
2283            .collect();
2284        rows.sort_by(|a, b| b.1.total_cmp(&a.1));
2285        rows
2286    }
2287
2288    /// Choose the next duel by **expected information gain about θ** (BALD),
2289    /// traded off against how pleasant the duel is to answer and penalized for
2290    /// repetition, then sampled from a softmax rather than argmaxed.
2291    ///
2292    /// Returns pool indices `(a, b)`; `None` if fewer than two candidates are
2293    /// standardized. See [`Engine::next_duel_full`] for the annotated form.
2294    ///
2295    /// ## Why not dueling Thompson sampling
2296    ///
2297    /// The obvious acquisition here — draw two posterior samples, duel each
2298    /// one's champion — is a real algorithm, correctly implemented, and the
2299    /// wrong objective. DTS is **best-arm identification**: it converges on
2300    /// finding the single top patch. What this system needs from a duel is
2301    /// *information about θ*, because θ is what reshapes the proposal
2302    /// distribution and paints the taste map. Those goals diverge sharply.
2303    /// The Fisher information in one Bradley–Terry duel is
2304    ///
2305    /// ```text
2306    /// I(θ) = p(1−p) · Δ Δᵀ ,   Δ = φ_a − φ_b ,   p = σ(θ·Δ)
2307    /// ```
2308    ///
2309    /// which scales with `p(1−p)` **and** with `‖Δ‖²`. DTS maximizes the
2310    /// first (champions tie at p ≈ 0.5) while actively *minimizing* the
2311    /// second: two champions of the same concentrating posterior are two
2312    /// high-utility patches, which in a 48-member pool means two *similar*
2313    /// patches. It systematically picks the least informative near-tie
2314    /// available. And once the draw set concentrates, both champions become
2315    /// the same index and the user is shown top-1 vs top-2 over and over.
2316    ///
2317    /// BALD scores the mutual information between the outcome and θ,
2318    /// `I = H(E_s[p_s]) − E_s[H(p_s)]` — high exactly when the posterior
2319    /// *disagrees with itself* about who wins, which is the definition of a
2320    /// question worth asking.
2321    ///
2322    /// Measured against DTS on the synthetic user (10 paired seeds, 72
2323    /// duels): pool-ranking correlation +0.101 ± 0.058, predictive excess
2324    /// −0.040 ± 0.017 nats. Measured against *uniformly random* pairing: no
2325    /// difference outside noise on any metric. See [`Acquisition`] for the
2326    /// full table and for why `Bald` is still the default.
2327    pub fn next_duel<R: Rng>(&mut self, rng: &mut R) -> Option<(usize, usize)> {
2328        self.next_duel_full(rng).map(|d| (d.a, d.b))
2329    }
2330
2331    /// [`Engine::next_duel`] with the reasoning attached: which rule chose the
2332    /// pair, its expected information gain in nats, and whether it is one of
2333    /// the uniformly-random check duels that calibration is scored on.
2334    pub fn next_duel_full<R: Rng>(&mut self, rng: &mut R) -> Option<DuelChoice> {
2335        // Un-standardized candidates score utility exactly 0 (`dot` over an
2336        // empty vector), which beats every real utility once a user has killed
2337        // enough patches — they must not be selectable, the same guard
2338        // `ranked()` applies.
2339        let cands: Vec<usize> = (0..self.pool.len())
2340            .filter(|&i| !self.pool[i].phi_std.is_empty())
2341            .collect();
2342        if cands.len() < 2 {
2343            return None;
2344        }
2345        let uniform = |rng: &mut R| -> (usize, usize) {
2346            let i = rng.gen_range(0..cands.len());
2347            let mut j = rng.gen_range(0..cands.len() - 1);
2348            if j >= i {
2349                j += 1;
2350            }
2351            (cands[i], cands[j])
2352        };
2353
2354        let check = self.cfg.duel_check_every > 0
2355            && self.duels_shown > 0
2356            && self.duels_shown.is_multiple_of(self.cfg.duel_check_every);
2357
2358        let choice = match (&self.posterior, check) {
2359            // No taste yet, or a scheduled check duel: uniform at random.
2360            // A uniform pair *is* a calibration check, so it is tagged as one
2361            // whether it was scheduled or is simply how this engine picks
2362            // every duel. Under the default rule that makes the unbiased
2363            // subsample the entire sample, which is the whole reason to
2364            // prefer it: the reliability diagram needs no asterisk.
2365            (None, _) => {
2366                let (a, b) = uniform(rng);
2367                DuelChoice {
2368                    a,
2369                    b,
2370                    info_gain: 0.0,
2371                    random_check: true,
2372                    method: "random",
2373                }
2374            }
2375            (Some(_), true) => {
2376                let (a, b) = uniform(rng);
2377                DuelChoice {
2378                    a,
2379                    b,
2380                    info_gain: 0.0,
2381                    random_check: true,
2382                    method: "check",
2383                }
2384            }
2385            (Some(_), false) if self.cfg.acquisition == Acquisition::Random => {
2386                let (a, b) = uniform(rng);
2387                DuelChoice {
2388                    a,
2389                    b,
2390                    info_gain: 0.0,
2391                    random_check: true,
2392                    method: "random",
2393                }
2394            }
2395            (Some(posterior), false) if self.cfg.acquisition == Acquisition::Thompson => {
2396                let (a, b) = thompson_pair(posterior, &self.pool, &cands, rng);
2397                DuelChoice {
2398                    a,
2399                    b,
2400                    info_gain: 0.0,
2401                    random_check: false,
2402                    method: "thompson",
2403                }
2404            }
2405            (Some(posterior), false) => {
2406                let (a, b, info) = self.bald_pair(posterior, &cands, rng);
2407                DuelChoice {
2408                    a,
2409                    b,
2410                    info_gain: info,
2411                    random_check: false,
2412                    method: "bald",
2413                }
2414            }
2415        };
2416
2417        self.duels_shown += 1;
2418        let key = pair_key(self.pool[choice.a].id, self.pool[choice.b].id);
2419        *self.shown_pairs.entry(key).or_insert(0) += 1;
2420        *self.shown_candidates.entry(key.0).or_insert(0) += 1;
2421        *self.shown_candidates.entry(key.1).or_insert(0) += 1;
2422        self.last_check_pair = choice.random_check.then_some(key);
2423        Some(choice)
2424    }
2425
2426    /// The BALD scan itself. Utilities are precomputed once per candidate per
2427    /// draw (`S × |pool|`), then every pair is scored from that table — the
2428    /// whole sweep is a few hundred thousand sigmoids, milliseconds in wasm.
2429    fn bald_pair<R: Rng>(
2430        &self,
2431        posterior: &TastePosterior,
2432        cands: &[usize],
2433        rng: &mut R,
2434    ) -> (usize, usize, f64) {
2435        let s_n = posterior.samples.len();
2436        if s_n == 0 {
2437            let i = rng.gen_range(0..cands.len());
2438            let mut j = rng.gen_range(0..cands.len() - 1);
2439            if j >= i {
2440                j += 1;
2441            }
2442            return (cands[i], cands[j], 0.0);
2443        }
2444        // u[s][c] over the *standardized* pool.
2445        let u: Vec<Vec<f64>> = posterior
2446            .samples
2447            .iter()
2448            .map(|s| {
2449                cands
2450                    .iter()
2451                    .map(|&i| s.utility_mix(&self.pool[i].phi_std))
2452                    .collect()
2453            })
2454            .collect();
2455        let w: Vec<f64> = (0..s_n).map(|s| posterior.weight(s)).collect();
2456        let mean_u: Vec<f64> = (0..cands.len())
2457            .map(|c| (0..s_n).map(|s| w[s] * u[s][c]).sum())
2458            .collect();
2459        // The enjoyment term is scored on **pool-standardized** utility, not
2460        // raw utility. Raw utility has no fixed scale: it grows without bound
2461        // as the posterior sharpens, so a fixed λ against it starts as a
2462        // gentle nudge and ends up swamping the information term entirely —
2463        // at which point the acquisition function has silently turned back
2464        // into the best-arm rule this one replaced. Standardized, λ means the
2465        // same thing at duel 10 and duel 200.
2466        let u_mu = mean_u.iter().sum::<f64>() / mean_u.len().max(1) as f64;
2467        let u_sd = (mean_u.iter().map(|u| (u - u_mu) * (u - u_mu)).sum::<f64>()
2468            / mean_u.len().max(1) as f64)
2469            .sqrt()
2470            .max(1e-9);
2471        let z_u: Vec<f64> = mean_u.iter().map(|u| (u - u_mu) / u_sd).collect();
2472
2473        let lambda = self.cfg.duel_utility_weight;
2474        let gamma = self.cfg.duel_repeat_penalty;
2475        let rho = self.cfg.duel_exposure_penalty;
2476        // How often each candidate has been *put in front of the user*, by any
2477        // pairing. See `duel_exposure_penalty`: without this the top-utility
2478        // candidate is nominated over and over through pairs that are all
2479        // technically distinct.
2480        let seen: Vec<f64> = cands
2481            .iter()
2482            .map(|&i| {
2483                self.shown_candidates
2484                    .get(&self.pool[i].id)
2485                    .copied()
2486                    .unwrap_or(0) as f64
2487            })
2488            .collect();
2489        let mut best = Vec::with_capacity(cands.len() * cands.len() / 2);
2490        for ci in 0..cands.len() {
2491            for cj in (ci + 1)..cands.len() {
2492                let mut p_bar = 0.0;
2493                let mut mean_h = 0.0;
2494                for s in 0..s_n {
2495                    let p = sigmoid(u[s][ci] - u[s][cj]);
2496                    p_bar += w[s] * p;
2497                    mean_h += w[s] * binary_entropy(p);
2498                }
2499                let info = binary_entropy(p_bar) - mean_h;
2500                let shown = self
2501                    .shown_pairs
2502                    .get(&pair_key(self.pool[cands[ci]].id, self.pool[cands[cj]].id))
2503                    .copied()
2504                    .unwrap_or(0) as f64;
2505                let j = info + lambda * (z_u[ci] + z_u[cj]) / 2.0
2506                    - gamma * shown
2507                    - rho * (seen[ci] + seen[cj]);
2508                best.push((ci, cj, j, info));
2509            }
2510        }
2511        // Softmax over the objective, at a temperature set by the objective's
2512        // *own* spread. An absolute temperature is a bet on how far apart the
2513        // scores happen to be, and at 0.05 nats against a spread of several
2514        // tenths this "softmax" was an argmax — which is exactly the best-arm
2515        // lock-in BALD exists to avoid.
2516        let j_mu = best.iter().map(|x| x.2).sum::<f64>() / best.len().max(1) as f64;
2517        let j_sd = (best
2518            .iter()
2519            .map(|x| (x.2 - j_mu) * (x.2 - j_mu))
2520            .sum::<f64>()
2521            / best.len().max(1) as f64)
2522            .sqrt();
2523        let t = (self.cfg.duel_temperature * j_sd).max(1e-9);
2524        let max_j = best.iter().map(|x| x.2).fold(f64::NEG_INFINITY, f64::max);
2525        let total: f64 = best.iter().map(|x| ((x.2 - max_j) / t).exp()).sum();
2526        let mut r = rng.gen::<f64>() * total;
2527        for &(ci, cj, j, info) in &best {
2528            r -= ((j - max_j) / t).exp();
2529            if r <= 0.0 {
2530                return (cands[ci], cands[cj], info);
2531            }
2532        }
2533        let &(ci, cj, _, info) = best.last().expect("at least one pair");
2534        (cands[ci], cands[cj], info)
2535    }
2536
2537    /// Append one feedback event and fold it into the current posterior.
2538    ///
2539    /// `raw` is what the log keeps — un-standardized values plus the names
2540    /// they belong to, so the log stays interpretable across feature-set
2541    /// changes. `standardized` is the same event on the current scale, used
2542    /// only to reweight the existing posterior draws by sequential importance
2543    /// sampling: an O(S) update that makes the *next* duel respond to this
2544    /// one instead of waiting for the next multi-second MCMC refit.
2545    fn observe(&mut self, raw: Feedback, standardized: Feedback) {
2546        self.observe_as(raw, standardized, Provenance::Duel);
2547    }
2548
2549    /// The same, carrying how the answer was collected. The tag reaches the
2550    /// log and nothing else: `standardized` goes into the posterior update
2551    /// untouched, so two observations that differ only in provenance move the
2552    /// posterior identically. Provenance is evidence *about the evidence*, and
2553    /// weighting by it would be a modeling claim with no measurement behind it
2554    /// — see [`Provenance`].
2555    fn observe_as(&mut self, raw: Feedback, standardized: Feedback, provenance: Provenance) {
2556        self.log.push(Observation::tagged(
2557            raw,
2558            self.session,
2559            &phi_names(),
2560            provenance,
2561        ));
2562        if self.cfg.sis_between_fits {
2563            if let Some(p) = &self.posterior {
2564                let mut updated = p.reweighted(&standardized, self.session);
2565                // Degenerate weights make the acquisition function read a
2566                // one-point "posterior" as certainty. Resample back to a
2567                // uniform set rather than let that happen; the impoverishment
2568                // is bounded by how soon the next full refit lands.
2569                if updated.ess() < updated.samples.len() as f64 / 2.0 {
2570                    updated = updated.resampled();
2571                    self.resamples_since_fit += 1;
2572                }
2573                self.posterior = Some(Arc::new(updated));
2574            }
2575        }
2576    }
2577
2578    /// Record a duel outcome between two pool members (by pool index).
2579    ///
2580    /// The out-of-sample forecast is scored *here*, before the observation is
2581    /// appended — the model has to commit before it is told the answer, which
2582    /// is what makes [`Engine::calibration`] prequential rather than a
2583    /// in-sample self-assessment.
2584    pub fn record_duel(&mut self, a: usize, b: usize, chose_a: bool) {
2585        self.record_duel_as(a, b, chose_a, Provenance::Duel);
2586    }
2587
2588    /// The same, for a pair the app assembled itself rather than dealt — a
2589    /// hand edit against the patch it was edited from. One code path, so the
2590    /// editor's answers are scored, logged and folded into the posterior by
2591    /// exactly the machinery a dealt duel is, and differ only in the tag that
2592    /// says where they came from.
2593    fn record_duel_as(&mut self, a: usize, b: usize, chose_a: bool, provenance: Provenance) {
2594        if let Some(p_a) = self.predict_duel(a, b) {
2595            let key = pair_key(self.pool[a].id, self.pool[b].id);
2596            self.forecasts.push(Forecast {
2597                p_a,
2598                chose_a,
2599                random_check: self.last_check_pair == Some(key),
2600                provenance,
2601            });
2602        }
2603        let raw = Feedback::Duel {
2604            a: self.pool[a].features.phi(),
2605            b: self.pool[b].features.phi(),
2606            chose_a,
2607        };
2608        let std = Feedback::Duel {
2609            a: self.pool[a].phi_std.clone(),
2610            b: self.pool[b].phi_std.clone(),
2611            chose_a,
2612        };
2613        self.observe_as(raw, std, provenance);
2614    }
2615
2616    /// Record a keep/kill decision on a pool member (by pool index).
2617    pub fn record_keep(&mut self, idx: usize, kept: bool) {
2618        let raw = Feedback::KeepKill {
2619            x: self.pool[idx].features.phi(),
2620            kept,
2621        };
2622        let std = Feedback::KeepKill {
2623            x: self.pool[idx].phi_std.clone(),
2624            kept,
2625        };
2626        self.observe(raw, std);
2627    }
2628
2629    /// Record a star rating on a pool member (by pool index).
2630    pub fn record_stars(&mut self, idx: usize, rating: u8) {
2631        let raw = Feedback::Stars {
2632            x: self.pool[idx].features.phi(),
2633            rating,
2634        };
2635        let std = Feedback::Stars {
2636            x: self.pool[idx].phi_std.clone(),
2637            rating,
2638        };
2639        self.observe(raw, std);
2640    }
2641
2642    /// Prequential calibration over every duel forecast so far.
2643    pub fn calibration(&self) -> Calibration {
2644        calibration(&self.forecasts)
2645    }
2646
2647    /// Exact per-feature decomposition of a candidate's utility under the lens
2648    /// that claims it (B9 — see [`Explanation`]).
2649    pub fn explain(&self, id: u64) -> Option<Explanation> {
2650        let i = self.find(id)?;
2651        self.explain_std(id, &self.pool[i].phi_std.clone())
2652    }
2653
2654    /// The same decomposition for a φ that is **not** a pool member — the
2655    /// workbench, which is a patch under the player's hands and not a
2656    /// candidate until they commit it.
2657    ///
2658    /// This is what makes the readout above the rack honest. The WHY line used
2659    /// to be fetched once, for the candidate that was loaded, and then went on
2660    /// describing it through any number of edits: it named features of a patch
2661    /// the player had already edited away. The bench re-featurizes on every
2662    /// edit anyway, so the true decomposition is a dot product away — there
2663    /// was never a cost reason for the stale one.
2664    ///
2665    /// Takes **raw** φ and standardizes here, because raw is what the
2666    /// featurizer produces and what the log stores; θ is denominated in the
2667    /// standardizer, so the transform is not optional.
2668    pub fn explain_phi(&self, phi_raw: &[f64]) -> Option<Explanation> {
2669        let sz = self.standardizer.as_ref()?;
2670        self.explain_std(0, &sz.transform(phi_raw))
2671    }
2672
2673    fn explain_std(&self, id: u64, phi: &[f64]) -> Option<Explanation> {
2674        let p = self.posterior.as_ref()?;
2675        if phi.is_empty() {
2676            return None;
2677        }
2678        let responsibilities = p.responsibilities(phi);
2679        let style = responsibilities
2680            .iter()
2681            .enumerate()
2682            .max_by(|(_, x), (_, y)| x.total_cmp(y))
2683            .map(|(k, _)| k)
2684            .unwrap_or(0);
2685        let theta = p.theta_mean(style);
2686        let names = phi_names();
2687        let mut contributions: Vec<Contribution> = names
2688            .iter()
2689            .zip(&theta)
2690            .zip(phi)
2691            .map(|((name, t), x)| Contribution {
2692                name: name.clone(),
2693                theta: *t,
2694                phi_std: *x,
2695                contribution: t * x,
2696            })
2697            .collect();
2698        contributions.sort_by(|a, b| b.contribution.abs().total_cmp(&a.contribution.abs()));
2699        // `utility`/`utility_std` describe the lens quantity the contributions
2700        // sum to; `mix_utility` is what the bank is sorted by. Both are
2701        // returned because they are genuinely different claims and the caller
2702        // needs to know which one it is drawing.
2703        let (utility, utility_std) = p.utility(phi, style);
2704        Some(Explanation {
2705            id,
2706            style,
2707            style_name: self.style_names.get(style).cloned().unwrap_or_default(),
2708            utility,
2709            utility_std,
2710            mix_utility: p.utility_mix(phi).0,
2711            responsibility: responsibilities.get(style).copied().unwrap_or(0.0),
2712            contributions,
2713        })
2714    }
2715
2716    /// Musical display names for the whole pool, unique across it. Keyed by
2717    /// candidate id; a user-given name always wins.
2718    ///
2719    /// User and preset names are claimed **first and through the same
2720    /// registry** as generated ones. Substituting them afterwards, as this
2721    /// once did, let a preset called `Glass Pad` and a generated `Glass Pad`
2722    /// both survive into the bank: the preset occupied the name without ever
2723    /// competing for it.
2724    pub fn display_names(&self) -> HashMap<u64, String> {
2725        let scale = NameScale::fit(self.pool.iter().map(|c| &c.features));
2726        let mut taken: HashSet<String> = HashSet::new();
2727        let mut out: HashMap<u64, String> = HashMap::new();
2728
2729        // Explicit names first — they are not negotiable, so they get to
2730        // reserve their spelling before anything is generated.
2731        for c in &self.pool {
2732            if let Some(name) = &c.name {
2733                out.insert(c.id, claim_name(name, &mut taken));
2734            }
2735        }
2736        // Then generated ones, in id order: a patch's numeral must not
2737        // reshuffle when the pool is re-ranked underneath it.
2738        let mut rest: Vec<&Candidate> = self.pool.iter().filter(|c| c.name.is_none()).collect();
2739        rest.sort_by_key(|c| c.id);
2740        for c in rest {
2741            out.insert(c.id, claim_name(&scale.name(&c.features), &mut taken));
2742        }
2743        out
2744    }
2745
2746    /// Name (or rename; empty clears) a candidate.
2747    pub fn set_name(&mut self, id: u64, name: &str) {
2748        if let Some(i) = self.find(id) {
2749            let trimmed = name.trim();
2750            self.pool[i].name = (!trimmed.is_empty()).then(|| trimmed.chars().take(40).collect());
2751        }
2752    }
2753
2754    /// How many patches may be pinned at once: a quarter of the pool.
2755    ///
2756    /// The pool is the model's *working set*, not storage — duel pairing is
2757    /// uniform over it and refinement seeds from the top of `ranked()` — so
2758    /// pins are spent capacity, and the only wholly wasted duel is one where
2759    /// both sides are pinned. At a quarter of the pool that is ~6% of pairs,
2760    /// with three quarters of the pool still free to churn; at half it is 25%.
2761    /// A quarter buys the user far more than they lose.
2762    ///
2763    /// The cap also keeps "everything is pinned" unreachable, which matters
2764    /// because that state has no honest report: it surfaces as
2765    /// [`Engine::insert_candidate`] returning `None`, which every caller
2766    /// already renders as "no proposal beat its parent" — a statement about
2767    /// the search that would then be a lie about storage.
2768    pub fn pin_cap(&self) -> usize {
2769        (self.cfg.pool_size / 4).max(1)
2770    }
2771
2772    /// How many pool members are currently pinned.
2773    pub fn pinned_count(&self) -> usize {
2774        self.pool.iter().filter(|c| c.pinned).count()
2775    }
2776
2777    /// Pin or unpin a patch against eviction. Returns `false` when the id is
2778    /// unknown, or when pinning would exceed [`Engine::pin_cap`] — callers are
2779    /// expected to say which, rather than letting the control fail silently.
2780    ///
2781    /// Records **no observation**: a pin says what the user wants to keep, not
2782    /// what they think of it. See [`Candidate::pinned`].
2783    pub fn set_pinned(&mut self, id: u64, pinned: bool) -> bool {
2784        let Some(i) = self.find(id) else {
2785            return false;
2786        };
2787        if pinned && !self.pool[i].pinned && self.pinned_count() >= self.pin_cap() {
2788            return false;
2789        }
2790        self.pool[i].pinned = pinned;
2791        true
2792    }
2793
2794    /// Insert a named preset into the pool (protected from immediate
2795    /// eviction pressure only by its utility, like any candidate). Returns
2796    /// the new id.
2797    pub fn insert_preset(&mut self, tree: PatchTree, name: &str) -> Option<u64> {
2798        if let Some(existing) = self.pool.iter().find(|c| c.tree == tree) {
2799            return Some(existing.id);
2800        }
2801        let id = self.insert_candidate(tree, Origin::Preset, None)?;
2802        self.set_name(id, name);
2803        Some(id)
2804    }
2805
2806    /// Export the portable profile (log + standardizer, which only mean
2807    /// anything together).
2808    pub fn export_profile(&self) -> Profile {
2809        Profile {
2810            log: self.log.clone(),
2811            standardizer: self.standardizer.as_deref().cloned(),
2812        }
2813    }
2814
2815    /// Export the full session (profile + bank + lineage) for persistence.
2816    /// Renders and features are intentionally omitted — trees re-featurize
2817    /// deterministically on import.
2818    pub fn export_state(&self) -> SessionState {
2819        SessionState {
2820            profile: self.export_profile(),
2821            bank: self
2822                .pool
2823                .iter()
2824                .map(|c| BankEntry {
2825                    id: c.id,
2826                    tree: c.tree.clone(),
2827                    origin: c.origin,
2828                    name: c.name.clone(),
2829                    pinned: c.pinned,
2830                })
2831                .collect(),
2832            lineage: self.lineage.clone(),
2833            generation: self.generation,
2834            style_names: self.style_names.clone(),
2835            events: self.events.clone(),
2836            forecasts: self.forecasts.clone(),
2837            style_shares: self.style_shares.clone(),
2838        }
2839    }
2840
2841    /// Restore a saved session, replacing pool, log, standardizer, lineage,
2842    /// and id allocation. Each bank tree is re-featurized (and re-rendered
2843    /// when `keep_renders`); entries that no longer vet are dropped. Returns
2844    /// how many bank entries were restored.
2845    pub fn import_state(&mut self, state: SessionState) -> usize {
2846        let want_audio = self.wants_admitted_audio();
2847        let bank = self.import_state_deferred(state);
2848        for entry in bank {
2849            let Ok((cached, audition)) =
2850                featurize_memo(&entry.tree, &self.cfg.phrase, &self.memo, want_audio)
2851            else {
2852                continue;
2853            };
2854            let pre = PreFeaturized {
2855                tree: entry.tree.clone(),
2856                cached,
2857                audition,
2858            };
2859            self.absorb_bank_entry(entry, pre);
2860        }
2861        self.finish_restore()
2862    }
2863
2864    /// Restore a saved session **without rendering the bank**: everything
2865    /// [`Engine::import_state`] does except the per-entry featurize, returning
2866    /// the bank entries for off-engine work, in bank order.
2867    ///
2868    /// Restore is the returning user's boot and today it is *worse* than a
2869    /// cold one — a full bank of serial re-renders behind a bar that cannot
2870    /// move, because nothing lands until all of it finishes. This is the seam
2871    /// that lets the farm do it: each entry comes back through
2872    /// [`Engine::absorb_bank_entry`] and [`Engine::finish_restore`] closes the
2873    /// restore, and the three together are exactly `import_state`.
2874    ///
2875    /// Profile-then-clear ordering is preserved from `import_state`:
2876    /// [`Engine::import_profile`] may re-fit a standardizer over the *current*
2877    /// pool, so clearing before it would change the scale a restore lands on.
2878    pub fn import_state_deferred(&mut self, state: SessionState) -> Vec<BankEntry> {
2879        self.import_profile(state.profile);
2880        self.lineage = state.lineage;
2881        self.generation = state.generation;
2882        self.style_names = state.style_names;
2883        self.events = state.events;
2884        self.forecasts = state.forecasts;
2885        self.style_shares = state.style_shares;
2886        // The implicit stream stores raw φ on both sides of a hand edit, so it
2887        // is the fourth carrier of the corruption after the pool, the log and
2888        // the HELD tray — and the only one nothing reads yet, which is exactly
2889        // why it would have been the one still poisoned on the day it was
2890        // first fitted on.
2891        let names = phi_names();
2892        for e in &mut self.events {
2893            self.repaired_cells +=
2894                crate::migrate::repair_phi_pair(&mut e.phi_before, &mut e.phi_after, &names);
2895        }
2896        self.pool.clear();
2897        self.audio_lru.clear();
2898        // Every saved term, repaired on the way in. This is the *only* place a
2899        // tree written by an older build enters the engine, and a bank entry
2900        // carrying a knob outside its range would otherwise be quarantined by
2901        // the featurizer a few lines later and silently disappear from the
2902        // player's bank — losing four patches to fix a bug in one number.
2903        // Repair keeps the patch and loses only the corruption, which is the
2904        // standing rule for saved state: migration, never deletion.
2905        let mut bank = state.bank;
2906        for entry in &mut bank {
2907            if entry.tree.clamp_domains() > 0 {
2908                self.repaired_terms += 1;
2909            }
2910        }
2911        bank
2912    }
2913
2914    /// How many saved terms, log cells and whole observations the last
2915    /// [`Engine::import_state_deferred`] had to repair. All three are zero for
2916    /// a session written by a build that has this gate.
2917    ///
2918    /// Reported rather than logged because the frontend is the only thing that
2919    /// can tell the player their profile was mended, and a silent repair of the
2920    /// evidence a model is fitted on is exactly the kind of quiet the rest of
2921    /// this app was built to stop.
2922    pub fn repair_report(&self) -> (usize, usize, usize) {
2923        (
2924            self.repaired_terms,
2925            self.repaired_cells,
2926            self.dropped_observations,
2927        )
2928    }
2929
2930    /// Reinstate one restored bank entry with its saved identity, from a
2931    /// featurization performed off-engine.
2932    ///
2933    /// Bypasses the pool-size and novelty checks, as `import_state`'s push
2934    /// does: a bank is a bank, not a candidate competition. `entry` supplies
2935    /// the identity (id, origin, name) and the term; `pre` supplies φ.
2936    pub fn absorb_bank_entry(&mut self, entry: BankEntry, pre: PreFeaturized) {
2937        let PreFeaturized {
2938            tree: _,
2939            cached,
2940            audition,
2941        } = pre;
2942        self.memo.put(cached.clone(), audition.clone());
2943        let phi_std = self
2944            .standardizer
2945            .as_ref()
2946            .map(|sz| sz.transform(&cached.features.phi()))
2947            .unwrap_or_default();
2948        let render = self.admitted_render(&entry.tree, &cached.features, audition);
2949        self.next_id = self.next_id.max(entry.id + 1);
2950        self.pool.push(Candidate {
2951            id: entry.id,
2952            tree: settled(entry.tree),
2953            features: cached.features,
2954            phi_std,
2955            key: cached.key,
2956            render,
2957            origin: entry.origin,
2958            name: entry.name,
2959            pinned: entry.pinned,
2960        });
2961    }
2962
2963    /// Close a deferred restore once every entry that was going to land has.
2964    /// Returns the restored bank size.
2965    ///
2966    /// The standardizer normally comes from the profile; a session saved
2967    /// before the first fit completes has none — fit one from the restored
2968    /// bank so φ isn't left raw. Idempotent, and safe on an empty pool.
2969    pub fn finish_restore(&mut self) -> usize {
2970        if self.standardizer.is_none() && !self.pool.is_empty() {
2971            let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
2972            let sz = Arc::new(Standardizer::fit(&rows));
2973            for c in &mut self.pool {
2974                c.phi_std = sz.transform(&c.features.phi());
2975            }
2976            self.standardizer = Some(sz);
2977        }
2978        self.pool.len()
2979    }
2980
2981    /// Import a profile: replaces the log and re-establishes a standardizer
2982    /// for it.
2983    ///
2984    /// A profile written before raw-φ logging carries *standardized* vectors,
2985    /// which are only interpretable through the standardizer that shipped with
2986    /// them — so that pairing is exactly what makes the migration possible
2987    /// ([`crate::migrate`]): invert the transform, convert the coordinates
2988    /// whose units changed, and the log becomes raw evidence again. Its
2989    /// standardizer is then obsolete by construction (it has the wrong
2990    /// dimension for the current feature set) and a fresh one is fit from the
2991    /// migrated data. A same-schema profile keeps its standardizer, so
2992    /// imported θ geometry stays valid until the next fit refreshes it.
2993    pub fn import_profile(&mut self, profile: Profile) {
2994        self.log = profile.log;
2995        // A fresh restore reports on *itself*. `import_state_deferred` runs
2996        // this first and then counts the bank, so clearing all three here is
2997        // also what keeps a standalone "load taste profile" from inheriting the
2998        // patch count of whatever was loaded before it.
2999        self.repaired_terms = 0;
3000        self.repaired_cells = 0;
3001        self.dropped_observations = 0;
3002        let names = phi_names();
3003        if let Some(sz) = &profile.standardizer {
3004            if crate::migrate::needs_migration(&self.log) {
3005                // Schema-1 values were measured under the v1 stimulus, so
3006                // they land on the v1 names — FitSet::build keeps their
3007                // structural coordinates and imputes today's stimulus-tagged
3008                // audio coordinates at "no evidence" (migrate::v1_names).
3009                crate::migrate::migrate_log(
3010                    &mut self.log,
3011                    sz,
3012                    &crate::migrate::v1_names(),
3013                    self.cfg.phrase.sample_rate / 2.0,
3014                );
3015            }
3016        }
3017        // Before stamping: a coordinate that was renamed since this profile
3018        // was written still holds the right *value*, and `FitSet::build`
3019        // matches by name — so the rename has to be applied to the stored
3020        // names or the evidence is imputed away as "no opinion".
3021        crate::migrate::apply_renames(&mut self.log);
3022        crate::migrate::stamp_names(&mut self.log, &names);
3023        // After the names are stamped, because the repair is by name — and
3024        // before the standardizer is adopted, because a standardizer fitted
3025        // over a poisoned column is itself poisoned. When anything was
3026        // repaired the saved one is *discarded* and re-fitted from the
3027        // repaired rows plus the pool: keeping it would mean the load re-read
3028        // its own corruption back out of the scale it set.
3029        let (clamped, dropped) = crate::migrate::repair_log(&mut self.log);
3030        self.repaired_cells = clamped;
3031        self.dropped_observations = dropped;
3032        let poisoned = clamped > 0 || dropped > 0;
3033        match profile.standardizer {
3034            Some(sz) if sz.dimension() == names.len() && !poisoned => {
3035                let sz = Arc::new(sz);
3036                for c in &mut self.pool {
3037                    c.phi_std = sz.transform(&c.features.phi());
3038                }
3039                self.standardizer = Some(sz);
3040            }
3041            _ => {
3042                self.standardizer = None;
3043                self.refit_standardizer();
3044            }
3045        }
3046        self.session = self.log.n_sessions();
3047        self.posterior = None;
3048    }
3049}