auracle_session/engine.rs
1//! The two-loop session engine (the reference: *The two loops*).
2//!
3//! - **Patch loop** (machine-paced, silent): fill a pool with vetted prior
4//! draws; once a posterior exists, *refine* — warm-start fugue-evo's typed
5//! MH from the best pool members on the Boltzmann target
6//! `π_β ∝ p_grammar · exp(β·E[u(x)])` and inject improved candidates. The
7//! refinement run is a **short local MH walk** from each seed (a dozen
8//! steps, final state kept), not a draw from `π_β`: it moves candidates
9//! uphill on that target, which is what the pool needs, but nothing here
10//! claims the pool is distributed as `π_β`.
11//! - **Taste loop** (human-paced, persistent): feedback events append to the
12//! [`ObservationLog`] as **raw** φ; the posterior is re-fit from the log,
13//! standardizing at fit time.
14//!
15//! Between them, **acquisition**: [`Engine::next_duel`] maximizes expected
16//! information about θ (BALD). See its docs for why the obvious alternative —
17//! dueling Thompson sampling — is the wrong objective for this product.
18//!
19//! **Locks** (partial evolution): any set of trace addresses can be frozen
20//! during refinement. The MH kernel still proposes over all sites; a proposal
21//! that touches a locked address is rejected outside the kernel. Because the
22//! underlying kernel satisfies detailed balance on the full space, rejecting
23//! locked-coordinate moves yields a valid Metropolis-within-Gibbs sampler on
24//! the *conditional* posterior given the locked values — locking is exact,
25//! not a heuristic. That exactness depends on the rejection region being
26//! *symmetric*: [`Engine::violates_locks`] therefore checks births as well as
27//! deaths and edits. Wasted proposals are compensated by scaling step counts.
28//!
29//! All UI modes are emitters into the same observation stream: the engine
30//! does not know which surface produced an event. Candidates carry stable
31//! `id`s — pool positions shift on eviction, ids never do.
32
33use std::collections::{HashMap, HashSet, VecDeque};
34use std::sync::Arc;
35
36use auracle_features::{
37 featurize_memo, render_playback, Audition, Features, PhraseSpec, RenderMemo,
38};
39use auracle_grammar::prior::N_OPS;
40use auracle_grammar::{tree_diff, DiffEntry, PatchGrammarPrior, PatchTree};
41use auracle_taste::{
42 Feedback, FitSet, Observation, ObservationLog, Provenance, Standardizer, TasteConfig,
43 TasteModel, TastePosterior,
44};
45use fugue::Trace;
46use fugue_evo::inference::mh::EvolutionChain;
47use fugue_evo::inference::model::EvolutionModel;
48use rand::rngs::StdRng;
49use rand::{Rng, SeedableRng};
50use serde::{Deserialize, Serialize};
51
52use crate::calib::{calibration, Calibration, Forecast};
53use crate::farm::{draw_seed, Draw, PreFeaturized};
54use crate::naming::{claim_name, NameScale};
55use crate::surrogate::SurrogateFitness;
56
57/// Ceiling on the step-count compensation for locked sites — the most a locked
58/// refinement walk may cost relative to an unlocked one.
59///
60/// 4× fully compensates a walk with three quarters of its sites pinned, which is
61/// already a heavier lock than the hand-build → pin → breed loop produces. Past
62/// that the walk is deliberately under-compensated, because `⚡ evolve from
63/// this` is a button press with a person waiting behind it and a 90%-locked
64/// patch would otherwise ask for ten times the budget. See the note at the use
65/// site in `refine_one` for what that costs.
66const LOCK_SCALE_CAP: f64 = 4.0;
67
68/// The φ coordinate names, as owned strings (what the log records).
69pub fn phi_names() -> Vec<String> {
70 Features::phi_names()
71 .into_iter()
72 .map(|n| n.to_string())
73 .collect()
74}
75
76/// Which rule picks the next duel.
77///
78/// Selectable because the choice is an empirical claim, and
79/// `learn_synthetic --compare` measures it. Both alternatives are kept so
80/// that comparison stays runnable — a rule chosen on evidence should stay
81/// re-checkable, and a rule rejected on evidence doubly so.
82///
83/// ## The measurement, and what it is a measurement *of*
84///
85/// `cargo run -p auracle-session --example learn_synthetic --release --
86/// --compare 20`, on the synthetic user: 20 seeds, 72 duels, refit every 12.
87/// **Common random numbers** — pool fill, the user's coin flip at duel *t*,
88/// MCMC seed at round *r*, and refinement seeds are all shared across arms,
89/// so only the acquisition draw differs. Both regimes are graded on one fixed
90/// held-out exam under a single reference scale, so arms that built different
91/// pools are still answering the same questions. `±` is two standard errors
92/// of the paired difference.
93///
94/// ### Static pool (i.i.d. prior draws, `refine_steps: 0`)
95///
96/// | | cos θ\* ↑ | rank r ↑ | excess nats ↓ |
97/// |---|---|---|---|
98/// | **random** | 0.460 | 0.731 | 0.211 |
99/// | thompson | 0.416 | 0.628 | 0.254 |
100/// | bald | 0.484 | 0.762 | 0.199 |
101/// | bald − thompson | **+0.068 ± 0.062** | **+0.134 ± 0.044** | **−0.055 ± 0.014** |
102/// | bald − random | +0.025 ± 0.058 | +0.031 ± 0.046 | −0.012 ± 0.013 |
103///
104/// Dueling Thompson sampling is the one clear loser, at t = 2.2 / 6.1 / −8.0.
105/// It is a best-arm rule: it converges on identifying the top patch, which is
106/// not what a duel is for here. BALD and uniform pairing are inside two
107/// standard errors of each other on every metric.
108///
109/// A static i.i.d. pool is also a weak regime to conclude from on its own:
110/// prior draws are spread over feature space *by construction*, which is
111/// exactly where uniform pairs already achieve near-optimal `‖φ_a − φ_b‖`
112/// coverage and an information-seeking rule has no redundancy to prune. The
113/// concern was that the shipped pool is not that pool — refinement injects
114/// children near the current best and `insert_candidate` evicts the worst —
115/// so `--compare` runs an **evolving** regime too, with real refinement
116/// between rounds (the `Regime` type in `learn_synthetic.rs` documents the
117/// design).
118///
119/// ### Evolving pool (`refine_steps: 12`, refinement between rounds)
120///
121/// | | cos θ\* ↑ | rank r ↑ | excess nats ↓ |
122/// |---|---|---|---|
123/// | **random** | 0.479 | 0.694 | 0.232 |
124/// | thompson | 0.459 | 0.583 | 0.276 |
125/// | bald | 0.465 | 0.707 | 0.232 |
126/// | bald − thompson | +0.006 ± 0.068 | **+0.124 ± 0.066** | **−0.044 ± 0.017** |
127/// | bald − random | −0.015 ± 0.055 | +0.013 ± 0.048 | −0.000 ± 0.014 |
128///
129/// Same answer: Thompson loses, BALD and uniform pairing tie on every metric.
130///
131/// The run's manipulation check is itself a finding. Final pool spread (mean
132/// pairwise `‖Δφ‖`, reference scale) was **7.7–7.9 evolving vs 7.2 static**:
133/// six generations over a 72-duel session did not concentrate the pool at
134/// all — frontier-biased injection plus worst-eviction *widened* it slightly,
135/// because mutation pushes children into feature-space extremes faster than
136/// eviction trims them. So the concentrated regime BALD was hypothesized to
137/// win never arises at session horizon, and the tie is not an artifact of a
138/// spread pool that only the static setup guaranteed — the product's own
139/// dynamics keep the pool spread.
140///
141/// ## Why `Random` is the default
142///
143/// Measured in both the regime the product starts in and the regime it
144/// evolves into, uniform pairing is indistinguishable from BALD — and a rule
145/// with four tuning constants that ties a rule with none should not ship on
146/// a tie. Two supporting justifications survived checking, one did not: the
147/// `info_gain` BALD reports had **zero** consumers in the frontend, and BALD's
148/// repeat avoidance, while real, is barely needed over a 48-candidate pool
149/// that uniform pairing already samples without repeating (measured in
150/// `duels_spread_over_candidates_not_just_pairs`). `Random` also makes
151/// **every** duel an unbiased calibration sample rather than one in ten —
152/// a virtue that holds regardless of which rule learns θ faster.
153///
154/// One earlier justification was retracted for a bad reason, and the record
155/// should say so. The "pool grows and concentrates" argument was dismissed on
156/// the grounds that `insert_candidate` caps the pool — but a capped *size* is
157/// not an unchanging *spread*, and evicting the worst member could in
158/// principle concentrate a pool. Dismissing the concentration argument
159/// *because it was unmeasured*, while treating a measurement from the other
160/// regime as decisive, had the burden of proof backwards. The evolving run
161/// above is that measurement; it happens to show the concentration never
162/// materializes, but the default rests on the measured tie, not on the
163/// dismissal.
164///
165/// ## What `Bald` is still for
166///
167/// It is not dead code and it is not a fallback. It decisively beats the
168/// best-arm rule, so it is the right thing to reach for if acquisition ever
169/// needs to *do* something uniform pairing cannot: bias duels toward patches
170/// the user will enjoy auditioning ([`SessionConfig::duel_utility_weight`]),
171/// bound how often one patch reappears ([`SessionConfig::duel_exposure_penalty`]),
172/// or report why a question was asked. Those levers exist and are measured;
173/// none of them is currently worth the tie.
174///
175/// ## A correction worth recording
176///
177/// An earlier version of this rule scored its enjoyment term on *unnormalized*
178/// utility and used an *absolute* softmax temperature of 0.05 nats. Both are
179/// scale bets, and both lost: the enjoyment term grew without bound as the
180/// posterior sharpened, and `exp(ΔJ/T)` ran to `e¹⁰`, so the "softmax" was an
181/// argmax. That version was measurably *worse* than random, and it is the
182/// version an independent replication measured. It is also what produced the
183/// duel repetition seen in the running app — the same defect, observed from
184/// two directions. Fixed, BALD ties random; the numbers above are the fixed
185/// rule.
186#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
187pub enum Acquisition {
188 /// Uniformly random pairs. The default — see the type doc.
189 #[default]
190 Random,
191 /// Dueling Thompson sampling: two posterior draws, duel their champions.
192 /// Best-arm identification — converges on the top patch, not on θ.
193 Thompson,
194 /// Expected information gain about θ, plus an enjoyment term and a
195 /// repeat penalty, sampled from a softmax. Beats [`Acquisition::Thompson`]
196 /// decisively and ties [`Acquisition::Random`]; see the type doc.
197 Bald,
198}
199
200/// Which state of a refinement walk becomes the injected child.
201///
202/// Selectable because the choice is an empirical claim, and the same rule
203/// applies here as to [`Acquisition`]: a rule chosen on evidence should stay
204/// re-checkable, and a rule rejected on evidence doubly so. `make climb` and
205/// `search_health --budget-ab` are where the comparison runs.
206///
207/// ## Why this is a question at all
208///
209/// A refinement walk renders and featurizes ~40 candidates and injects **one**.
210/// Which one is free to choose — the whole walk is already in the memo, and
211/// every trace the kernel returns already carries its own `log π_β` — so the
212/// choice costs nothing either way and has never been measured.
213///
214/// The tension is real in both directions. [`Self::Last`] is a draw from where
215/// the chain ended up, which respects the target's own weighting and is
216/// robust: it cannot be fooled by a single point where the surrogate happens
217/// to be over-optimistic. [`Self::Best`] takes the walk's argmax, which is what
218/// a *shortlist* wants — the pool is not a sample, it is a few dozen patches a
219/// person will listen to — but argmax over a surrogate is the classic way to
220/// find that surrogate's errors rather than the user's preferences.
221///
222/// ## The A/B, run, and its result — a tie
223///
224/// `make climb SEEDS=16` on both arms, same seed list, so the per-seed lines
225/// pair directly:
226///
227/// ```text
228/// Last Best
229/// mean gain +1.927 ± 0.452 +1.774 ± 0.302
230/// median gain +2.058 +1.819
231/// 10% trimmed +1.840 ± 0.383 +1.925 ± 0.190
232/// climbed on 14/16 15/16
233///
234/// paired (Best − Last) mean −0.153 ± 0.384 (−0.40 se)
235/// median −0.185
236/// trimmed −0.113 ± 0.318
237/// sign 8 better / 8 worse, p = 1.000
238/// ```
239///
240/// Eight and eight is as exact a tie as sixteen seeds can produce. The
241/// difference does not clear zero at 2 se on any of the three statistics, so
242/// **the default stays [`Self::Last`]** — kept re-checkable rather than
243/// deleted, the same way [`Acquisition::Thompson`] is kept after losing.
244///
245/// Two things worth reading off it rather than leaving in the table:
246///
247/// - **The feared failure did not happen, and neither did the hoped-for win.**
248/// The worry was that argmax over a surrogate would find the surrogate's
249/// errors and deepen the catastrophic tail. Across the pair the tails are a
250/// wash — the worst `Last` seed goes −0.74 → −1.80 under `Best`, and the
251/// next two worst go −0.64 → +0.52 and +0.12 → +1.29. `Best` climbs on one
252/// more seed and means marginally less.
253/// - **`Best` is the lower-variance rule, not the better one.** Its trimmed
254/// standard error is half `Last`'s (0.190 against 0.383). Injecting the
255/// walk's argmax is more *consistent* than injecting where it stopped; it
256/// just does not aim anywhere better on average. That is a coherent thing
257/// for argmax-over-a-noisy-surrogate to be, and it is the argument to
258/// re-run this on if the surrogate ever gets sharper.
259#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
260pub enum RefineKeep {
261 /// Inject the state the walk ended on. The shipped behaviour, and the
262 /// default — the A/B above ran and tied, so nothing moved it.
263 #[default]
264 Last,
265 /// Inject the highest-`log π_β` state the walk occupied, seed included —
266 /// so a walk that found nothing better than its seed injects nothing.
267 Best,
268}
269
270/// What the pool does with audition audio.
271///
272/// Renders are the engine's only expensive artifact and its bulkiest one: at
273/// the default phrase a single audition buffer is ~565 KB of f32, and a full
274/// pool of them is tens of megabytes of wasm heap — resident forever, for
275/// audio the user will mostly never ask to hear. But φ, not audio, is what
276/// the pool exists to hold, and [`auracle_features::render_playback`] can
277/// reproduce any buffer bit-identically from the term. So retention is a
278/// policy, not a structural requirement.
279#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
280pub enum RenderPolicy {
281 /// Materialize every candidate's buffer at admission and keep it for the
282 /// lifetime of the candidate. Fastest audition, largest footprint.
283 Eager,
284 /// Materialize on [`Engine::render_of`], keeping the most recently
285 /// auditioned [`SessionConfig::audio_cache`] buffers and dropping the
286 /// rest. A cold audition costs one render.
287 Lazy,
288 /// Never keep audio. Headless callers (tests, `learn_synthetic`) never
289 /// audition anything, and this is what they should pay.
290 #[default]
291 None,
292}
293
294/// Engine configuration.
295#[derive(Clone, Debug)]
296pub struct SessionConfig {
297 /// Vetted candidates to maintain in the pool.
298 pub pool_size: usize,
299 /// Maximum prior draws attempted per `fill_pool` (vet failures burn
300 /// attempts).
301 pub max_draws: usize,
302 /// MH refinement steps per seed (scaled up when locks waste proposals).
303 ///
304 /// The default scales with [`N_OPS`]: a structural proposal picks a new
305 /// operator from a categorical that the v2 palette widened from six to
306 /// twenty, so a fixed budget would spend the same number of proposals
307 /// covering a far wider move set and land the pool's children in a
308 /// visibly thinner slice of it.
309 ///
310 /// ## The split is measured, not reasoned
311 ///
312 /// `2·N_OPS` steps from `N_OPS/2` seeds was an argument, and the argument
313 /// could have been wrong in either direction. `search_health --budget-ab`
314 /// exists to settle it; over 8 seeds, 6 generations, graded against the
315 /// synthetic user's true utility:
316 ///
317 /// | steps | seeds | proposals | mean u | max u | |
318 /// |---|---|---|---|---|---|
319 /// | 40 | 10 | 400 | **1.714** | **8.154** | shipped |
320 /// | 40 | 3 | 120 | 1.241 | 6.178 | same depth, fewer seeds |
321 /// | 66 | 3 | 198 | 0.774 | 6.281 | same total, fewer seeds |
322 /// | 20 | 20 | 400 | 0.568 | 6.790 | half depth, double breadth |
323 ///
324 /// The shipped split wins on both metrics, and it is a genuine optimum
325 /// rather than the top of a slope: moving off it in *either* direction is
326 /// worse. Two rows are worth more than the headline.
327 ///
328 /// **Depth from few seeds is actively harmful.** 66×3 runs 65% more
329 /// proposals than 40×3 and scores *lower* (0.774 against 1.241) — a long
330 /// chain from a bad starting point converges confidently on somewhere you
331 /// did not want to be, and the extra steps are what get it there.
332 ///
333 /// **Breadth is not free either.** 20×20 spends the shipped budget and is
334 /// the worst row of the four. Twenty steps is not enough for a chain to
335 /// leave its seed, so the generation is twenty barely-moved copies of the
336 /// current top — which is also why it has the second-best `max`: it
337 /// preserves the frontier by never straying from it.
338 ///
339 /// Re-run this before changing either number.
340 pub refine_steps: usize,
341 /// How many top candidates to refine from. Also scaled with [`N_OPS`] —
342 /// more seeds is more *starting points*, which is what actually buys
343 /// coverage of a wider palette, whereas more steps per seed buys depth
344 /// around one. See [`SessionConfig::refine_steps`] for the measurement
345 /// that fixes the ratio between them.
346 pub refine_seeds: usize,
347 /// Boltzmann sharpness β of the refinement target.
348 pub beta: f64,
349 /// Which state of a refinement walk becomes the injected child.
350 pub refine_keep: RefineKeep,
351 /// Maximum style components in the taste mixture
352 /// (max-of-linear-experts); the fitted K grows with evidence up to this
353 /// cap.
354 ///
355 /// K is also the fit's dominant cost driver, because single-site MH
356 /// rebuilds the whole program every step and the site count is
357 /// `d·K + n_sessions + 5` — at today's d = 40, that is 46 at K = 1 and
358 /// **206 at K = 5** (printed by `fit_bench`, so it moves with φ). Two
359 /// consequences, both measured by `auracle-taste/examples/fit_bench.rs`:
360 /// the fit is ~4× slower at the cap than at the first fit, and the step
361 /// budget is *fixed*, so a mature fit gets ~4× fewer sweeps per site than
362 /// an early one — growing K makes the fit both slower and statistically
363 /// thinner.
364 ///
365 /// **Open option, deliberately not taken here: cap this at 3** (sites
366 /// 206 → 126, a ~1.6× mature-fit win at no engineering cost). It is left
367 /// open because unlike the address hoist and the budget cut it is not a
368 /// pure efficiency change — it removes model *capacity*, and capacity is
369 /// the whole point of the mixture (a user with four islands of taste
370 /// cannot be represented by three lenses). Take it only on evidence:
371 /// [`TastePosterior::style_share`](auracle_taste::TastePosterior::style_share)
372 /// reports what fraction of the pool each lens claims, and if lenses 4
373 /// and 5 sit near zero share across real sessions they are paying 54
374 /// sites per step for nothing. `learn_synthetic --compare` is the A/B.
375 pub k_styles: usize,
376 /// The audition stimulus.
377 pub phrase: PhraseSpec,
378 /// How the pool retains audition audio.
379 pub render_policy: RenderPolicy,
380 /// Audition buffers kept resident under [`RenderPolicy::Lazy`], most
381 /// recently auditioned first. Sized for the current duel pair, the bench
382 /// subject, and enough recent history that stepping back through the bank
383 /// is free.
384 pub audio_cache: usize,
385 /// Post-warmup MH steps per posterior fit.
386 ///
387 /// This is the one knob in this struct that buys wall time with
388 /// *statistics*, so it is set from a measurement rather than a guess.
389 /// Only 500 draws survive thinning at any budget, so the budget does not
390 /// buy draws — it buys **sweeps per site**, and at K = 5 (206 sites) even
391 /// 10 000 steps is only ~49 sweeps.
392 ///
393 /// Recovery vs budget at the mature operating point (K = 5, n_obs = 100,
394 /// 12 seeds, `cargo run --release -p auracle-taste --example fit_bench
395 /// -- sweep 12`): held-out duel agreement with the noiseless ground-truth
396 /// ordering, and the cosine of the best lens against θ\*.
397 ///
398 /// | steps | held-out acc | best-lens cos | native fit |
399 /// |---|---|---|---|
400 /// | 30 000 | 0.767 | 0.724 | 1.79 s |
401 /// | 20 000 | 0.757 | 0.717 | 1.16 s |
402 /// | **10 000** | **0.746** | **0.686** | **0.60 s** |
403 /// | 8 000 | 0.738 | 0.690 | 0.49 s |
404 /// | 6 000 | 0.737 | 0.653 | 0.38 s |
405 /// | 5 000 | 0.729 | 0.655 | 0.33 s |
406 /// | 3 000 | 0.713 | 0.599 | 0.20 s |
407 ///
408 /// That curve is smooth, so it says where the trade *stops paying*. The
409 /// second instrument is the end-to-end M4 gate
410 /// (`closed_loop_learns_synthetic_taste`, which runs at exactly this
411 /// budget through the real render → vet → feature pipeline). One run of
412 /// it is a **single draw** — over the pool lottery, the duel answers and
413 /// the chain — so it is replicated over 13 seeds here (`cargo run
414 /// --release -p auracle-session --example closed_loop_sweep`). Its
415 /// pool/truth correlation `r` against the 0.6 gate, plus the other two
416 /// metrics the test asserts:
417 ///
418 /// | steps | mean r | min r | seeds with r ≤ 0.6 | mean top-5 | mean cos |
419 /// |---|---|---|---|---|---|
420 /// | 30 000 | 0.736 | 0.575 | 1/13 | 3.14 | 0.528 |
421 /// | 20 000 | 0.722 | 0.576 | 2/13 | 2.88 | 0.497 |
422 /// | **10 000** | **0.726** | **0.551** | **2/13** | **2.78** | **0.475** |
423 /// | 8 000 | 0.715 | 0.503 | 1/13 | 3.07 | 0.456 |
424 /// | 6 000 | 0.747 | 0.600 | 1/13 | 3.39 | 0.497 |
425 /// | 5 000 | 0.689 | 0.476 | 2/13 | 3.15 | 0.392 |
426 ///
427 /// Read that as a noisy measurement, because it is one. Within a single
428 /// budget the seed-to-seed spread of `r` is sd ≈ 0.07–0.10 over a range
429 /// of ≈ 0.25; between budgets from 6 000 up the means sit in
430 /// 0.715–0.747, i.e. inside one standard error (≈ 0.02) of each other —
431 /// and 6 000 posts the *highest* mean of the six, which is the plainest
432 /// sign that this instrument's ranking of the upper budgets is noise.
433 /// **From 6 000 to 30 000 it cannot tell them apart.** Only 5 000
434 /// separates at all — lowest on mean `r`, on min `r` and on cos — and
435 /// even that gap to 30 000 (0.047) is barely over one standard error of
436 /// the difference.
437 ///
438 /// So the argument for 10 000 is *not* that it passes where 5 000 fails.
439 /// Every budget here fails the 0.6 gate on some seed, including the old
440 /// 30 000 (1 of 13), and 5 000 clears it on 11 of 13. The argument is:
441 /// 10 000 is 3× cheaper than 30 000 and gives up 0.010 of mean `r`, which
442 /// is inside the noise; the `fit_bench` sweep above — 12 seeds on a
443 /// metric with far less variance — prices the same cut at 0.021 of
444 /// held-out accuracy and 0.038 of cos; and cutting further to 5 000 saves
445 /// only another 0.27 s per fit while costing 0.017 more held-out
446 /// accuracy, 0.031 more cos and 0.037 of mean `r`, the one budget *both*
447 /// instruments mark down. 10 000 is where the two instruments agree, not
448 /// where a threshold was crossed.
449 ///
450 /// (An earlier revision of this table read the M4 gate at a single seed,
451 /// `0xE05`, and concluded that 5 000 "fails outright" at r = 0.565 while
452 /// 10 000 held "the widest margin of any budget tried". Both are
453 /// artifacts of that one draw: 0xE05 sits ~1.2 sd low at 5 000 and right
454 /// on the mean at 10 000. The per-seed numbers reproduce exactly — the
455 /// inference from one of them did not.)
456 ///
457 /// The earlier 30 000 also predated the address hoist in
458 /// [`auracle_taste::model`], which made every step ~1.7× cheaper on its
459 /// own; the two together take a mature fit from ~1.86 s to ~0.60 s
460 /// natively (~13 s → ~4 s in the browser).
461 pub mcmc_samples: usize,
462 /// Warmup (adaptation) steps per fit, held at ~30 % of
463 /// [`Self::mcmc_samples`]. Warmup only tunes the per-site proposal
464 /// scales; it produces no draws, so it is pure overhead beyond the point
465 /// the scales converge.
466 pub mcmc_warmup: usize,
467 /// Recency half-life for the taste likelihood, in observations
468 /// (`None` = no forgetting). Tastes drift; old votes should fade.
469 pub recency_half_life: Option<f64>,
470 /// Strength of the taste→grammar proposal tilt (0 disables): structural
471 /// θ components multiply the grammar's kind weights by
472 /// `exp(η·θ)` during refinement.
473 pub proposal_tilt: f64,
474 /// λ in the duel objective: how much the *pleasantness* of a duel counts
475 /// against its informativeness, applied to **pool-standardized** utility.
476 /// The user's enjoyment is a resource too — two mud patches are a cheap
477 /// question and an expensive answer.
478 ///
479 /// Keep it small. Information gain is bounded by `ln 2 ≈ 0.693` nats, so
480 /// a λ near 0.3 lets the ±2σ enjoyment term swing the objective by ±0.6 —
481 /// as much as the entire information range — and the acquisition function
482 /// quietly reverts to "duel the two best patches", which is the best-arm
483 /// behaviour BALD was adopted to escape. Measured on the synthetic user
484 /// (`learn_synthetic --compare`), λ = 0.3 cost 0.15 of pool-ranking
485 /// correlation against λ = 0; 0.1 leaves it a tie-breaker.
486 pub duel_utility_weight: f64,
487 /// γ in the duel objective: penalty per previous showing of the same
488 /// pair. Without it the acquisition function re-asks its favourite
489 /// question until the next refit.
490 pub duel_repeat_penalty: f64,
491 /// Penalty per previous *appearance of either candidate*, regardless of
492 /// who it was paired against.
493 ///
494 /// The pair penalty alone does not stop degeneracy, and the shipped app
495 /// proved it: over twelve consecutive duels one candidate appeared in
496 /// six. Every pairing `#1 vs #7`, `#1 vs #15`, `#1 vs #22` is a *distinct*
497 /// pair and pays no pair penalty at all, while the enjoyment term keeps
498 /// nominating the highest-utility candidate. The user does not experience
499 /// "distinct pairs"; they experience hearing the same patch over and over.
500 /// This term is what makes the *candidate* budget finite.
501 pub duel_exposure_penalty: f64,
502 /// Softmax temperature over the duel objective, as a **fraction of the
503 /// objective's own spread** across the candidate pairs.
504 ///
505 /// Scale-free for the same reason the enjoyment term is standardized: an
506 /// absolute temperature is a bet on how far apart the scores happen to
507 /// be. Shipped at an absolute 0.05 nats it was a bad bet — the objective
508 /// spans several tenths of a nat once the enjoyment term is in it, so
509 /// `exp(ΔJ/T)` ran to `e¹⁰` and the "softmax" was an argmax with extra
510 /// steps. Expressed as a fraction of the observed SD, 0.6 means the same
511 /// softness whatever the spread.
512 pub duel_temperature: f64,
513 /// Show one uniformly-random "check" duel every N duels. An
514 /// information-seeking acquisition deliberately picks pairs near p = 0.5,
515 /// so calibration measured on acquisition-chosen duels is
516 /// selection-biased; these are the unbiased subsample.
517 ///
518 /// Redundant under [`Acquisition::Random`], where every duel is already
519 /// uniform and is tagged as a check — the setting is kept because it is
520 /// exactly what [`Acquisition::Bald`] would need, and because one in ten
521 /// was measured to be underpowered anyway (a few forecasts out of fifty
522 /// cannot fill a five-bin reliability diagram). 0 disables.
523 pub duel_check_every: usize,
524 /// Which rule picks the next duel.
525 pub acquisition: Acquisition,
526 /// Fold each new observation into the posterior weights by importance
527 /// sampling between full refits. Off makes the posterior frozen between
528 /// fits, which is what the A/B compares against.
529 pub sis_between_fits: bool,
530}
531
532impl Default for SessionConfig {
533 fn default() -> Self {
534 Self {
535 pool_size: 48,
536 max_draws: 400,
537 // 12 and 3 were tuned against the six-operator v1 palette; both
538 // ride N_OPS so the same tuning survives a palette change.
539 refine_steps: 2 * N_OPS,
540 refine_seeds: N_OPS.div_ceil(2),
541 beta: 2.0,
542 refine_keep: RefineKeep::default(),
543 k_styles: 5,
544 phrase: PhraseSpec::default(),
545 render_policy: RenderPolicy::None,
546 audio_cache: auracle_features::DEFAULT_AUDIO_CAP,
547 mcmc_samples: 10_000,
548 mcmc_warmup: 3_000,
549 recency_half_life: Some(150.0),
550 proposal_tilt: 0.6,
551 duel_utility_weight: 0.1,
552 duel_repeat_penalty: 0.5,
553 duel_exposure_penalty: 0.25,
554 duel_temperature: 0.6,
555 duel_check_every: 10,
556 acquisition: Acquisition::default(),
557 sis_between_fits: true,
558 }
559 }
560}
561
562/// Where a candidate came from.
563#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
564#[serde(rename_all = "snake_case")]
565pub enum Origin {
566 /// Drawn from the grammar prior.
567 Prior,
568 /// Produced by taste-guided MH refinement.
569 Refined,
570 /// Hand-edited on the panel and committed.
571 Edited,
572 /// Loaded from the built-in preset bank.
573 Preset,
574}
575
576/// Give a tree its node identities on the way into the pool.
577///
578/// The pool is where a term stops being a search intermediate and becomes a
579/// patch someone can open, lock, lay out and breed from, so it is exactly where
580/// identities are worth minting — and the only place. Search itself hands
581/// through thousands of anonymous trees per generation; a prior draw carries
582/// none, and a tree restored from a save written before uids existed carries
583/// none either, which is the whole of that migration: old saves deserialize
584/// with every `uid` defaulted to unset and are settled here on the way in.
585fn settled(mut tree: PatchTree) -> PatchTree {
586 tree.ensure_uids();
587 tree
588}
589
590/// A vetted pool member.
591pub struct Candidate {
592 /// Stable id (unique for the lifetime of the engine; survives pool
593 /// reordering and eviction of *other* members).
594 pub id: u64,
595 /// The term.
596 pub tree: PatchTree,
597 /// Its extracted features.
598 pub features: Features,
599 /// Standardized feature vector (empty until the standardizer exists).
600 pub phi_std: Vec<f64>,
601 /// Content address of this candidate's `(term, spec)` featurization.
602 /// Carried rather than recomputed because hashing the term is the one
603 /// thing every cache path needs and the term never changes.
604 pub key: String,
605 /// The audition buffer, when resident. Governed by
606 /// [`SessionConfig::render_policy`] — under [`RenderPolicy::Lazy`] this is
607 /// `None` until [`Engine::render_of`] materializes it, and may go back to
608 /// `None` when a newer audition evicts it. Never a signal that the
609 /// candidate is unplayable; ask [`Engine::render_of`] for that.
610 ///
611 /// Shared with the memo (and with whoever last asked for it) through an
612 /// [`Arc`] — one allocation per audition, however many holders it has.
613 pub render: Option<Arc<Audition>>,
614 /// Provenance.
615 pub origin: Origin,
616 /// User-given name (frontends fall back to `tree.signature()`).
617 pub name: Option<String>,
618 /// The user asked to keep this one: [`Engine::insert_candidate`] will never
619 /// evict it.
620 ///
621 /// Deliberately **not** derived from the star rating. A star is an
622 /// observation that enters the log and moves θ; if a rating also decided
623 /// what survives, users would rate strategically to protect patches, and
624 /// every protective over-rating is a preference they never held — under
625 /// exactly the pressure where they care most. So the two channels stay
626 /// separate: stars are what you think, pins are what you keep.
627 ///
628 /// Capped by [`Engine::pin_cap`]; see there for why the pool cannot be
629 /// pinned solid.
630 pub pinned: bool,
631}
632
633/// One recorded evolution/edit step, for the lineage display.
634#[derive(Clone, Debug, Serialize, Deserialize)]
635pub struct LineageEvent {
636 /// Generation counter at the time of the event (increments per
637 /// `refine`/`refine_from` call).
638 pub generation: usize,
639 /// `"refine"` or `"edit"`.
640 pub kind: String,
641 /// Parent candidate id.
642 pub parent_id: u64,
643 /// Child candidate id.
644 pub child_id: u64,
645 /// What changed, in trace-address terms.
646 pub diff: Vec<DiffEntry>,
647 /// Parent posterior-mean utility at event time (0 with no posterior).
648 pub parent_utility: f64,
649 /// Child posterior-mean utility at event time.
650 pub child_utility: f64,
651}
652
653/// A portable taste profile: the observation log **plus the standardizer its
654/// φ vectors were standardized under**. θ is only meaningful relative to its
655/// standardizer, so the two persist together.
656#[derive(Clone, Debug, Serialize, Deserialize)]
657pub struct Profile {
658 /// The observation log (source of truth).
659 pub log: ObservationLog,
660 /// The standardizer under which every φ in the log was recorded.
661 pub standardizer: Option<Standardizer>,
662}
663
664/// Tilt categorical proposal weights by taste: `w'_i ∝ w_i · exp(η·t_i)`,
665/// with each multiplier clamped to `[1/4, 4]` so no kind is ever starved or
666/// monopolized, and the result renormalized. Pure, so the taste→grammar
667/// mapping is testable without an MCMC fit.
668/// Shrink a posterior mean toward zero by its own uncertainty:
669/// `θ·|θ|/(|θ| + σ)`.
670///
671/// The factor is 1 when the coefficient is many standard deviations from
672/// zero, ½ when `σ = |θ|`, and →0 when the posterior is mostly prior. It is
673/// the same shape as a signal-to-noise weighting, chosen over a hard
674/// significance cut because a cut makes the proposal distribution jump
675/// discontinuously as evidence accumulates, and users hear that as the
676/// instrument changing its mind.
677fn shrink(mean: f64, std: f64) -> f64 {
678 let m = mean.abs();
679 if m <= 0.0 {
680 return 0.0;
681 }
682 mean * m / (m + std.max(0.0))
683}
684
685pub fn tilt_weights(base: &[f64], tilts: &[f64], eta: f64) -> Vec<f64> {
686 let mut out: Vec<f64> = base
687 .iter()
688 .zip(tilts)
689 .map(|(w, t)| w * (eta * t).exp().clamp(0.25, 4.0))
690 .collect();
691 let sum: f64 = out.iter().sum();
692 if sum > 0.0 {
693 for w in &mut out {
694 *w /= sum;
695 }
696 }
697 out
698}
699
700/// One bank entry of a saved session (renders and features are re-derived
701/// on import — trees are the source of truth).
702#[derive(Clone, Debug, Serialize, Deserialize)]
703pub struct BankEntry {
704 /// The candidate's stable id (preserved so lineage references stay
705 /// meaningful).
706 pub id: u64,
707 /// The patch term.
708 pub tree: PatchTree,
709 /// Provenance.
710 pub origin: Origin,
711 /// User-given name.
712 pub name: Option<String>,
713 /// Whether the user pinned this patch against eviction.
714 ///
715 /// `#[serde(default)]` is what makes this change safe for sessions saved
716 /// before pins existed: the record is one IndexedDB key with no schema
717 /// version, so compatibility has to be by construction. An old session
718 /// loads with nothing pinned, which is exactly what it meant.
719 #[serde(default)]
720 pub pinned: bool,
721}
722
723/// An implicit preference signal, logged but (for now) not modeled: promote
724/// events, hand-edit commits, per-patch play counts. Un-logged signal is
725/// gone forever; modeling can come later.
726///
727/// The three optional fields carry the editor's stream (WS-8 §3). The single
728/// most informative row in it is a **revert**: the player made an edit, heard
729/// it, sat with it for a few seconds, and took it back. That is a preference
730/// statement about a pair of patches neither of which is in the bank, at edit
731/// granularity — far denser than the duel stream and the natural training set
732/// for an edit-level model. It is unbuildable without a year of this log, and
733/// the log is unbuildable retroactively, which is the whole argument for
734/// writing it before anything reads it.
735///
736/// Deliberately **not** in the likelihood, exactly as the play counts already
737/// here are not: a revert is confounded with curiosity, and the honest place
738/// for it is a v2 fit that can be validated, not a silent term in the model
739/// the player is being shown a number from.
740#[derive(Clone, Debug, Serialize, Deserialize)]
741pub struct ImplicitEvent {
742 /// `"promote"`, `"play"`, `"edit"`, `"revert"`, `"commit"`, …
743 pub kind: String,
744 /// Candidate id the event is about (0 when it is about the bench, which
745 /// is not a candidate until it is committed).
746 pub id: u64,
747 /// Magnitude (play counts, dwell in ms, 1 for point events).
748 pub value: f64,
749 /// Session index when it happened.
750 pub session: usize,
751 /// Free-form JSON detail: the `StructOp` and module kind for an edit, the
752 /// query string for a link-drag search, the outcome of a commit. A string
753 /// rather than a typed field per event kind, because the point of this log
754 /// is to be *written* now and interpreted later — a schema fixed today is
755 /// a schema that stops the next event kind from being logged at all.
756 #[serde(default, skip_serializing_if = "String::is_empty")]
757 pub detail: String,
758 /// Raw φ before the event, where the event is a transition (revert).
759 #[serde(default, skip_serializing_if = "Vec::is_empty")]
760 pub phi_before: Vec<f64>,
761 /// Raw φ after it.
762 #[serde(default, skip_serializing_if = "Vec::is_empty")]
763 pub phi_after: Vec<f64>,
764}
765
766/// What one posterior fit claimed for each style lens.
767///
768/// Recorded per fit and persisted, because the question it answers is about
769/// **real sessions over time** and cannot be answered from one of them.
770/// [`SessionConfig::k_styles`] documents an option that is deliberately not
771/// taken — cap K at 3, taking the fit from 206 sites to 126 for a ~1.6×
772/// mature-fit win — and gates it explicitly on whether lenses 4 and 5 sit near
773/// zero share across real sessions. Nothing collected that, so the decision
774/// could not be made either way; this is the collection.
775///
776/// It is deliberately not a judgment. A row says what the shares *were* at a
777/// given evidence count, and how many lenses the fit was even allowed (`k`
778/// grows with the log and is capped by config, so an early row with two lenses
779/// is not evidence that lenses 3–5 are idle — it is evidence they did not
780/// exist yet). Reading rows where `k == k_styles` is what the option needs.
781#[derive(Clone, Debug, Serialize, Deserialize)]
782pub struct StyleShareRecord {
783 /// Observations in the log at the time of the fit.
784 pub observations: usize,
785 /// Lenses this fit was allowed — `min(1 + log/20, k_styles)`.
786 pub k: usize,
787 /// Share of the pool claimed by each lens, aligned, summing to ~1.
788 pub shares: Vec<f64>,
789}
790
791/// A full saved session: everything needed to restore the app across a
792/// reload — the portable profile plus the bank and its history.
793#[derive(Clone, Debug, Serialize, Deserialize)]
794pub struct SessionState {
795 /// Log + standardizer.
796 pub profile: Profile,
797 /// The patch bank (trees, origins, names).
798 pub bank: Vec<BankEntry>,
799 /// Evolution/edit history.
800 pub lineage: Vec<LineageEvent>,
801 /// Generation counter.
802 pub generation: usize,
803 /// User-given style names (index = aligned style index).
804 #[serde(default)]
805 pub style_names: Vec<String>,
806 /// Implicit preference events.
807 #[serde(default)]
808 pub events: Vec<ImplicitEvent>,
809 /// Out-of-sample duel forecasts (calibration survives a reload).
810 #[serde(default)]
811 pub forecasts: Vec<Forecast>,
812 /// Per-fit style shares — the evidence [`SessionConfig::k_styles`]' open
813 /// option is gated on.
814 #[serde(default)]
815 pub style_shares: Vec<StyleShareRecord>,
816}
817
818/// A chosen duel, with the reasoning that produced it.
819#[derive(Clone, Copy, Debug, Serialize, Deserialize)]
820pub struct DuelChoice {
821 /// Pool index of candidate A.
822 pub a: usize,
823 /// Pool index of candidate B.
824 pub b: usize,
825 /// Expected information gain about θ, in nats (0 for random pairs).
826 /// Bounded above by `ln 2 ≈ 0.693`, the entropy of a coin flip.
827 pub info_gain: f64,
828 /// True when this pair was drawn uniformly at random as a calibration
829 /// check rather than chosen by the acquisition function.
830 pub random_check: bool,
831 /// `"random"` (no posterior), `"check"`, or `"bald"`.
832 pub method: &'static str,
833}
834
835/// What a hand edit's commit reported about the edit against the original.
836///
837/// The type exists because the old `as_improvement: bool` could not say the
838/// most informative thing a player can say. "I edited this, listened to both,
839/// and the original was better" is a duel with a known answer — and under a
840/// boolean it was **unrepresentable**: `false` meant "said nothing", so the
841/// loss was silently discarded and the log only ever saw edits that won. A
842/// preference log that records only successes is a biased sample of exactly
843/// the kind the model has no defence against, and hand editing is the richest
844/// signal in the app.
845#[derive(Clone, Copy, Debug, PartialEq, Eq)]
846pub enum EditOutcome {
847 /// Nothing was claimed. Lineage still links the two; no observation.
848 Untold,
849 /// The player heard both and picked. `edited_won: false` is the losing
850 /// direction — the half that used to be inexpressible.
851 Heard {
852 /// True when the edit beat the original.
853 edited_won: bool,
854 },
855 /// The player asserted the edit is better without hearing them back to
856 /// back (the express "my edit is better" checkbox). Same claim, weaker
857 /// evidence, tagged so it can be scored separately.
858 SelfReported,
859}
860
861impl EditOutcome {
862 /// `(edited_won, provenance)` when this outcome makes a claim.
863 pub fn told(&self) -> Option<(bool, Provenance)> {
864 match self {
865 EditOutcome::Untold => None,
866 EditOutcome::Heard { edited_won } => Some((*edited_won, Provenance::HeardEdit)),
867 EditOutcome::SelfReported => Some((true, Provenance::SelfReport)),
868 }
869 }
870}
871
872/// One feature's exact share of a candidate's utility.
873#[derive(Clone, Debug, Serialize, Deserialize)]
874pub struct Contribution {
875 /// φ coordinate name.
876 pub name: String,
877 /// The lens's weight on it.
878 pub theta: f64,
879 /// The candidate's standardized value on it.
880 pub phi_std: f64,
881 /// `theta · phi_std` — this feature's signed share of the utility.
882 pub contribution: f64,
883}
884
885/// Why the model scores one candidate the way it does.
886///
887/// Utility is **exactly linear within a style lens**, so this decomposition
888/// is exact rather than a local surrogate: `Σ contribution = utility`. No
889/// SHAP, no LIME, no approximation error to caveat.
890#[derive(Clone, Debug, Serialize, Deserialize)]
891pub struct Explanation {
892 /// Candidate id.
893 pub id: u64,
894 /// Aligned index of the lens that claims this candidate.
895 pub style: usize,
896 /// That lens's user-given name (`""` if unnamed).
897 pub style_name: String,
898 /// Posterior-mean utility **under that lens** — exactly the sum of the
899 /// contributions. This is the quantity the decomposition explains.
900 pub utility: f64,
901 /// Posterior std of that same lens utility — how sure the model is about
902 /// this score.
903 pub utility_std: f64,
904 /// Posterior-mean **mixture** utility `E[max_k u_k]` — the number the
905 /// bank is ranked by, and the one to show as *the score*.
906 ///
907 /// It is not the same number as `utility`, and it is never smaller: the
908 /// ranking takes the max over lenses inside the expectation, while the
909 /// decomposition necessarily fixes one lens first. Jensen's inequality
910 /// does the rest. Showing `utility` next to a bank ordered by
911 /// `mix_utility` would render a systematically lower number beside the
912 /// row it is supposed to explain.
913 pub mix_utility: f64,
914 /// Posterior probability that `style` really is this candidate's best
915 /// lens. Near 1, `utility ≈ mix_utility` and the explanation is the whole
916 /// story; well below 1, the candidate sits between islands and the gap is
917 /// worth surfacing rather than hiding.
918 pub responsibility: f64,
919 /// Every feature's contribution, sorted by descending magnitude.
920 pub contributions: Vec<Contribution>,
921}
922
923fn sigmoid(x: f64) -> f64 {
924 1.0 / (1.0 + (-x).exp())
925}
926
927/// Binary entropy in nats, guarded at the ends.
928fn binary_entropy(p: f64) -> f64 {
929 let p = p.clamp(1e-12, 1.0 - 1e-12);
930 -p * p.ln() - (1.0 - p) * (1.0 - p).ln()
931}
932
933/// Dueling Thompson sampling, kept for the acquisition A/B (see
934/// [`Acquisition`]). Draw two posterior samples and duel each one's champion;
935/// if they agree, duel the champion against the runner-up.
936fn thompson_pair<R: Rng>(
937 posterior: &TastePosterior,
938 pool: &[Candidate],
939 cands: &[usize],
940 rng: &mut R,
941) -> (usize, usize) {
942 let n = posterior.samples.len();
943 if n == 0 {
944 return (cands[0], cands[1]);
945 }
946 let champion = |s: &auracle_taste::TasteSample, skip: Option<usize>| -> usize {
947 cands
948 .iter()
949 .copied()
950 .filter(|i| Some(*i) != skip)
951 .max_by(|x, y| {
952 s.utility_mix(&pool[*x].phi_std)
953 .total_cmp(&s.utility_mix(&pool[*y].phi_std))
954 })
955 .unwrap_or(cands[0])
956 };
957 let s1 = &posterior.samples[rng.gen_range(0..n)];
958 let s2 = &posterior.samples[rng.gen_range(0..n)];
959 let a = champion(s1, None);
960 let b = champion(s2, None);
961 if a == b {
962 (a, champion(s2, Some(a)))
963 } else {
964 (a, b)
965 }
966}
967
968/// A hashable fingerprint of a feature vector, for de-duplicating the
969/// standardizer's reference sample. Bit patterns of the raw values: identical
970/// candidates featurize deterministically, so exact equality is the right
971/// test and rounding would only invent collisions.
972fn quantize(row: &[f64]) -> Vec<u64> {
973 row.iter().map(|x| x.to_bits()).collect()
974}
975
976/// Unordered key for a candidate pair.
977fn pair_key(a: u64, b: u64) -> (u64, u64) {
978 if a <= b {
979 (a, b)
980 } else {
981 (b, a)
982 }
983}
984
985/// The session engine.
986pub struct Engine {
987 /// Configuration.
988 pub cfg: SessionConfig,
989 /// The patch prior.
990 pub prior: PatchGrammarPrior,
991 /// Standardizer fit on the first pool fill; persisted with the profile.
992 pub standardizer: Option<Arc<Standardizer>>,
993 /// The observation log (source of truth).
994 pub log: ObservationLog,
995 /// The current posterior, if fit (label-aligned).
996 pub posterior: Option<Arc<TastePosterior>>,
997 /// Current session index.
998 pub session: usize,
999 /// The candidate pool.
1000 pub pool: Vec<Candidate>,
1001 /// Evolution/edit history.
1002 pub lineage: Vec<LineageEvent>,
1003 /// Generation counter (one per refinement call).
1004 pub generation: usize,
1005 /// User-given style names (index = aligned style index; empty = unnamed).
1006 pub style_names: Vec<String>,
1007 /// Implicit preference events (logged, not yet modeled).
1008 pub events: Vec<ImplicitEvent>,
1009 /// Out-of-sample duel forecasts, scored before each answer was known.
1010 pub forecasts: Vec<Forecast>,
1011 /// Style shares recorded at each posterior fit. See [`StyleShareRecord`].
1012 style_shares: Vec<StyleShareRecord>,
1013 /// How many times each (unordered) candidate pair has been shown, keyed
1014 /// by stable id. Drives the repeat penalty in [`Engine::next_duel`].
1015 shown_pairs: HashMap<(u64, u64), u32>,
1016 /// How many times each candidate has been offered, by any pairing.
1017 shown_candidates: HashMap<u64, u32>,
1018 /// Duels offered this run (not the same as observations recorded — the
1019 /// user may skip). Paces the random check duels.
1020 duels_shown: usize,
1021 /// The most recently offered *check* pair, so the forecast it produces can
1022 /// be tagged as unbiased even though the frontend records it like any
1023 /// other duel.
1024 last_check_pair: Option<(u64, u64)>,
1025 /// How many times the importance weights have collapsed and been
1026 /// resampled since the last full MCMC fit — the staleness signal behind
1027 /// [`Engine::needs_refit`].
1028 resamples_since_fit: usize,
1029 /// The featurization memo every featurize in this engine consults.
1030 memo: RenderMemo,
1031 /// Ids whose audition buffer is resident under [`RenderPolicy::Lazy`],
1032 /// least recently used first.
1033 audio_lru: VecDeque<u64>,
1034 /// Base seed of the indexed pool-draw stream ([`crate::farm`]). Taken from
1035 /// the caller's RNG on the first fill, so [`Engine::fill_pool_step`]'s
1036 /// signature — and the amount of that RNG's stream a fill consumes — stay
1037 /// what they were.
1038 fill_seed: Option<u64>,
1039 /// Next index of that stream the *fold* will consume. Advances only on
1040 /// absorption, which is what makes it independent of how many draws are in
1041 /// flight.
1042 draw_cursor: u64,
1043 /// Next index handed out by [`Engine::fill_draw`]. Runs ahead of
1044 /// `draw_cursor` by whatever is in flight; speculative distance between
1045 /// them is free, because an index that is never absorbed never happened.
1046 issue_cursor: u64,
1047 next_id: u64,
1048 /// What the last session restore had to mend — see
1049 /// [`Engine::repair_report`]. Saved terms whose knobs were out of range,
1050 /// log cells clamped, and observations dropped as uninterpretable.
1051 repaired_terms: usize,
1052 repaired_cells: usize,
1053 dropped_observations: usize,
1054}
1055
1056impl Engine {
1057 /// Create an engine over the given prior.
1058 pub fn new(prior: PatchGrammarPrior, cfg: SessionConfig) -> Self {
1059 Self {
1060 cfg,
1061 prior,
1062 standardizer: None,
1063 log: ObservationLog::new(),
1064 posterior: None,
1065 session: 0,
1066 pool: Vec::new(),
1067 lineage: Vec::new(),
1068 generation: 0,
1069 style_names: Vec::new(),
1070 events: Vec::new(),
1071 forecasts: Vec::new(),
1072 style_shares: Vec::new(),
1073 shown_pairs: HashMap::new(),
1074 shown_candidates: HashMap::new(),
1075 duels_shown: 0,
1076 last_check_pair: None,
1077 resamples_since_fit: 0,
1078 memo: RenderMemo::default(),
1079 audio_lru: VecDeque::new(),
1080 fill_seed: None,
1081 draw_cursor: 0,
1082 issue_cursor: 0,
1083 next_id: 1,
1084 repaired_terms: 0,
1085 repaired_cells: 0,
1086 dropped_observations: 0,
1087 }
1088 }
1089
1090 /// Replace the featurization memo every featurize in this engine consults
1091 /// — fill, insert, restore, and the refinement surrogate.
1092 ///
1093 /// Shared rather than owned so a frontend can pre-load one and read back
1094 /// what the engine learned. Refinement captures it by clone at
1095 /// [`Engine::refine_one`] time, so swapping it mid-generation is not
1096 /// something to do.
1097 pub fn set_memo(&mut self, memo: RenderMemo) {
1098 self.memo = memo;
1099 }
1100
1101 /// The featurization memo.
1102 pub fn memo(&self) -> &RenderMemo {
1103 &self.memo
1104 }
1105
1106 /// Content address of candidate `id`'s featurization.
1107 pub fn key_of(&self, id: u64) -> Option<&str> {
1108 self.find(id).map(|i| self.pool[i].key.as_str())
1109 }
1110
1111 /// The audition buffer of candidate `id`, materializing it if
1112 /// [`RenderPolicy::Lazy`] deferred it.
1113 ///
1114 /// `None` for an unknown id, for [`RenderPolicy::None`], or for a term
1115 /// that no longer renders — a restored bank outlives the DSP that made it,
1116 /// and a caller that cannot distinguish "not yet" from "never" will wait
1117 /// forever. This is the *only* honest source of that answer.
1118 ///
1119 /// Bit-identical to the buffer the candidate's features were measured on
1120 /// ([`auracle_features::render_playback`]).
1121 ///
1122 /// Shared rather than copied: the pool, the memo and the caller all hold
1123 /// the same ~565 KB allocation through an [`Arc`], so a repeat request is
1124 /// a refcount bump. Callers that must own samples clone the inner value at
1125 /// their own call site, where the cost is visible.
1126 pub fn render_of(&mut self, id: u64) -> Option<Arc<Audition>> {
1127 let i = self.find(id)?;
1128 if let Some(a) = self.pool[i].render.clone() {
1129 if self.cfg.render_policy == RenderPolicy::Lazy {
1130 self.touch_audition(id);
1131 }
1132 return Some(a);
1133 }
1134 if self.cfg.render_policy != RenderPolicy::Lazy {
1135 // Eager already stored one at admission; None keeps nothing.
1136 return None;
1137 }
1138 let key = self.pool[i].key.clone();
1139 let audio = match self.memo.get_audio(&key) {
1140 Some(a) => a,
1141 None => Arc::new(
1142 render_playback(
1143 &self.pool[i].tree,
1144 &self.cfg.phrase,
1145 self.pool[i].features.gain_db,
1146 )
1147 .ok()?,
1148 ),
1149 };
1150 self.pool[i].render = Some(Arc::clone(&audio));
1151 // `touch_audition` may evict other members but never `id`, which it
1152 // marks most-recently-used; the returned handle is valid regardless.
1153 self.touch_audition(id);
1154 Some(audio)
1155 }
1156
1157 /// Mark `id`'s buffer as most recently used and drop whatever falls out of
1158 /// [`SessionConfig::audio_cache`].
1159 fn touch_audition(&mut self, id: u64) {
1160 // Evicted candidates leave their ids behind; dropping them here keeps
1161 // the cache from being consumed by ghosts and holding live buffers
1162 // past the cap.
1163 let live: HashSet<u64> = self.pool.iter().map(|c| c.id).collect();
1164 self.audio_lru.retain(|x| *x != id && live.contains(x));
1165 self.audio_lru.push_back(id);
1166 let cap = self.cfg.audio_cache.max(1);
1167 while self.audio_lru.len() > cap {
1168 let Some(evicted) = self.audio_lru.pop_front() else {
1169 break;
1170 };
1171 if let Some(i) = self.find(evicted) {
1172 self.pool[i].render = None;
1173 }
1174 }
1175 }
1176
1177 /// Whether an admitting featurize should bother producing samples.
1178 ///
1179 /// Only [`RenderPolicy::Eager`] keeps a buffer, so under the other two
1180 /// policies asking for one would convert 141 k samples straight into a
1181 /// `drop`. This is the flag every `featurize_memo` call in the engine
1182 /// passes, and the reason the pool fill under `Lazy` costs φ only.
1183 fn wants_admitted_audio(&self) -> bool {
1184 self.cfg.render_policy == RenderPolicy::Eager
1185 }
1186
1187 /// The audition buffer a freshly-admitted candidate should carry, per
1188 /// policy. `fresh` is the buffer the admitting featurize produced, if it
1189 /// rendered rather than hitting the memo.
1190 fn admitted_render(
1191 &self,
1192 tree: &PatchTree,
1193 features: &Features,
1194 fresh: Option<Arc<Audition>>,
1195 ) -> Option<Arc<Audition>> {
1196 match self.cfg.render_policy {
1197 RenderPolicy::Eager => fresh.or_else(|| {
1198 render_playback(tree, &self.cfg.phrase, features.gain_db)
1199 .ok()
1200 .map(Arc::new)
1201 }),
1202 RenderPolicy::Lazy | RenderPolicy::None => None,
1203 }
1204 }
1205
1206 fn alloc_id(&mut self) -> u64 {
1207 let id = self.next_id;
1208 self.next_id += 1;
1209 id
1210 }
1211
1212 /// Pool index of a candidate id.
1213 pub fn find(&self, id: u64) -> Option<usize> {
1214 self.pool.iter().position(|c| c.id == id)
1215 }
1216
1217 /// Start a new session (its own τ latent). Returns its index.
1218 pub fn begin_session(&mut self) -> usize {
1219 if !self.log.is_empty() {
1220 self.session = self.log.n_sessions();
1221 }
1222 self.session
1223 }
1224
1225 /// Fill the pool with vetted prior draws (up to `pool_size`). Fits the
1226 /// standardizer on the first successful fill.
1227 pub fn fill_pool<R: Rng>(&mut self, rng: &mut R) {
1228 let target = self.cfg.pool_size;
1229 while self.pool.len() < target {
1230 if self.fill_pool_step(rng, target - self.pool.len()) == 0 {
1231 break;
1232 }
1233 }
1234 // Fill fell short (vet failures exhausted the draw budget): fit the
1235 // standardizer on what we have rather than leaving φ un-standardized.
1236 if self.standardizer.is_none() && !self.pool.is_empty() {
1237 let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1238 self.standardizer = Some(Arc::new(Standardizer::fit(&rows)));
1239 for c in &mut self.pool {
1240 c.phi_std = self
1241 .standardizer
1242 .as_ref()
1243 .unwrap()
1244 .transform(&c.features.phi());
1245 }
1246 }
1247 }
1248
1249 /// Add up to `max_new` vetted candidates (bounded by `max_draws`
1250 /// attempts). Returns how many were added — the incremental unit that
1251 /// lets a frontend post progress between batches. Standardization runs
1252 /// once the pool first reaches `pool_size` (or on any later addition).
1253 ///
1254 /// This is the serial fold of the indexed draw stream ([`crate::farm`]):
1255 /// index `i` is consumed whatever its outcome, and dedupe / vetting decide
1256 /// only whether it *lands*. The farm path ([`Engine::fill_draw`] +
1257 /// [`Engine::absorb_prior`]) is the same fold with the render moved
1258 /// off-engine, so the two produce the same pool from the same
1259 /// `fill_seed` — and so does any chunking of `max_new`, because the cursor
1260 /// lives in the engine rather than in a loop variable.
1261 pub fn fill_pool_step<R: Rng>(&mut self, rng: &mut R, max_new: usize) -> usize {
1262 self.ensure_fill_seed(rng);
1263 let mut added = 0;
1264 while added < max_new
1265 && self.pool.len() < self.cfg.pool_size
1266 && self.draw_cursor < self.cfg.max_draws as u64
1267 {
1268 let index = self.draw_cursor;
1269 let Some(tree) = self.draw_at(index) else {
1270 break;
1271 };
1272 self.consume_draw(index);
1273 if self.pool.iter().any(|c| c.tree == tree) {
1274 continue;
1275 }
1276 let want_audio = self.wants_admitted_audio();
1277 if let Ok((cached, audition)) =
1278 featurize_memo(&tree, &self.cfg.phrase, &self.memo, want_audio)
1279 {
1280 self.push_prior(PreFeaturized {
1281 tree,
1282 cached,
1283 audition,
1284 });
1285 added += 1;
1286 }
1287 }
1288 self.standardize_pool();
1289 added
1290 }
1291
1292 // ------------------------------------------------------------------
1293 // The indexed draw stream and its off-engine fold (see `crate::farm`)
1294 // ------------------------------------------------------------------
1295
1296 /// Base seed of this engine's pool-draw stream, taking one from `rng` if
1297 /// the stream has not started yet.
1298 ///
1299 /// Exactly one `u64` is drawn from the caller's RNG per engine, on the
1300 /// first fill. That is deliberate: the serial and farm paths consume the
1301 /// same amount of the caller's stream, so everything downstream of the
1302 /// fill that shares that RNG (duel selection, MCMC) stays aligned between
1303 /// them.
1304 pub fn ensure_fill_seed<R: Rng>(&mut self, rng: &mut R) -> u64 {
1305 match self.fill_seed {
1306 Some(s) => s,
1307 None => {
1308 let s = rng.gen::<u64>();
1309 self.fill_seed = Some(s);
1310 s
1311 }
1312 }
1313 }
1314
1315 /// Base seed of the pool-draw stream, if it has started.
1316 pub fn fill_seed(&self) -> Option<u64> {
1317 self.fill_seed
1318 }
1319
1320 /// Pin the pool-draw stream to an explicit base seed. Only meaningful
1321 /// before the first draw; a fill in progress keeps the seed it started on.
1322 pub fn set_fill_seed(&mut self, seed: u64) {
1323 if self.draw_cursor == 0 && self.issue_cursor == 0 {
1324 self.fill_seed = Some(seed);
1325 }
1326 }
1327
1328 /// Next index of the draw stream the fold will consume.
1329 pub fn draw_cursor(&self) -> u64 {
1330 self.draw_cursor
1331 }
1332
1333 /// The term at `index` of this engine's draw stream — a pure function of
1334 /// `(fill_seed, index)` and the prior, costing microseconds and no render.
1335 ///
1336 /// This is what makes a lost farm job re-issuable with no retained state:
1337 /// the job *is* its index.
1338 pub fn draw_at(&self, index: u64) -> Option<PatchTree> {
1339 let base = self.fill_seed?;
1340 let mut sub = StdRng::seed_from_u64(draw_seed(base, index));
1341 Some(self.prior.sample_with_rng(&mut sub))
1342 }
1343
1344 /// Hand out up to `n` unrendered draws for off-engine featurization.
1345 ///
1346 /// Returns fewer than `n` — or nothing — when the pool has as much work
1347 /// outstanding as it can still use, or the `max_draws` budget is spent.
1348 /// An empty return is *not* by itself a stop signal: it may simply mean
1349 /// every slot the pool can still fill is already in flight. The caller
1350 /// stops when the pool reaches its target, or when an empty return
1351 /// coincides with nothing outstanding.
1352 ///
1353 /// Requires a started stream ([`Engine::ensure_fill_seed`] or
1354 /// [`Engine::set_fill_seed`]); yields nothing otherwise.
1355 pub fn fill_draw(&mut self, n: usize) -> Vec<Draw> {
1356 let mut out = Vec::new();
1357 if self.fill_seed.is_none() {
1358 return out;
1359 }
1360 let need = self.cfg.pool_size.saturating_sub(self.pool.len());
1361 if need == 0 {
1362 return out;
1363 }
1364 // Over-issue by a quarter for vet failures and duplicates, plus one so
1365 // a single remaining slot still gets a second attempt in flight. More
1366 // than that is not wrong — over-issue is discardable by construction —
1367 // just wasted work on a machine that could have been rendering
1368 // something the pool will keep.
1369 let ceiling = (need + need / 4 + 1) as u64;
1370 let outstanding = self.issue_cursor.saturating_sub(self.draw_cursor);
1371 let room = ceiling.saturating_sub(outstanding);
1372 for _ in 0..(n as u64).min(room) {
1373 if self.issue_cursor >= self.cfg.max_draws as u64 {
1374 break;
1375 }
1376 let index = self.issue_cursor;
1377 let Some(tree) = self.draw_at(index) else {
1378 break;
1379 };
1380 let dup = self.pool.iter().any(|c| c.tree == tree);
1381 self.issue_cursor = index + 1;
1382 out.push(Draw { index, tree, dup });
1383 }
1384 out
1385 }
1386
1387 /// Fold one off-engine result into the pool.
1388 ///
1389 /// `index` **must** be [`Engine::draw_cursor`] — results are absorbed in
1390 /// index order, and that ordering is the entire determinism argument: the
1391 /// pool at index `i` is a pure function of indices `< i`, so it cannot
1392 /// depend on how many renders were running. Anything else is refused
1393 /// (returns `None` without consuming), because silently absorbing out of
1394 /// order would produce a pool no width reproduces.
1395 ///
1396 /// `pre` is `None` for a draw the farm rejected — a vet failure, a
1397 /// compile failure, or a result that failed to survive transport. The
1398 /// index is consumed either way, exactly as a failed draw burns an attempt
1399 /// in the serial loop.
1400 ///
1401 /// Returns the new candidate id, or `None` when the draw did not land
1402 /// (rejected, duplicate, or the pool was already full).
1403 pub fn absorb_prior(&mut self, index: u64, pre: Option<PreFeaturized>) -> Option<u64> {
1404 if index != self.draw_cursor
1405 || self.pool.len() >= self.cfg.pool_size
1406 || index >= self.cfg.max_draws as u64
1407 {
1408 return None;
1409 }
1410 self.consume_draw(index);
1411 let mut id = None;
1412 if let Some(pre) = pre {
1413 if !self.pool.iter().any(|c| c.tree == pre.tree) {
1414 id = Some(self.push_prior(pre));
1415 }
1416 }
1417 self.standardize_pool();
1418 id
1419 }
1420
1421 /// Mark index `index` as folded in, whatever its outcome.
1422 fn consume_draw(&mut self, index: u64) {
1423 self.draw_cursor = index + 1;
1424 self.issue_cursor = self.issue_cursor.max(self.draw_cursor);
1425 }
1426
1427 /// Admit a prior draw whose featurization is already done. The single push
1428 /// site for [`Origin::Prior`], shared by the serial and farm paths so
1429 /// there is no second copy of the admission rules to drift.
1430 fn push_prior(&mut self, pre: PreFeaturized) -> u64 {
1431 let PreFeaturized {
1432 tree,
1433 cached,
1434 audition,
1435 } = pre;
1436 // Fold the off-engine work into this engine's memo: a farm render is
1437 // exactly the artifact a later audition or refinement would otherwise
1438 // recompute, and the memo is where every other path looks for it.
1439 self.memo.put(cached.clone(), audition.clone());
1440 let id = self.alloc_id();
1441 let render = self.admitted_render(&tree, &cached.features, audition);
1442 self.pool.push(Candidate {
1443 id,
1444 tree: settled(tree),
1445 phi_std: Vec::new(),
1446 key: cached.key,
1447 render,
1448 features: cached.features,
1449 origin: Origin::Prior,
1450 name: None,
1451 pinned: false,
1452 });
1453 id
1454 }
1455
1456 /// Fit the standardizer once the pool first reaches `pool_size`, then give
1457 /// every un-standardized member its φ_std. The tail of a fill step, lifted
1458 /// so the serial and farm paths run the identical bookkeeping.
1459 fn standardize_pool(&mut self) {
1460 if self.standardizer.is_none() && self.pool.len() >= self.cfg.pool_size {
1461 let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1462 self.standardizer = Some(Arc::new(Standardizer::fit(&rows)));
1463 }
1464 let Some(sz) = self.standardizer.clone() else {
1465 return;
1466 };
1467 for c in &mut self.pool {
1468 if c.phi_std.is_empty() {
1469 c.phi_std = sz.transform(&c.features.phi());
1470 }
1471 }
1472 }
1473
1474 /// Give every pool member a φ_std **now**, fitting a standardizer from
1475 /// the current pool if none exists yet — so a *partially filled* pool is
1476 /// already duel-able.
1477 ///
1478 /// [`Engine::fill_pool_step`] only fits once the pool reaches
1479 /// `pool_size`, and that single condition is what forces a frontend to sit
1480 /// out the entire fill before it can ask its first question:
1481 /// [`Engine::next_duel_full`] skips candidates whose `phi_std` is empty,
1482 /// so a half-filled pool contains no legal pair at all. This is the
1483 /// escape hatch a progressive boot needs — it costs no renders, only the
1484 /// mean/variance of what has already been drawn.
1485 ///
1486 /// It never *replaces* an existing standardizer. θ is only meaningful
1487 /// relative to the standardization its φ were measured under, so an
1488 /// imported profile's geometry has to survive a boot that tops the pool
1489 /// up ([`Engine::import_profile`]). Re-fitting is
1490 /// [`Engine::restandardize_if_untaught`]'s job, and it is only safe
1491 /// before a posterior exists.
1492 pub fn standardize_now(&mut self) {
1493 if self.pool.is_empty() {
1494 return;
1495 }
1496 if self.standardizer.is_none() {
1497 let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1498 self.standardizer = Some(Arc::new(Standardizer::fit(&rows)));
1499 }
1500 let sz = self.standardizer.clone().expect("just fit above");
1501 for c in &mut self.pool {
1502 if c.phi_std.is_empty() {
1503 c.phi_std = sz.transform(&c.features.phi());
1504 }
1505 }
1506 }
1507
1508 /// Re-fit the standardizer over the finished pool — a no-op the moment a
1509 /// posterior exists.
1510 ///
1511 /// A progressive boot fits a *provisional* standardizer over the first
1512 /// handful of draws ([`Engine::standardize_now`]) so the user can start
1513 /// voting; the completed pool is a better reference population, and
1514 /// re-expressing φ on it is lossless because the log stores **raw**
1515 /// values (`refit_standardizer`'s rationale). But once θ has
1516 /// been fit, its coordinates are denominated in the standardizer that was
1517 /// live at fit time, and moving the scale under a live posterior would
1518 /// silently rescale every utility in the app. So this refuses in exactly
1519 /// that case: the next [`Engine::fit_posterior`] re-fits both together,
1520 /// in the order that keeps them consistent.
1521 pub fn restandardize_if_untaught(&mut self) {
1522 if self.posterior.is_none() {
1523 self.refit_standardizer();
1524 }
1525 }
1526
1527 /// Re-fit the standardizer over everything the model is about to see: the
1528 /// raw φ in the log **and** the live pool.
1529 ///
1530 /// Fitting it once on the first 40 prior draws and freezing it meant that
1531 /// as the pool drifted toward refined candidates the z-scores drifted with
1532 /// it, and the linear model ended up extrapolating well outside the range
1533 /// it was calibrated on. Because the log now stores raw values, re-fitting
1534 /// is free and lossless — it re-expresses the same evidence on a scale
1535 /// that still matches where the data actually is.
1536 fn refit_standardizer(&mut self) {
1537 let names = phi_names();
1538 // The reference population is *the patches the user has encountered*,
1539 // each counted once — the live pool plus anything in the log that has
1540 // since been evicted. Deliberately not the multiset of comparisons:
1541 // acquisition decides which candidates get dueled repeatedly, and
1542 // letting that decide the coordinate system closes a feedback loop
1543 // between the question-asker and the units the answers are measured
1544 // in. Same reason the standardizer exists at all.
1545 let mut rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
1546 let mut seen: HashSet<Vec<u64>> = rows.iter().map(|r| quantize(r)).collect();
1547 for row in self.log.raw_rows(&names) {
1548 // Width guard, belt-and-braces: a ragged row reaches an assertion
1549 // inside `Standardizer::fit` and panics the whole engine. A log
1550 // that survived a bad migration should cost us that vote, not the
1551 // session.
1552 if row.len() == names.len() && seen.insert(quantize(&row)) {
1553 rows.push(row);
1554 }
1555 }
1556 if rows.is_empty() {
1557 return;
1558 }
1559 let sz = Arc::new(Standardizer::fit(&rows));
1560 for c in &mut self.pool {
1561 c.phi_std = sz.transform(&c.features.phi());
1562 }
1563 self.standardizer = Some(sz);
1564 }
1565
1566 /// Fit (or re-fit) the taste posterior from the observation log. The
1567 /// stored posterior is label-aligned (safe for per-style summaries) and
1568 /// its importance weights are reset to uniform.
1569 pub fn fit_posterior<R: Rng>(&mut self, rng: &mut R) {
1570 if self.log.is_empty() {
1571 return;
1572 }
1573 self.refit_standardizer();
1574 let Some(sz) = self.standardizer.clone() else {
1575 return;
1576 };
1577 let names = phi_names();
1578 let d = names.len();
1579 // Style capacity grows with evidence: one lens per ~20 observations,
1580 // capped by config. Idle lenses collapse to ~0 share on their own,
1581 // so K is an upper bound the data may or may not use.
1582 let k = (1 + self.log.len() / 20).min(self.cfg.k_styles).max(1);
1583 let mut taste_cfg = TasteConfig::mixture(d, k);
1584 taste_cfg.recency_half_life = self.cfg.recency_half_life;
1585 // The brightness cluster shares a latent mean per style. Resolved by
1586 // *name* here because this is the layer that knows them; the taste
1587 // crate is handed indices and never learns what they mean. A name that
1588 // is not in φ simply does not join the group, so a stimulus-tag bump
1589 // or a dropped column degrades to the flat prior rather than panicking
1590 // or silently fusing the wrong coordinate.
1591 let bright: Vec<usize> = ["rolloff_mean", "zcr_mean", "centroid_mean"]
1592 .iter()
1593 .filter_map(|want| names.iter().position(|n| n.split(':').next() == Some(want)))
1594 .collect();
1595 if bright.len() > 1 {
1596 taste_cfg.fused = vec![bright];
1597 }
1598 let model = TasteModel::new(taste_cfg);
1599 let data = FitSet::build(&self.log, &names, &sz);
1600 let posterior = model.fit(rng, &data, self.cfg.mcmc_samples, self.cfg.mcmc_warmup);
1601 let posterior = Arc::new(posterior.aligned());
1602 // Measured against the pool the fit is about to be used on, which is
1603 // the population the shares are a statement about — not against the
1604 // log, whose φ are the things already judged.
1605 let pool_phis: Vec<Vec<f64>> = self
1606 .pool
1607 .iter()
1608 .filter(|c| !c.phi_std.is_empty())
1609 .map(|c| c.phi_std.clone())
1610 .collect();
1611 if !pool_phis.is_empty() {
1612 self.style_shares.push(StyleShareRecord {
1613 observations: self.log.len(),
1614 k,
1615 shares: posterior.style_share(&pool_phis),
1616 });
1617 }
1618 self.posterior = Some(posterior);
1619 self.resamples_since_fit = 0;
1620 }
1621
1622 /// Style shares recorded at each fit, oldest first. See
1623 /// [`StyleShareRecord`].
1624 pub fn style_shares(&self) -> &[StyleShareRecord] {
1625 &self.style_shares
1626 }
1627
1628 /// Effective sample size of the current posterior's importance weights —
1629 /// how much of the draw set still carries information after the
1630 /// observations folded in since the last full fit. `None` before the
1631 /// first fit.
1632 pub fn posterior_ess(&self) -> Option<f64> {
1633 self.posterior.as_ref().map(|p| p.ess())
1634 }
1635
1636 /// True when the cheap between-fit updates have run out of road and a
1637 /// full MCMC refit is worth its seconds: the weights have collapsed
1638 /// (ESS below half the draws) at least once since the last fit, or the
1639 /// log has evidence no posterior has seen. A frontend can drive refits
1640 /// off this instead of a fixed vote count.
1641 pub fn needs_refit(&self) -> bool {
1642 match &self.posterior {
1643 Some(_) => self.resamples_since_fit > 0,
1644 None => !self.log.is_empty(),
1645 }
1646 }
1647
1648 /// Posterior-mean mixture utility of a standardized φ (0 with no
1649 /// posterior).
1650 pub fn utility_of(&self, phi_std: &[f64]) -> f64 {
1651 match &self.posterior {
1652 Some(p) if !phi_std.is_empty() => p.utility_mix(phi_std).0,
1653 _ => 0.0,
1654 }
1655 }
1656
1657 /// Did the step from `prev` to `next` touch any locked address?
1658 /// "Touch" = change its value, delete it, **or create it** (structure
1659 /// moves that would rewrite a locked module's path are rejected too —
1660 /// locked means *don't touch*).
1661 ///
1662 /// Both directions are checked, and that is not pedantry. Scanning only
1663 /// `prev` lets a *birth* at a locked address through while rejecting the
1664 /// death that would undo it. The constraint region is then asymmetric —
1665 /// x → x′ allowed, x′ → x rejected — which breaks detailed balance and
1666 /// makes the Metropolis-within-Gibbs argument for locking being exact
1667 /// simply false. The chain would drift into locked structure it can never
1668 /// leave.
1669 ///
1670 /// **What this does and does not guarantee.** `locked` is a set of exact
1671 /// address strings, typically snapshotted from the UI. Every address in
1672 /// it is frozen, in both directions, and *that* is exact. It is not the
1673 /// same as freezing a module: a structural move that grows a brand-new
1674 /// address inside a locked module — one that was in neither trace when
1675 /// the set was taken, so it cannot be in the set — is not caught. That
1676 /// case is symmetric (unmatched by construction in both directions), so
1677 /// it costs nothing in detailed balance; it just means "locked" is a
1678 /// guarantee about *addresses*, not about subtrees.
1679 pub fn violates_locks(prev: &Trace, next: &Trace, locked: &HashSet<String>) -> bool {
1680 if locked.is_empty() {
1681 return false;
1682 }
1683 for (addr, c) in &prev.choices {
1684 if locked.contains(&**addr) {
1685 match next.choices.get(addr) {
1686 Some(n) if n.value == c.value => {}
1687 _ => return true,
1688 }
1689 }
1690 }
1691 for addr in next.choices.keys() {
1692 if locked.contains(&**addr) && !prev.choices.contains_key(addr) {
1693 return true;
1694 }
1695 }
1696 false
1697 }
1698
1699 /// Grammar prior with kind-weights tilted toward the fitted taste: each
1700 /// structural θ component (share-weighted across styles) multiplies its
1701 /// kind's proposal weight by `exp(η·θ)`. This is θ_struct → grammar
1702 /// feedback — refinement *proposes* toward the user instead of merely
1703 /// filtering, which is where visible directionality comes from.
1704 ///
1705 /// Two things make the mapping from φ names to grammar weights less than
1706 /// a lookup, and both are consequences of φ carrying **families**
1707 /// (`auracle_features::StructFeatures`):
1708 ///
1709 /// - Several kinds share one coefficient. `n_drive` speaks for the
1710 /// wavefolder, the distortion, the bitcrusher and the ring modulator;
1711 /// `n_mod_fx` for the chorus, phaser, flanger, tremolo and vibrato;
1712 /// `n_time` for the delay, the granulator and the pitch shifter;
1713 /// `n_filter` for the filter, the EQ and the vocoder; `n_dynamics` for
1714 /// the compressor, the ducker and the gate. They each get the family's
1715 /// tilt, which is the honest reading: the evidence never distinguished
1716 /// them, so the proposal should not pretend it did. Their *base* weights
1717 /// still differ, so the tilt shifts the family without flattening it.
1718 /// - `n_mix` is not in φ at all — it is determined by the source count and
1719 /// the other five binary counts under the exact identity that removed it
1720 /// — so its tilt comes from the sources: wanting more sources is wanting
1721 /// more binary nodes to combine them, and that is the only sense in
1722 /// which the taste model has an opinion here. The other five binaries
1723 /// take the tilt of whichever family they are counted under.
1724 ///
1725 /// Each coefficient is also **shrunk by its own posterior uncertainty**,
1726 /// `θ·|θ|/(|θ| + σ)`, before it tilts anything. The new palette's
1727 /// coefficients are the least identified ones in the model — a pool of 48
1728 /// draws contains a handful of bitcrushers — so a raw posterior mean is
1729 /// as likely to be sampling noise as signal, and feeding noise into the
1730 /// *proposal* distribution compounds it: the pool drifts toward the
1731 /// spurious kind, which produces more evidence about it, which is not the
1732 /// same as producing more evidence *for* it. A coefficient whose σ equals
1733 /// its mean tilts half as hard; one that is mostly noise tilts not at all.
1734 fn biased_prior(&self) -> PatchGrammarPrior {
1735 let mut prior = self.prior.clone();
1736 let eta = self.cfg.proposal_tilt;
1737 let Some(p) = &self.posterior else {
1738 return prior;
1739 };
1740 if eta <= 0.0 {
1741 return prior;
1742 }
1743 let names = Features::phi_names();
1744 let pool_phis: Vec<Vec<f64>> = self
1745 .pool
1746 .iter()
1747 .filter(|c| !c.phi_std.is_empty())
1748 .map(|c| c.phi_std.clone())
1749 .collect();
1750 let shares = p.style_share(&pool_phis);
1751 let (mut theta, mut sd) = (vec![0.0; names.len()], vec![0.0; names.len()]);
1752 for k in 0..p.k_styles() {
1753 let w = shares.get(k).copied().unwrap_or(0.0);
1754 for (t, mi) in theta.iter_mut().zip(p.theta_mean(k)) {
1755 *t += w * mi;
1756 }
1757 for (s, si) in sd.iter_mut().zip(p.theta_std(k)) {
1758 *s += w * si;
1759 }
1760 }
1761 let g = |name: &str| {
1762 names
1763 .iter()
1764 .position(|n| *n == name)
1765 .map(|i| shrink(theta[i], sd[i]))
1766 .unwrap_or(0.0)
1767 };
1768 let sources = [
1769 g("n_vco"),
1770 g("n_supersaw"),
1771 g("n_noise"),
1772 g("n_wavetable"),
1773 g("n_pluck"),
1774 g("n_formant"),
1775 // `Silence` is deliberately **not** tilted by taste, and this zero
1776 // is the whole of that decision. The tilt exists to move proposals
1777 // toward source kinds the listener is enjoying; a hole is not a
1778 // timbre anyone can enjoy, it is the absence of one, and its
1779 // prevalence is meant to come from a player unplugging a socket
1780 // rather than from a fitted coefficient.
1781 //
1782 // It is also the column where a tilt would be least trustworthy.
1783 // At a 0.5% prior rate `n_silence` is zero in nearly every row a
1784 // fit sees — the near-indicator shape that kept `n_ringmod` out of
1785 // φ as a column of its own. `shrink` would damp a spurious
1786 // coefficient, but the multiplier it feeds is exponential, and
1787 // amplifying holes into the pool is a failure a listener notices
1788 // immediately.
1789 0.0,
1790 ];
1791 let src = tilt_weights(&prior.source_weights, &sources, eta);
1792 prior.source_weights = src.try_into().expect("source weight arity");
1793 // Mix inherits the sources' average tilt — it is the node that exists
1794 // to combine them, and it is the one column the identity removed.
1795 //
1796 // Wave 3 tried to replace this proxy with a measurement: a
1797 // `branch_width_max` φ coordinate, so that "I like parallel routing"
1798 // would be a thing a user could say and this line could hand back. The
1799 // VIF sweep threw the column out (10.4, and it took every source count
1800 // with it), and the reason is the same identity that removed `n_mix`
1801 // in the first place — the leaf count is `1 + Σ binaries` exactly, so
1802 // a patch cannot gain a mixer without gaining a source. Which means
1803 // this proxy was never a proxy. Wanting more sources *is* wanting more
1804 // binaries, as an equation, and the average below is reading the
1805 // evidence for both. The wave-3 coordinates that did survive
1806 // (`chain_balance`, `frac_sidechained`, `mod_at_source`) describe how
1807 // a patch is arranged rather than how wide it is, and none of them
1808 // maps onto a single production's weight, so none of them belongs
1809 // here: a tilt is a claim about one categorical outcome, and
1810 // "asymmetric" is not an outcome any one production produces.
1811 let binary_tilt = sources.iter().sum::<f64>() / sources.len() as f64;
1812 let (drive, mod_fx) = (g("n_drive"), g("n_mod_fx"));
1813 // `n_filter` and `n_time` are families now too — the eq and the
1814 // vocoder are counted under the first, the granulator and the pitch
1815 // shifter under the second — so every member of each takes the same
1816 // tilt, exactly as the drive and movement families already did.
1817 let (spectral, time) = (g("n_filter"), g("n_time"));
1818 // Wave 2B's three level-shapers share one coefficient for the same
1819 // reason: the evidence never distinguished a compressor from a ducker
1820 // from a gate, so the proposal must not pretend it did.
1821 let dynamics = g("n_dynamics");
1822 let op = tilt_weights(
1823 &prior.op_weights,
1824 &[
1825 binary_tilt, // mix
1826 spectral, // filter
1827 drive, // fold
1828 time, // delay
1829 mod_fx, // chorus
1830 g("n_reverb"), // reverb
1831 drive, // distortion
1832 drive, // bitcrush
1833 mod_fx, // phaser
1834 drive, // ring mod — counted inside n_drive
1835 mod_fx, // flanger
1836 mod_fx, // tremolo
1837 mod_fx, // vibrato
1838 spectral, // eq — counted inside n_filter
1839 time, // granular — counted inside n_time
1840 time, // pitch shift — counted inside n_time
1841 dynamics, // compressor
1842 dynamics, // ducker
1843 dynamics, // gate
1844 spectral, // vocoder — counted inside n_filter
1845 ],
1846 eta,
1847 );
1848 prior.op_weights = op.try_into().expect("op weight arity");
1849 // "no modulation" carries no tilt — only the filled kinds compete.
1850 // Wave 2C's three take their family's coefficient, on the same rule as
1851 // the op table: the euclidean generator and the two recursive
1852 // productions are all counted inside `n_mod_logic` or `n_mod_shape`,
1853 // and the evidence never separated a quantizer from a slew limiter.
1854 //
1855 // `Op` reads `n_mod_shape` and `Pair` reads `n_mod_logic`, but the
1856 // euclid — which is a *leaf* — reads `n_mod_logic` too, because that
1857 // is the column it is counted in. Tilting it by anything else would
1858 // move the prior in a direction no observation supports.
1859 let (shape, logic) = (g("n_mod_shape"), g("n_mod_logic"));
1860 let md = tilt_weights(
1861 &prior.mod_weights,
1862 &[
1863 0.0,
1864 g("n_lfo"),
1865 g("n_env"),
1866 g("n_rand"),
1867 g("n_follow"),
1868 logic, // euclid — counted inside n_mod_logic
1869 shape, // op
1870 logic, // pair
1871 ],
1872 eta,
1873 );
1874 prior.mod_weights = md.try_into().expect("mod weight arity");
1875 prior
1876 }
1877
1878 /// Posterior probability that pool member `a` beats `b` in a duel
1879 /// (`None` before the first fit).
1880 pub fn predict_duel(&self, a: usize, b: usize) -> Option<f64> {
1881 let p = self.posterior.as_ref()?;
1882 let (pa, pb) = (&self.pool[a].phi_std, &self.pool[b].phi_std);
1883 if pa.is_empty() || pb.is_empty() {
1884 return None;
1885 }
1886 Some(p.prob_prefers(pa, pb))
1887 }
1888
1889 /// Log an implicit preference event (promote, play time, …). Logged
1890 /// only — not yet part of the likelihood.
1891 pub fn log_event(&mut self, kind: &str, id: u64, value: f64) {
1892 self.log_event_detail(kind, id, value, "", Vec::new(), Vec::new());
1893 }
1894
1895 /// The same, carrying the editor's detail and (for a transition) the raw
1896 /// φ on both sides of it. See [`ImplicitEvent`] for why the detail is an
1897 /// opaque string and why none of this reaches the likelihood.
1898 pub fn log_event_detail(
1899 &mut self,
1900 kind: &str,
1901 id: u64,
1902 value: f64,
1903 detail: &str,
1904 phi_before: Vec<f64>,
1905 phi_after: Vec<f64>,
1906 ) {
1907 self.events.push(ImplicitEvent {
1908 kind: kind.into(),
1909 id,
1910 value,
1911 session: self.session,
1912 detail: detail.into(),
1913 phi_before,
1914 phi_after,
1915 });
1916 }
1917
1918 /// Name (or rename; empty clears) an aligned style index.
1919 pub fn set_style_name(&mut self, k: usize, name: &str) {
1920 if k >= 16 {
1921 return;
1922 }
1923 if self.style_names.len() <= k {
1924 self.style_names.resize(k + 1, String::new());
1925 }
1926 self.style_names[k] = name.trim().chars().take(24).collect();
1927 }
1928
1929 /// Run locked MH refinement from one seed. Returns the end state if it
1930 /// differs from the seed.
1931 fn refine_one<R: Rng>(
1932 &self,
1933 rng: &mut R,
1934 seed: &PatchTree,
1935 locked: &HashSet<String>,
1936 steps: usize,
1937 ) -> Option<PatchTree> {
1938 let (posterior, standardizer) = match (&self.posterior, &self.standardizer) {
1939 (Some(p), Some(s)) => (Arc::clone(p), Arc::clone(s)),
1940 _ => return None,
1941 };
1942 let fitness = SurrogateFitness {
1943 posterior,
1944 standardizer,
1945 phrase: self.cfg.phrase.clone(),
1946 memo: self.memo.clone(),
1947 };
1948 let model = EvolutionModel::new(self.biased_prior(), fitness).with_beta(self.cfg.beta);
1949 let mut chain = EvolutionChain::new(model);
1950 let mut trace = chain.init_from(seed)?;
1951
1952 // Scale steps for proposals wasted on locked sites. The kernel picks a
1953 // target site uniformly over all of them, so with a fraction `f` free
1954 // only `f` of the proposals can be accepted and the walk needs `1/f`
1955 // times as many steps to travel as far.
1956 //
1957 // **The cap is a cost bound, not a correction**, and it is stated
1958 // rather than left silent. Past 75% of sites locked, `LOCK_SCALE_CAP`
1959 // stops the compensation short — a patch with 90% of its sites pinned
1960 // would otherwise ask for ten times the budget, and a `⚡ evolve from
1961 // this` on a heavily-pinned patch is a button press with a person
1962 // waiting behind it. So a very heavily locked walk *does* explore less
1963 // than the config nominally buys. That is the intended trade; the thing
1964 // to avoid is believing otherwise.
1965 let total_sites = trace.choices.len().max(1);
1966 let locked_present = trace
1967 .choices
1968 .keys()
1969 .filter(|a| locked.contains(&***a))
1970 .count();
1971 let free = total_sites.saturating_sub(locked_present).max(1);
1972 let factor = (total_sites as f64 / free as f64).min(LOCK_SCALE_CAP);
1973 let steps = ((steps as f64) * factor).ceil() as usize;
1974
1975 let mut current = seed.clone();
1976 // The elite archive, and it is **free**.
1977 //
1978 // Every trace the kernel hands back is already scored under the target
1979 // program, so `total_log_weight()` *is* `log π_β = log p_grammar +
1980 // β·E[u]` for the state it accompanies — no extra model execution, no
1981 // extra featurization, one f64 compare per step.
1982 //
1983 // Scored on the target rather than on fitness alone, which is the
1984 // choice worth stating. Taking the argmax of `E[u]` would discard the
1985 // parsimony half of the very distribution the walk is sampling, and it
1986 // would do so with a bias: a bigger term has more modules to score
1987 // well with, so fitness-argmax systematically returns the largest tree
1988 // the walk touched. `log π_β` is what the walk is climbing, so it is
1989 // what "the best point this walk found" has to mean.
1990 //
1991 // The seed is in the archive. A walk that never improves on where it
1992 // started therefore returns the seed and is filtered to `None` below,
1993 // instead of injecting whatever it happened to be standing on at step
1994 // 40 — which is what `Last` does, and is the thing being A/B'd.
1995 let mut best: Option<(f64, PatchTree)> = match self.cfg.refine_keep {
1996 RefineKeep::Last => None,
1997 RefineKeep::Best => Some((trace.total_log_weight(), seed.clone())),
1998 };
1999 for _ in 0..steps {
2000 let (g, t) = chain.step(rng, &trace);
2001 if Self::violates_locks(&trace, &t, locked) {
2002 continue; // reject outside the kernel; stay at `trace`
2003 }
2004 if let Some((best_w, best_tree)) = &mut best {
2005 let w = t.total_log_weight();
2006 if w > *best_w {
2007 *best_w = w;
2008 *best_tree = g.clone();
2009 }
2010 }
2011 current = g;
2012 trace = t;
2013 }
2014 if let Some((_, best_tree)) = best {
2015 current = best_tree;
2016 }
2017 // The mutation boundary, and the reason the clamp is *here* rather than
2018 // at the knob that draws the number: everything downstream of this line
2019 // — φ, the observation log, the faceplate, the exported PNG — takes the
2020 // term as given, so a value that leaves this function wrong is wrong in
2021 // six places by the time anyone can see it.
2022 //
2023 // The kernel should never produce one. Every continuous site is
2024 // `Uniform(0,1)`, whose `log_prob` is −∞ outside the unit interval, so
2025 // a proposal that escapes scores `log α = −∞` and is rejected — and
2026 // that is measured, not assumed: `auracle-grammar --example
2027 // mh_escape` runs 8 chains × 20 000 single-site transitions through
2028 // this exact kernel and observes zero escapes. So this is a belt on a
2029 // proven brace, costing one trace walk per accepted child, and its real
2030 // job is to be the line that has to be deleted before the invariant can
2031 // be broken again.
2032 debug_assert_eq!(
2033 current.domain_violations().len(),
2034 0,
2035 "MH seated an out-of-domain site: {:?}",
2036 current.domain_violations()
2037 );
2038 current.clamp_domains();
2039 (current != *seed).then_some(current)
2040 }
2041
2042 /// Insert a candidate (evicting the worst if full, never `protect`).
2043 /// Returns the new id, or `None` if the newcomer ranks below the evictee.
2044 fn insert_candidate(
2045 &mut self,
2046 tree: PatchTree,
2047 origin: Origin,
2048 protect: Option<u64>,
2049 ) -> Option<u64> {
2050 let standardizer = self.standardizer.as_ref()?;
2051 // Memoized: refinement and the edit bench both featurize the tree they
2052 // hand here, so on every one of those paths this is a hit.
2053 let want_audio = self.wants_admitted_audio();
2054 let (cf, fresh) = featurize_memo(&tree, &self.cfg.phrase, &self.memo, want_audio).ok()?;
2055 let phi_std = standardizer.transform(&cf.features.phi());
2056 let mean_new = self.utility_of(&phi_std);
2057 if self.pool.len() >= self.cfg.pool_size {
2058 // Rank un-standardized members as *worst*, explicitly, rather
2059 // than letting `utility_of` score them 0.0 and land them
2060 // mid-pack above genuinely-disliked patches. Today this cannot
2061 // happen — the `?` above means a standardizer exists, and every
2062 // path that admits a candidate under one also transforms its φ —
2063 // but that is an invariant three functions away, and the same
2064 // "empty φ scores exactly zero" reasoning already produced one
2065 // live bug in duel selection. Cheaper to be unconditionally right
2066 // here than to rely on the invariant holding after the next edit.
2067 let rank = |c: &Candidate| (!c.phi_std.is_empty(), self.utility_of(&c.phi_std));
2068 let worst = self
2069 .pool
2070 .iter()
2071 .enumerate()
2072 .filter(|(_, c)| Some(c.id) != protect && !c.pinned)
2073 .min_by(|(_, x), (_, y)| {
2074 let (sx, ux) = rank(x);
2075 let (sy, uy) = rank(y);
2076 sx.cmp(&sy).then(ux.total_cmp(&uy))
2077 })
2078 .map(|(i, c)| (i, self.utility_of(&c.phi_std)));
2079 match worst {
2080 Some((worst_idx, worst_mean)) => {
2081 // Hand edits always land (the user asked for them);
2082 // refined candidates must earn their slot.
2083 if origin == Origin::Refined && mean_new <= worst_mean {
2084 return None;
2085 }
2086 self.pool.swap_remove(worst_idx);
2087 }
2088 None => return None,
2089 }
2090 }
2091 let id = self.alloc_id();
2092 let render = self.admitted_render(&tree, &cf.features, fresh);
2093 self.pool.push(Candidate {
2094 id,
2095 tree: settled(tree),
2096 phi_std,
2097 key: cf.key,
2098 render,
2099 features: cf.features,
2100 origin,
2101 name: None,
2102 pinned: false,
2103 });
2104 Some(id)
2105 }
2106
2107 /// Taste-guided refinement: run fugue-evo typed MH on the Boltzmann
2108 /// target from each of the top seeds, and add improved, vetted, novel
2109 /// candidates to the pool (evicting the worst if full). Each injection
2110 /// is recorded as a lineage event.
2111 pub fn refine<R: Rng>(&mut self, rng: &mut R) {
2112 for parent_id in self.refine_begin() {
2113 self.refine_seed(rng, parent_id);
2114 }
2115 }
2116
2117 /// Open a generation and return the parent ids it will refine from, best
2118 /// first. Empty if there is nothing to refine toward yet (no posterior),
2119 /// in which case the generation counter is **not** advanced.
2120 ///
2121 /// This exists so a caller can drive refinement one seed at a time and
2122 /// report progress between seeds. A whole generation is tens of seconds of
2123 /// render-bound work — running it as one opaque call is what made the app
2124 /// look hung.
2125 pub fn refine_begin(&mut self) -> Vec<u64> {
2126 if self.posterior.is_none() || self.standardizer.is_none() {
2127 return Vec::new();
2128 }
2129 self.generation += 1;
2130 self.ranked()
2131 .iter()
2132 .take(self.cfg.refine_seeds)
2133 .map(|&(i, _, _)| self.pool[i].id)
2134 .collect()
2135 }
2136
2137 /// Refine from one seed of the open generation. Returns the injected child
2138 /// id, or `None` if the walk was rejected or landed on a patch the pool
2139 /// already holds.
2140 pub fn refine_seed<R: Rng>(&mut self, rng: &mut R, parent_id: u64) -> Option<u64> {
2141 let seed = self.pool[self.find(parent_id)?].tree.clone();
2142 let no_locks = HashSet::new();
2143 let end = self.refine_one(rng, &seed, &no_locks, self.cfg.refine_steps)?;
2144 if self.pool.iter().any(|c| c.tree == end) {
2145 return None;
2146 }
2147 self.record_child(parent_id, &seed, end, "refine", None)
2148 }
2149
2150 /// Locked refinement from one explicit seed candidate: evolve everything
2151 /// *except* the locked addresses. Returns the injected child id.
2152 pub fn refine_from<R: Rng>(
2153 &mut self,
2154 rng: &mut R,
2155 seed_id: u64,
2156 locked: &[String],
2157 ) -> Option<u64> {
2158 let seed = self.pool[self.find(seed_id)?].tree.clone();
2159 let locked: HashSet<String> = locked.iter().cloned().collect();
2160 self.generation += 1;
2161 let end = self.refine_one(rng, &seed, &locked, self.cfg.refine_steps)?;
2162 if self.pool.iter().any(|c| c.tree == end) {
2163 return None;
2164 }
2165 self.record_child(seed_id, &seed, end, "refine", Some(seed_id))
2166 }
2167
2168 /// Commit a hand-edited tree as a new candidate. If `original_id` is
2169 /// given, a lineage event links them; `outcome` says what the player
2170 /// reported about the pair, and only a *told* outcome writes an
2171 /// observation.
2172 pub fn commit_edit(
2173 &mut self,
2174 original_id: Option<u64>,
2175 tree: PatchTree,
2176 outcome: EditOutcome,
2177 ) -> Option<u64> {
2178 if let Some(i) = self.pool.iter().position(|c| c.tree == tree) {
2179 // The edit landed on a patch the bank already holds, so there is
2180 // no new candidate to insert. There *was* still a comparison: the
2181 // player heard two patches and picked one, and discarding that
2182 // answer because the winner happened to already exist would throw
2183 // away a real vote on the grounds of a bookkeeping collision. The
2184 // pair is scored against the twin instead.
2185 let (existing, told) = (self.pool[i].id, outcome.told());
2186 if let (Some(oid), Some((edited_won, provenance))) = (original_id, told) {
2187 if let Some(pi) = self.find(oid) {
2188 if existing != oid {
2189 self.record_duel_as(i, pi, edited_won, provenance);
2190 }
2191 }
2192 }
2193 return None;
2194 }
2195 let original = original_id.and_then(|id| self.find(id)).map(|i| {
2196 (
2197 self.pool[i].id,
2198 self.pool[i].tree.clone(),
2199 self.pool[i].phi_std.clone(),
2200 )
2201 });
2202 let child_id = self.insert_candidate(tree, Origin::Edited, original_id)?;
2203 if let Some((pid, ptree, pphi)) = original {
2204 let ci = self.find(child_id).expect("just inserted");
2205 let (ctree, cphi) = (self.pool[ci].tree.clone(), self.pool[ci].phi_std.clone());
2206 self.lineage.push(LineageEvent {
2207 generation: self.generation,
2208 kind: "edit".into(),
2209 parent_id: pid,
2210 child_id,
2211 diff: tree_diff(&ptree, &ctree),
2212 parent_utility: self.utility_of(&pphi),
2213 child_utility: self.utility_of(&cphi),
2214 });
2215 // A committed edit is a genuine one-step-ahead question — the
2216 // model has never seen this tree — so it goes through the same
2217 // forecast-then-observe path a dealt duel does, in the same
2218 // (edit, original) order, and carries the tag that lets a
2219 // self-report be scored against a heard comparison rather than
2220 // averaged into it.
2221 if let Some((edited_won, provenance)) = outcome.told() {
2222 if let (Some(ci), Some(pi)) = (self.find(child_id), self.find(pid)) {
2223 self.record_duel_as(ci, pi, edited_won, provenance);
2224 }
2225 }
2226 }
2227 Some(child_id)
2228 }
2229
2230 fn record_child(
2231 &mut self,
2232 parent_id: u64,
2233 seed: &PatchTree,
2234 mut end: PatchTree,
2235 kind: &str,
2236 protect: Option<u64>,
2237 ) -> Option<u64> {
2238 // The one place a refined child meets its seed, and therefore the one
2239 // place its node identities can be recovered.
2240 //
2241 // `refine_one` runs typed MH over the *trace*, and every accepted step
2242 // rebuilds the whole genome through `crate::genome`'s decoder — a trace
2243 // is a map from address to value and has no room for a uid, so what
2244 // comes back is structurally almost the seed and completely anonymous.
2245 // Without this line every ⚡ would look to the panel like a brand-new
2246 // patch: locks gone, hand-placed positions gone, selection gone, on the
2247 // one action the whole instrument is built around. Positions and locks
2248 // are the point of uids, and evolution is the point of auracle.
2249 end.inherit_uids(seed);
2250 let parent_phi = self
2251 .find(parent_id)
2252 .map(|i| self.pool[i].phi_std.clone())
2253 .unwrap_or_default();
2254 let child_id = self.insert_candidate(end, Origin::Refined, protect)?;
2255 let ci = self.find(child_id).expect("just inserted");
2256 let (ctree, cphi) = (self.pool[ci].tree.clone(), self.pool[ci].phi_std.clone());
2257 self.lineage.push(LineageEvent {
2258 generation: self.generation,
2259 kind: kind.into(),
2260 parent_id,
2261 child_id,
2262 diff: tree_diff(seed, &ctree),
2263 parent_utility: self.utility_of(&parent_phi),
2264 child_utility: self.utility_of(&cphi),
2265 });
2266 Some(child_id)
2267 }
2268
2269 /// Pool indices ranked by posterior-mean mixture utility (descending);
2270 /// with no posterior, arbitrary order with zero scores.
2271 pub fn ranked(&self) -> Vec<(usize, f64, f64)> {
2272 let mut rows: Vec<(usize, f64, f64)> = self
2273 .pool
2274 .iter()
2275 .enumerate()
2276 .map(|(i, c)| match &self.posterior {
2277 Some(p) if !c.phi_std.is_empty() => {
2278 let (m, s) = p.utility_mix(&c.phi_std);
2279 (i, m, s)
2280 }
2281 _ => (i, 0.0, 0.0),
2282 })
2283 .collect();
2284 rows.sort_by(|a, b| b.1.total_cmp(&a.1));
2285 rows
2286 }
2287
2288 /// Choose the next duel by **expected information gain about θ** (BALD),
2289 /// traded off against how pleasant the duel is to answer and penalized for
2290 /// repetition, then sampled from a softmax rather than argmaxed.
2291 ///
2292 /// Returns pool indices `(a, b)`; `None` if fewer than two candidates are
2293 /// standardized. See [`Engine::next_duel_full`] for the annotated form.
2294 ///
2295 /// ## Why not dueling Thompson sampling
2296 ///
2297 /// The obvious acquisition here — draw two posterior samples, duel each
2298 /// one's champion — is a real algorithm, correctly implemented, and the
2299 /// wrong objective. DTS is **best-arm identification**: it converges on
2300 /// finding the single top patch. What this system needs from a duel is
2301 /// *information about θ*, because θ is what reshapes the proposal
2302 /// distribution and paints the taste map. Those goals diverge sharply.
2303 /// The Fisher information in one Bradley–Terry duel is
2304 ///
2305 /// ```text
2306 /// I(θ) = p(1−p) · Δ Δᵀ , Δ = φ_a − φ_b , p = σ(θ·Δ)
2307 /// ```
2308 ///
2309 /// which scales with `p(1−p)` **and** with `‖Δ‖²`. DTS maximizes the
2310 /// first (champions tie at p ≈ 0.5) while actively *minimizing* the
2311 /// second: two champions of the same concentrating posterior are two
2312 /// high-utility patches, which in a 48-member pool means two *similar*
2313 /// patches. It systematically picks the least informative near-tie
2314 /// available. And once the draw set concentrates, both champions become
2315 /// the same index and the user is shown top-1 vs top-2 over and over.
2316 ///
2317 /// BALD scores the mutual information between the outcome and θ,
2318 /// `I = H(E_s[p_s]) − E_s[H(p_s)]` — high exactly when the posterior
2319 /// *disagrees with itself* about who wins, which is the definition of a
2320 /// question worth asking.
2321 ///
2322 /// Measured against DTS on the synthetic user (10 paired seeds, 72
2323 /// duels): pool-ranking correlation +0.101 ± 0.058, predictive excess
2324 /// −0.040 ± 0.017 nats. Measured against *uniformly random* pairing: no
2325 /// difference outside noise on any metric. See [`Acquisition`] for the
2326 /// full table and for why `Bald` is still the default.
2327 pub fn next_duel<R: Rng>(&mut self, rng: &mut R) -> Option<(usize, usize)> {
2328 self.next_duel_full(rng).map(|d| (d.a, d.b))
2329 }
2330
2331 /// [`Engine::next_duel`] with the reasoning attached: which rule chose the
2332 /// pair, its expected information gain in nats, and whether it is one of
2333 /// the uniformly-random check duels that calibration is scored on.
2334 pub fn next_duel_full<R: Rng>(&mut self, rng: &mut R) -> Option<DuelChoice> {
2335 // Un-standardized candidates score utility exactly 0 (`dot` over an
2336 // empty vector), which beats every real utility once a user has killed
2337 // enough patches — they must not be selectable, the same guard
2338 // `ranked()` applies.
2339 let cands: Vec<usize> = (0..self.pool.len())
2340 .filter(|&i| !self.pool[i].phi_std.is_empty())
2341 .collect();
2342 if cands.len() < 2 {
2343 return None;
2344 }
2345 let uniform = |rng: &mut R| -> (usize, usize) {
2346 let i = rng.gen_range(0..cands.len());
2347 let mut j = rng.gen_range(0..cands.len() - 1);
2348 if j >= i {
2349 j += 1;
2350 }
2351 (cands[i], cands[j])
2352 };
2353
2354 let check = self.cfg.duel_check_every > 0
2355 && self.duels_shown > 0
2356 && self.duels_shown.is_multiple_of(self.cfg.duel_check_every);
2357
2358 let choice = match (&self.posterior, check) {
2359 // No taste yet, or a scheduled check duel: uniform at random.
2360 // A uniform pair *is* a calibration check, so it is tagged as one
2361 // whether it was scheduled or is simply how this engine picks
2362 // every duel. Under the default rule that makes the unbiased
2363 // subsample the entire sample, which is the whole reason to
2364 // prefer it: the reliability diagram needs no asterisk.
2365 (None, _) => {
2366 let (a, b) = uniform(rng);
2367 DuelChoice {
2368 a,
2369 b,
2370 info_gain: 0.0,
2371 random_check: true,
2372 method: "random",
2373 }
2374 }
2375 (Some(_), true) => {
2376 let (a, b) = uniform(rng);
2377 DuelChoice {
2378 a,
2379 b,
2380 info_gain: 0.0,
2381 random_check: true,
2382 method: "check",
2383 }
2384 }
2385 (Some(_), false) if self.cfg.acquisition == Acquisition::Random => {
2386 let (a, b) = uniform(rng);
2387 DuelChoice {
2388 a,
2389 b,
2390 info_gain: 0.0,
2391 random_check: true,
2392 method: "random",
2393 }
2394 }
2395 (Some(posterior), false) if self.cfg.acquisition == Acquisition::Thompson => {
2396 let (a, b) = thompson_pair(posterior, &self.pool, &cands, rng);
2397 DuelChoice {
2398 a,
2399 b,
2400 info_gain: 0.0,
2401 random_check: false,
2402 method: "thompson",
2403 }
2404 }
2405 (Some(posterior), false) => {
2406 let (a, b, info) = self.bald_pair(posterior, &cands, rng);
2407 DuelChoice {
2408 a,
2409 b,
2410 info_gain: info,
2411 random_check: false,
2412 method: "bald",
2413 }
2414 }
2415 };
2416
2417 self.duels_shown += 1;
2418 let key = pair_key(self.pool[choice.a].id, self.pool[choice.b].id);
2419 *self.shown_pairs.entry(key).or_insert(0) += 1;
2420 *self.shown_candidates.entry(key.0).or_insert(0) += 1;
2421 *self.shown_candidates.entry(key.1).or_insert(0) += 1;
2422 self.last_check_pair = choice.random_check.then_some(key);
2423 Some(choice)
2424 }
2425
2426 /// The BALD scan itself. Utilities are precomputed once per candidate per
2427 /// draw (`S × |pool|`), then every pair is scored from that table — the
2428 /// whole sweep is a few hundred thousand sigmoids, milliseconds in wasm.
2429 fn bald_pair<R: Rng>(
2430 &self,
2431 posterior: &TastePosterior,
2432 cands: &[usize],
2433 rng: &mut R,
2434 ) -> (usize, usize, f64) {
2435 let s_n = posterior.samples.len();
2436 if s_n == 0 {
2437 let i = rng.gen_range(0..cands.len());
2438 let mut j = rng.gen_range(0..cands.len() - 1);
2439 if j >= i {
2440 j += 1;
2441 }
2442 return (cands[i], cands[j], 0.0);
2443 }
2444 // u[s][c] over the *standardized* pool.
2445 let u: Vec<Vec<f64>> = posterior
2446 .samples
2447 .iter()
2448 .map(|s| {
2449 cands
2450 .iter()
2451 .map(|&i| s.utility_mix(&self.pool[i].phi_std))
2452 .collect()
2453 })
2454 .collect();
2455 let w: Vec<f64> = (0..s_n).map(|s| posterior.weight(s)).collect();
2456 let mean_u: Vec<f64> = (0..cands.len())
2457 .map(|c| (0..s_n).map(|s| w[s] * u[s][c]).sum())
2458 .collect();
2459 // The enjoyment term is scored on **pool-standardized** utility, not
2460 // raw utility. Raw utility has no fixed scale: it grows without bound
2461 // as the posterior sharpens, so a fixed λ against it starts as a
2462 // gentle nudge and ends up swamping the information term entirely —
2463 // at which point the acquisition function has silently turned back
2464 // into the best-arm rule this one replaced. Standardized, λ means the
2465 // same thing at duel 10 and duel 200.
2466 let u_mu = mean_u.iter().sum::<f64>() / mean_u.len().max(1) as f64;
2467 let u_sd = (mean_u.iter().map(|u| (u - u_mu) * (u - u_mu)).sum::<f64>()
2468 / mean_u.len().max(1) as f64)
2469 .sqrt()
2470 .max(1e-9);
2471 let z_u: Vec<f64> = mean_u.iter().map(|u| (u - u_mu) / u_sd).collect();
2472
2473 let lambda = self.cfg.duel_utility_weight;
2474 let gamma = self.cfg.duel_repeat_penalty;
2475 let rho = self.cfg.duel_exposure_penalty;
2476 // How often each candidate has been *put in front of the user*, by any
2477 // pairing. See `duel_exposure_penalty`: without this the top-utility
2478 // candidate is nominated over and over through pairs that are all
2479 // technically distinct.
2480 let seen: Vec<f64> = cands
2481 .iter()
2482 .map(|&i| {
2483 self.shown_candidates
2484 .get(&self.pool[i].id)
2485 .copied()
2486 .unwrap_or(0) as f64
2487 })
2488 .collect();
2489 let mut best = Vec::with_capacity(cands.len() * cands.len() / 2);
2490 for ci in 0..cands.len() {
2491 for cj in (ci + 1)..cands.len() {
2492 let mut p_bar = 0.0;
2493 let mut mean_h = 0.0;
2494 for s in 0..s_n {
2495 let p = sigmoid(u[s][ci] - u[s][cj]);
2496 p_bar += w[s] * p;
2497 mean_h += w[s] * binary_entropy(p);
2498 }
2499 let info = binary_entropy(p_bar) - mean_h;
2500 let shown = self
2501 .shown_pairs
2502 .get(&pair_key(self.pool[cands[ci]].id, self.pool[cands[cj]].id))
2503 .copied()
2504 .unwrap_or(0) as f64;
2505 let j = info + lambda * (z_u[ci] + z_u[cj]) / 2.0
2506 - gamma * shown
2507 - rho * (seen[ci] + seen[cj]);
2508 best.push((ci, cj, j, info));
2509 }
2510 }
2511 // Softmax over the objective, at a temperature set by the objective's
2512 // *own* spread. An absolute temperature is a bet on how far apart the
2513 // scores happen to be, and at 0.05 nats against a spread of several
2514 // tenths this "softmax" was an argmax — which is exactly the best-arm
2515 // lock-in BALD exists to avoid.
2516 let j_mu = best.iter().map(|x| x.2).sum::<f64>() / best.len().max(1) as f64;
2517 let j_sd = (best
2518 .iter()
2519 .map(|x| (x.2 - j_mu) * (x.2 - j_mu))
2520 .sum::<f64>()
2521 / best.len().max(1) as f64)
2522 .sqrt();
2523 let t = (self.cfg.duel_temperature * j_sd).max(1e-9);
2524 let max_j = best.iter().map(|x| x.2).fold(f64::NEG_INFINITY, f64::max);
2525 let total: f64 = best.iter().map(|x| ((x.2 - max_j) / t).exp()).sum();
2526 let mut r = rng.gen::<f64>() * total;
2527 for &(ci, cj, j, info) in &best {
2528 r -= ((j - max_j) / t).exp();
2529 if r <= 0.0 {
2530 return (cands[ci], cands[cj], info);
2531 }
2532 }
2533 let &(ci, cj, _, info) = best.last().expect("at least one pair");
2534 (cands[ci], cands[cj], info)
2535 }
2536
2537 /// Append one feedback event and fold it into the current posterior.
2538 ///
2539 /// `raw` is what the log keeps — un-standardized values plus the names
2540 /// they belong to, so the log stays interpretable across feature-set
2541 /// changes. `standardized` is the same event on the current scale, used
2542 /// only to reweight the existing posterior draws by sequential importance
2543 /// sampling: an O(S) update that makes the *next* duel respond to this
2544 /// one instead of waiting for the next multi-second MCMC refit.
2545 fn observe(&mut self, raw: Feedback, standardized: Feedback) {
2546 self.observe_as(raw, standardized, Provenance::Duel);
2547 }
2548
2549 /// The same, carrying how the answer was collected. The tag reaches the
2550 /// log and nothing else: `standardized` goes into the posterior update
2551 /// untouched, so two observations that differ only in provenance move the
2552 /// posterior identically. Provenance is evidence *about the evidence*, and
2553 /// weighting by it would be a modeling claim with no measurement behind it
2554 /// — see [`Provenance`].
2555 fn observe_as(&mut self, raw: Feedback, standardized: Feedback, provenance: Provenance) {
2556 self.log.push(Observation::tagged(
2557 raw,
2558 self.session,
2559 &phi_names(),
2560 provenance,
2561 ));
2562 if self.cfg.sis_between_fits {
2563 if let Some(p) = &self.posterior {
2564 let mut updated = p.reweighted(&standardized, self.session);
2565 // Degenerate weights make the acquisition function read a
2566 // one-point "posterior" as certainty. Resample back to a
2567 // uniform set rather than let that happen; the impoverishment
2568 // is bounded by how soon the next full refit lands.
2569 if updated.ess() < updated.samples.len() as f64 / 2.0 {
2570 updated = updated.resampled();
2571 self.resamples_since_fit += 1;
2572 }
2573 self.posterior = Some(Arc::new(updated));
2574 }
2575 }
2576 }
2577
2578 /// Record a duel outcome between two pool members (by pool index).
2579 ///
2580 /// The out-of-sample forecast is scored *here*, before the observation is
2581 /// appended — the model has to commit before it is told the answer, which
2582 /// is what makes [`Engine::calibration`] prequential rather than a
2583 /// in-sample self-assessment.
2584 pub fn record_duel(&mut self, a: usize, b: usize, chose_a: bool) {
2585 self.record_duel_as(a, b, chose_a, Provenance::Duel);
2586 }
2587
2588 /// The same, for a pair the app assembled itself rather than dealt — a
2589 /// hand edit against the patch it was edited from. One code path, so the
2590 /// editor's answers are scored, logged and folded into the posterior by
2591 /// exactly the machinery a dealt duel is, and differ only in the tag that
2592 /// says where they came from.
2593 fn record_duel_as(&mut self, a: usize, b: usize, chose_a: bool, provenance: Provenance) {
2594 if let Some(p_a) = self.predict_duel(a, b) {
2595 let key = pair_key(self.pool[a].id, self.pool[b].id);
2596 self.forecasts.push(Forecast {
2597 p_a,
2598 chose_a,
2599 random_check: self.last_check_pair == Some(key),
2600 provenance,
2601 });
2602 }
2603 let raw = Feedback::Duel {
2604 a: self.pool[a].features.phi(),
2605 b: self.pool[b].features.phi(),
2606 chose_a,
2607 };
2608 let std = Feedback::Duel {
2609 a: self.pool[a].phi_std.clone(),
2610 b: self.pool[b].phi_std.clone(),
2611 chose_a,
2612 };
2613 self.observe_as(raw, std, provenance);
2614 }
2615
2616 /// Record a keep/kill decision on a pool member (by pool index).
2617 pub fn record_keep(&mut self, idx: usize, kept: bool) {
2618 let raw = Feedback::KeepKill {
2619 x: self.pool[idx].features.phi(),
2620 kept,
2621 };
2622 let std = Feedback::KeepKill {
2623 x: self.pool[idx].phi_std.clone(),
2624 kept,
2625 };
2626 self.observe(raw, std);
2627 }
2628
2629 /// Record a star rating on a pool member (by pool index).
2630 pub fn record_stars(&mut self, idx: usize, rating: u8) {
2631 let raw = Feedback::Stars {
2632 x: self.pool[idx].features.phi(),
2633 rating,
2634 };
2635 let std = Feedback::Stars {
2636 x: self.pool[idx].phi_std.clone(),
2637 rating,
2638 };
2639 self.observe(raw, std);
2640 }
2641
2642 /// Prequential calibration over every duel forecast so far.
2643 pub fn calibration(&self) -> Calibration {
2644 calibration(&self.forecasts)
2645 }
2646
2647 /// Exact per-feature decomposition of a candidate's utility under the lens
2648 /// that claims it (B9 — see [`Explanation`]).
2649 pub fn explain(&self, id: u64) -> Option<Explanation> {
2650 let i = self.find(id)?;
2651 self.explain_std(id, &self.pool[i].phi_std.clone())
2652 }
2653
2654 /// The same decomposition for a φ that is **not** a pool member — the
2655 /// workbench, which is a patch under the player's hands and not a
2656 /// candidate until they commit it.
2657 ///
2658 /// This is what makes the readout above the rack honest. The WHY line used
2659 /// to be fetched once, for the candidate that was loaded, and then went on
2660 /// describing it through any number of edits: it named features of a patch
2661 /// the player had already edited away. The bench re-featurizes on every
2662 /// edit anyway, so the true decomposition is a dot product away — there
2663 /// was never a cost reason for the stale one.
2664 ///
2665 /// Takes **raw** φ and standardizes here, because raw is what the
2666 /// featurizer produces and what the log stores; θ is denominated in the
2667 /// standardizer, so the transform is not optional.
2668 pub fn explain_phi(&self, phi_raw: &[f64]) -> Option<Explanation> {
2669 let sz = self.standardizer.as_ref()?;
2670 self.explain_std(0, &sz.transform(phi_raw))
2671 }
2672
2673 fn explain_std(&self, id: u64, phi: &[f64]) -> Option<Explanation> {
2674 let p = self.posterior.as_ref()?;
2675 if phi.is_empty() {
2676 return None;
2677 }
2678 let responsibilities = p.responsibilities(phi);
2679 let style = responsibilities
2680 .iter()
2681 .enumerate()
2682 .max_by(|(_, x), (_, y)| x.total_cmp(y))
2683 .map(|(k, _)| k)
2684 .unwrap_or(0);
2685 let theta = p.theta_mean(style);
2686 let names = phi_names();
2687 let mut contributions: Vec<Contribution> = names
2688 .iter()
2689 .zip(&theta)
2690 .zip(phi)
2691 .map(|((name, t), x)| Contribution {
2692 name: name.clone(),
2693 theta: *t,
2694 phi_std: *x,
2695 contribution: t * x,
2696 })
2697 .collect();
2698 contributions.sort_by(|a, b| b.contribution.abs().total_cmp(&a.contribution.abs()));
2699 // `utility`/`utility_std` describe the lens quantity the contributions
2700 // sum to; `mix_utility` is what the bank is sorted by. Both are
2701 // returned because they are genuinely different claims and the caller
2702 // needs to know which one it is drawing.
2703 let (utility, utility_std) = p.utility(phi, style);
2704 Some(Explanation {
2705 id,
2706 style,
2707 style_name: self.style_names.get(style).cloned().unwrap_or_default(),
2708 utility,
2709 utility_std,
2710 mix_utility: p.utility_mix(phi).0,
2711 responsibility: responsibilities.get(style).copied().unwrap_or(0.0),
2712 contributions,
2713 })
2714 }
2715
2716 /// Musical display names for the whole pool, unique across it. Keyed by
2717 /// candidate id; a user-given name always wins.
2718 ///
2719 /// User and preset names are claimed **first and through the same
2720 /// registry** as generated ones. Substituting them afterwards, as this
2721 /// once did, let a preset called `Glass Pad` and a generated `Glass Pad`
2722 /// both survive into the bank: the preset occupied the name without ever
2723 /// competing for it.
2724 pub fn display_names(&self) -> HashMap<u64, String> {
2725 let scale = NameScale::fit(self.pool.iter().map(|c| &c.features));
2726 let mut taken: HashSet<String> = HashSet::new();
2727 let mut out: HashMap<u64, String> = HashMap::new();
2728
2729 // Explicit names first — they are not negotiable, so they get to
2730 // reserve their spelling before anything is generated.
2731 for c in &self.pool {
2732 if let Some(name) = &c.name {
2733 out.insert(c.id, claim_name(name, &mut taken));
2734 }
2735 }
2736 // Then generated ones, in id order: a patch's numeral must not
2737 // reshuffle when the pool is re-ranked underneath it.
2738 let mut rest: Vec<&Candidate> = self.pool.iter().filter(|c| c.name.is_none()).collect();
2739 rest.sort_by_key(|c| c.id);
2740 for c in rest {
2741 out.insert(c.id, claim_name(&scale.name(&c.features), &mut taken));
2742 }
2743 out
2744 }
2745
2746 /// Name (or rename; empty clears) a candidate.
2747 pub fn set_name(&mut self, id: u64, name: &str) {
2748 if let Some(i) = self.find(id) {
2749 let trimmed = name.trim();
2750 self.pool[i].name = (!trimmed.is_empty()).then(|| trimmed.chars().take(40).collect());
2751 }
2752 }
2753
2754 /// How many patches may be pinned at once: a quarter of the pool.
2755 ///
2756 /// The pool is the model's *working set*, not storage — duel pairing is
2757 /// uniform over it and refinement seeds from the top of `ranked()` — so
2758 /// pins are spent capacity, and the only wholly wasted duel is one where
2759 /// both sides are pinned. At a quarter of the pool that is ~6% of pairs,
2760 /// with three quarters of the pool still free to churn; at half it is 25%.
2761 /// A quarter buys the user far more than they lose.
2762 ///
2763 /// The cap also keeps "everything is pinned" unreachable, which matters
2764 /// because that state has no honest report: it surfaces as
2765 /// [`Engine::insert_candidate`] returning `None`, which every caller
2766 /// already renders as "no proposal beat its parent" — a statement about
2767 /// the search that would then be a lie about storage.
2768 pub fn pin_cap(&self) -> usize {
2769 (self.cfg.pool_size / 4).max(1)
2770 }
2771
2772 /// How many pool members are currently pinned.
2773 pub fn pinned_count(&self) -> usize {
2774 self.pool.iter().filter(|c| c.pinned).count()
2775 }
2776
2777 /// Pin or unpin a patch against eviction. Returns `false` when the id is
2778 /// unknown, or when pinning would exceed [`Engine::pin_cap`] — callers are
2779 /// expected to say which, rather than letting the control fail silently.
2780 ///
2781 /// Records **no observation**: a pin says what the user wants to keep, not
2782 /// what they think of it. See [`Candidate::pinned`].
2783 pub fn set_pinned(&mut self, id: u64, pinned: bool) -> bool {
2784 let Some(i) = self.find(id) else {
2785 return false;
2786 };
2787 if pinned && !self.pool[i].pinned && self.pinned_count() >= self.pin_cap() {
2788 return false;
2789 }
2790 self.pool[i].pinned = pinned;
2791 true
2792 }
2793
2794 /// Insert a named preset into the pool (protected from immediate
2795 /// eviction pressure only by its utility, like any candidate). Returns
2796 /// the new id.
2797 pub fn insert_preset(&mut self, tree: PatchTree, name: &str) -> Option<u64> {
2798 if let Some(existing) = self.pool.iter().find(|c| c.tree == tree) {
2799 return Some(existing.id);
2800 }
2801 let id = self.insert_candidate(tree, Origin::Preset, None)?;
2802 self.set_name(id, name);
2803 Some(id)
2804 }
2805
2806 /// Export the portable profile (log + standardizer, which only mean
2807 /// anything together).
2808 pub fn export_profile(&self) -> Profile {
2809 Profile {
2810 log: self.log.clone(),
2811 standardizer: self.standardizer.as_deref().cloned(),
2812 }
2813 }
2814
2815 /// Export the full session (profile + bank + lineage) for persistence.
2816 /// Renders and features are intentionally omitted — trees re-featurize
2817 /// deterministically on import.
2818 pub fn export_state(&self) -> SessionState {
2819 SessionState {
2820 profile: self.export_profile(),
2821 bank: self
2822 .pool
2823 .iter()
2824 .map(|c| BankEntry {
2825 id: c.id,
2826 tree: c.tree.clone(),
2827 origin: c.origin,
2828 name: c.name.clone(),
2829 pinned: c.pinned,
2830 })
2831 .collect(),
2832 lineage: self.lineage.clone(),
2833 generation: self.generation,
2834 style_names: self.style_names.clone(),
2835 events: self.events.clone(),
2836 forecasts: self.forecasts.clone(),
2837 style_shares: self.style_shares.clone(),
2838 }
2839 }
2840
2841 /// Restore a saved session, replacing pool, log, standardizer, lineage,
2842 /// and id allocation. Each bank tree is re-featurized (and re-rendered
2843 /// when `keep_renders`); entries that no longer vet are dropped. Returns
2844 /// how many bank entries were restored.
2845 pub fn import_state(&mut self, state: SessionState) -> usize {
2846 let want_audio = self.wants_admitted_audio();
2847 let bank = self.import_state_deferred(state);
2848 for entry in bank {
2849 let Ok((cached, audition)) =
2850 featurize_memo(&entry.tree, &self.cfg.phrase, &self.memo, want_audio)
2851 else {
2852 continue;
2853 };
2854 let pre = PreFeaturized {
2855 tree: entry.tree.clone(),
2856 cached,
2857 audition,
2858 };
2859 self.absorb_bank_entry(entry, pre);
2860 }
2861 self.finish_restore()
2862 }
2863
2864 /// Restore a saved session **without rendering the bank**: everything
2865 /// [`Engine::import_state`] does except the per-entry featurize, returning
2866 /// the bank entries for off-engine work, in bank order.
2867 ///
2868 /// Restore is the returning user's boot and today it is *worse* than a
2869 /// cold one — a full bank of serial re-renders behind a bar that cannot
2870 /// move, because nothing lands until all of it finishes. This is the seam
2871 /// that lets the farm do it: each entry comes back through
2872 /// [`Engine::absorb_bank_entry`] and [`Engine::finish_restore`] closes the
2873 /// restore, and the three together are exactly `import_state`.
2874 ///
2875 /// Profile-then-clear ordering is preserved from `import_state`:
2876 /// [`Engine::import_profile`] may re-fit a standardizer over the *current*
2877 /// pool, so clearing before it would change the scale a restore lands on.
2878 pub fn import_state_deferred(&mut self, state: SessionState) -> Vec<BankEntry> {
2879 self.import_profile(state.profile);
2880 self.lineage = state.lineage;
2881 self.generation = state.generation;
2882 self.style_names = state.style_names;
2883 self.events = state.events;
2884 self.forecasts = state.forecasts;
2885 self.style_shares = state.style_shares;
2886 // The implicit stream stores raw φ on both sides of a hand edit, so it
2887 // is the fourth carrier of the corruption after the pool, the log and
2888 // the HELD tray — and the only one nothing reads yet, which is exactly
2889 // why it would have been the one still poisoned on the day it was
2890 // first fitted on.
2891 let names = phi_names();
2892 for e in &mut self.events {
2893 self.repaired_cells +=
2894 crate::migrate::repair_phi_pair(&mut e.phi_before, &mut e.phi_after, &names);
2895 }
2896 self.pool.clear();
2897 self.audio_lru.clear();
2898 // Every saved term, repaired on the way in. This is the *only* place a
2899 // tree written by an older build enters the engine, and a bank entry
2900 // carrying a knob outside its range would otherwise be quarantined by
2901 // the featurizer a few lines later and silently disappear from the
2902 // player's bank — losing four patches to fix a bug in one number.
2903 // Repair keeps the patch and loses only the corruption, which is the
2904 // standing rule for saved state: migration, never deletion.
2905 let mut bank = state.bank;
2906 for entry in &mut bank {
2907 if entry.tree.clamp_domains() > 0 {
2908 self.repaired_terms += 1;
2909 }
2910 }
2911 bank
2912 }
2913
2914 /// How many saved terms, log cells and whole observations the last
2915 /// [`Engine::import_state_deferred`] had to repair. All three are zero for
2916 /// a session written by a build that has this gate.
2917 ///
2918 /// Reported rather than logged because the frontend is the only thing that
2919 /// can tell the player their profile was mended, and a silent repair of the
2920 /// evidence a model is fitted on is exactly the kind of quiet the rest of
2921 /// this app was built to stop.
2922 pub fn repair_report(&self) -> (usize, usize, usize) {
2923 (
2924 self.repaired_terms,
2925 self.repaired_cells,
2926 self.dropped_observations,
2927 )
2928 }
2929
2930 /// Reinstate one restored bank entry with its saved identity, from a
2931 /// featurization performed off-engine.
2932 ///
2933 /// Bypasses the pool-size and novelty checks, as `import_state`'s push
2934 /// does: a bank is a bank, not a candidate competition. `entry` supplies
2935 /// the identity (id, origin, name) and the term; `pre` supplies φ.
2936 pub fn absorb_bank_entry(&mut self, entry: BankEntry, pre: PreFeaturized) {
2937 let PreFeaturized {
2938 tree: _,
2939 cached,
2940 audition,
2941 } = pre;
2942 self.memo.put(cached.clone(), audition.clone());
2943 let phi_std = self
2944 .standardizer
2945 .as_ref()
2946 .map(|sz| sz.transform(&cached.features.phi()))
2947 .unwrap_or_default();
2948 let render = self.admitted_render(&entry.tree, &cached.features, audition);
2949 self.next_id = self.next_id.max(entry.id + 1);
2950 self.pool.push(Candidate {
2951 id: entry.id,
2952 tree: settled(entry.tree),
2953 features: cached.features,
2954 phi_std,
2955 key: cached.key,
2956 render,
2957 origin: entry.origin,
2958 name: entry.name,
2959 pinned: entry.pinned,
2960 });
2961 }
2962
2963 /// Close a deferred restore once every entry that was going to land has.
2964 /// Returns the restored bank size.
2965 ///
2966 /// The standardizer normally comes from the profile; a session saved
2967 /// before the first fit completes has none — fit one from the restored
2968 /// bank so φ isn't left raw. Idempotent, and safe on an empty pool.
2969 pub fn finish_restore(&mut self) -> usize {
2970 if self.standardizer.is_none() && !self.pool.is_empty() {
2971 let rows: Vec<Vec<f64>> = self.pool.iter().map(|c| c.features.phi()).collect();
2972 let sz = Arc::new(Standardizer::fit(&rows));
2973 for c in &mut self.pool {
2974 c.phi_std = sz.transform(&c.features.phi());
2975 }
2976 self.standardizer = Some(sz);
2977 }
2978 self.pool.len()
2979 }
2980
2981 /// Import a profile: replaces the log and re-establishes a standardizer
2982 /// for it.
2983 ///
2984 /// A profile written before raw-φ logging carries *standardized* vectors,
2985 /// which are only interpretable through the standardizer that shipped with
2986 /// them — so that pairing is exactly what makes the migration possible
2987 /// ([`crate::migrate`]): invert the transform, convert the coordinates
2988 /// whose units changed, and the log becomes raw evidence again. Its
2989 /// standardizer is then obsolete by construction (it has the wrong
2990 /// dimension for the current feature set) and a fresh one is fit from the
2991 /// migrated data. A same-schema profile keeps its standardizer, so
2992 /// imported θ geometry stays valid until the next fit refreshes it.
2993 pub fn import_profile(&mut self, profile: Profile) {
2994 self.log = profile.log;
2995 // A fresh restore reports on *itself*. `import_state_deferred` runs
2996 // this first and then counts the bank, so clearing all three here is
2997 // also what keeps a standalone "load taste profile" from inheriting the
2998 // patch count of whatever was loaded before it.
2999 self.repaired_terms = 0;
3000 self.repaired_cells = 0;
3001 self.dropped_observations = 0;
3002 let names = phi_names();
3003 if let Some(sz) = &profile.standardizer {
3004 if crate::migrate::needs_migration(&self.log) {
3005 // Schema-1 values were measured under the v1 stimulus, so
3006 // they land on the v1 names — FitSet::build keeps their
3007 // structural coordinates and imputes today's stimulus-tagged
3008 // audio coordinates at "no evidence" (migrate::v1_names).
3009 crate::migrate::migrate_log(
3010 &mut self.log,
3011 sz,
3012 &crate::migrate::v1_names(),
3013 self.cfg.phrase.sample_rate / 2.0,
3014 );
3015 }
3016 }
3017 // Before stamping: a coordinate that was renamed since this profile
3018 // was written still holds the right *value*, and `FitSet::build`
3019 // matches by name — so the rename has to be applied to the stored
3020 // names or the evidence is imputed away as "no opinion".
3021 crate::migrate::apply_renames(&mut self.log);
3022 crate::migrate::stamp_names(&mut self.log, &names);
3023 // After the names are stamped, because the repair is by name — and
3024 // before the standardizer is adopted, because a standardizer fitted
3025 // over a poisoned column is itself poisoned. When anything was
3026 // repaired the saved one is *discarded* and re-fitted from the
3027 // repaired rows plus the pool: keeping it would mean the load re-read
3028 // its own corruption back out of the scale it set.
3029 let (clamped, dropped) = crate::migrate::repair_log(&mut self.log);
3030 self.repaired_cells = clamped;
3031 self.dropped_observations = dropped;
3032 let poisoned = clamped > 0 || dropped > 0;
3033 match profile.standardizer {
3034 Some(sz) if sz.dimension() == names.len() && !poisoned => {
3035 let sz = Arc::new(sz);
3036 for c in &mut self.pool {
3037 c.phi_std = sz.transform(&c.features.phi());
3038 }
3039 self.standardizer = Some(sz);
3040 }
3041 _ => {
3042 self.standardizer = None;
3043 self.refit_standardizer();
3044 }
3045 }
3046 self.session = self.log.n_sessions();
3047 self.posterior = None;
3048 }
3049}