Inference before refinement, and the family's configuration space assessed, not decided

Two ideas of Josu's of 15 October 2026, assessed for him to decide (L286). The first: before any refinement, from the data, the parent structure and the candidate low symmetries alone, search the configuration space and return the regions where the parameters most probably are. The second: carry that search over the configuration space of a whole family of materials — the double perovskites first — along the chemical axes (the B cation, the B′ cation, substitutions, chemical pressure), then to other families. Both come from the inverse-problems project with Iñigo González de Arrieta (its discourse and programme, P1–P7).

The verdict first, then each idea taken apart: what it is when said precisely, what it would give, where it is weak, what already exists in the literature and in this core, and what it would cost. Then the shape it would take in this project and what Josu decides. Claims about the field were checked by web search on 15 October 2026 and say so; what is believed and not opened is marked not checked.

0. The verdict

• Both ideas make sense, and they are one idea. The first is the formal version of what a refiner does by hand when choosing the subgroup and the starting amplitudes; it gives that choice a number and the result a region. But its answers depend on a prior over the amplitudes and the cell, and a vague prior makes the comparison of subgroups unreliable (§1.4). The second idea is where a measured prior comes from: the family's known structures, which for the double perovskites are already typed records in the Perovskite Atlas (47 materials, 41 structures, 13 series, every value with its source page). The family map is the prior of the single refinement, and every refinement updates the family map. Neither works well alone.
• What would be new. Bayesian Rietveld exists for one structure at a time (Fancher et al. 2016; Gagin and Levin 2015), probabilities over extinction symbols exist (Markvardsen et al. 2001), the symmetry-mode search with modes switched on and off exists (Kerman et al. 2012, with ISODISTORT and TOPAS), and black-box tuning of a refinement's settings exists (Ozaki et al. 2020). None of them, checked by search, compares the subgroups of a parent by their evidence in mode coordinates, folds the domain-equivalent solutions, marginalises the nuisance parameters, learns the prior from a family, and says which measurement would decide: and none lives inside a refinement program. That combination is the module.
• The word "machine learning" is wrong for the family and should not be used. A family is tens of compounds, not a hundred thousand patterns. At that size the honest tool is a hierarchical Bayesian model, which at \(n = 40\) works and states its uncertainty; a network would overfit and would not say so. The corpus of the inverse-problems project (P3, tens of thousands of simulated refinements) is a different thing and comes later, if at all.
• The risk to name first. The credible regions mean something only if the likelihood is honest. Rietveld's \(\chi^2\) is not a likelihood: the residuals are correlated and structured, and a posterior built on \(\exp(-\chi^2/2)\) is as overconfident as the printed e.s.d.'s. The design must declare how the misfit is modelled (§1.3), and the toy must test calibration: that a 90 % region contains the truth 90 % of the time. If that test fails, the module is a search tool, not an inference tool, and should say so.
• Recommended order. The toy first (two weeks, the inverse project's P2, on a synthetic double perovskite with this core); the module only if the toy's regions are calibrated. Then a crate of its own, drs-infer, with the Atlas as the family's source, and an app module that shows what the crate computes. Nothing before Josu decides §4.

1. The first idea: inference before refinement

1.1 What it is, said precisely

The information at hand is the pattern \(d\), the parent \(G\) with its structure, and a list of candidate subgroups \(H_1 \dots H_K\) (for the rock-salt-ordered double perovskites the list is known and finite: the twelve space groups of Howard, Kennedy and Woodward 2003, with their group–subgroup tree; checked by search). Each \(H_k\) is a model: the parent plus the symmetry modes \(H_k\) allows, with amplitudes \(Q\), a cell, a profile, a background and a scale. "Where are the parameters most probably" is three questions, nested:

1. Which subgroup. \(P(H_k \mid d) \propto P(d \mid H_k)\,P(H_k)\), where the evidence \(P(d \mid H_k) = \int L(\theta)\,\pi(\theta \mid H_k)\,d\theta\) averages the likelihood over the prior. The evidence penalises the modes a subgroup adds unless the data want them. That is the question the refiner answers today by comparing \(\chi^2\) and R-factors across models with different numbers of parameters, which is not a valid comparison and which everyone makes anyway.
2. Where, within the subgroup. The posterior over the amplitudes and the cell, with the profile, background and scale marginalised. This is the inverse project's tube: the region, not the point. It is multimodal for a trivial reason, the domain-equivalent amplitude sets (the core enumerates them: case A has 4, Modes::equivalent_amplitudes, L218), and for a real one when two distinct distortions fit (the ridge between two minima of Explore's map F10).
3. Which amplitudes this data can determine at all. The identifiability of each mode as a function of the radiation, the range and the noise: a property of the forward map and the instrument, not of the sample, computable before the measurement for a hypothetical structure (the inverse project's P1 and P4).

The search is not instead of the refinement; it is before it. The posterior's modes become the control file the refinement starts from, and Explore shows the path from there. The two modules share one object: Explore's \(\chi^2\) map is the conditional cut of the posterior with the other parameters held; the marginal, which Explore prices at \(n^2\) refinements and offers only as an option (its §2.4), is what a sampler returns for every pair at once.

1.2 What it would give that no program gives

• The subgroup choice with a number. Evidence ratios between I4/m, I2/m and P2₁/n for Sr₂ScSbO₆ at 450 K, instead of "the R-factors are similar and P2₁/n has three more parameters". Case C of Josu's material is exactly that situation.
• A start that is not a guess. A least-squares refinement by modes cannot start at the parent: at zero amplitude the superstructure reflections have zero derivative with respect to the X-point amplitudes and the matrix is singular (Explore's §6.2). It is started "near" the parent, at a value the user picks, and from +0.1 Å it reached the mirror solution (§6.3). A sampler does not care about the saddle, and its modes, folded by the domain equivalence, are the right starts.
• The region as the result (the inverse project's Q5): an amplitude is reported as a value only when its marginal is narrow; otherwise as its region, with the directions the data do not fix named in the modes' names.
• Which measurement decides. Given the posterior from the X-ray pattern, the expected information gain of a neutron pattern, of a longer range, of a lower temperature, on the subgroup question and on each amplitude: computed from the Jacobian over the posterior, before the beam time is asked for. No refinement program does this; it is P6's idea carried to diffraction.
• The k-point test first, which is cheap and decisive. Before any sampler, the pattern's own superstructure reflections say which k-points of the parent carry intensity, hence which irreps can be primary; that prunes the subgroup tree with the core's reflection lists and absences alone (the generalisation of Markvardsen et al.'s probabilities over extinction symbols, 2001, checked by search). A sampler that does not need to visit a branch the data have already excluded is a cheaper sampler.

1.3 Where it is weak

1. The likelihood. \(\exp(-\chi^2/2)\) with Poisson weights assumes independent residuals. A Rietveld residual is correlated over the width of a peak and structured by profile and background misfit; the printed e.s.d.'s are known to be too small for that reason (Bérar's factor is the standard patch, and the core computes it). A posterior built on the raw \(\chi^2\) inherits the error exactly. Fancher et al. 2016 (Sci. Rep. 6, 31625; checked by search) found their MCMC uncertainties larger than Rietveld's on the NIST silicon standard; Gagin and Levin 2015 add an explicit correlated systematic-error term to the model (not checked: J. Appl. Cryst. 48, 1201, from memory). The design must choose, and say, one of: the likelihood tempered by \(\chi^2_\nu\) (Bérar's factor as a temperature; crude, cheap, and what the e.s.d.'s already do), or a discrepancy term for the residual's structure after Kennedy and O'Hagan 2001 (right, and a model of its own with parameters to set). The toy decides between them by calibration, not by taste.
2. The prior volume decides the evidence. The evidence's penalty on an added mode is the ratio of the posterior's width to the prior's width in that amplitude: a prior ten times wider costs the mode a factor ten (Lindley's paradox; standard, not checked against a particular source). So evidence ratios between subgroups are only as meaningful as the amplitude prior, and "a few ångström, uniform" is not a prior anyone would defend. The measured prior of §2 is the answer; without it, the module reports the evidences with their sensitivity to the prior's width, and no verdict.
3. Cost. One pattern costs 5 ms on case A (3 199 points, 17 parameters) and 40 ms on Y₂O₃ (13 091 points, 79 parameters) (Explore's measured costs). A sampler's \(10^5\) evaluations are 8 min and 67 min per subgroup on one thread; the core's pool (half the cores) divides that by 4 to 8; twelve subgroups multiply it. Feasible on a Mac, not interactive. Three things cut it to minutes: the scale and the background are linear in the pattern and are solved exactly at every sample instead of sampled (the sampled dimension falls to the amplitudes, the cell, the zero and the profile: about twelve for case A); a Laplace approximation around every minimum found by multi-start least squares, which the core already has, is a hundred times cheaper than a sampler and is the first version (§3); and the k-point test prunes the tree.
4. The subgroup list is not computed by the core. The core decomposes a distortion given the parent, the subgroup and the transformation (mode_decomposition, L263) and reads ISODISTORT's files (L278); it does not enumerate the isotropy subgroups of a parent for a set of irreps, which is ISOTROPY's job and a large one. For the perovskite families the trees are tabulated: twelve space groups for the rock-salt-ordered double perovskites (Howard, Kennedy and Woodward 2003), fifteen Glazer tilt systems for ABX₃ (Howard and Stokes 1998; not checked, from memory). The first version takes the tabulated tree of its family as data, with its source; a general enumeration is set aside.
5. Multimodality is real, not only trivial. After the domain-equivalent sets are folded, two distortions of different physics can still fit one pattern within noise (the inverse project's structural and practical non-identifiability). A Laplace approximation sees one basin; a multi-start sees the basins its starts fall in; only a sampler built for several modes (nested sampling, Skilling 2004–2006, not checked; or an ensemble sampler) sees them with their relative mass. That is why the sampler comes after the Laplace version and not instead of it: it is needed exactly where the cheap version is wrong, and the toy shows where.
6. One pattern, today. The posterior from X-rays and neutrons together is the product of the two likelihoods, and it is the case where the module pays most (the oxygen tilts from the neutrons, the cation displacements from the X-rays). The core's multi-pattern refinement is not built (one cell for two patterns waits for it, L110). The module's joint posterior waits with it.

1.4 What exists, checked on 15 October 2026

So the pieces exist separately and the core has the expensive one, the forward model, with its derivatives. What has not been built anywhere is the composition: evidences across a family's tree in mode coordinates, the domain equivalence folded, the nuisances solved, the prior measured from the family, the design question answered, inside the program that then refines.

2. The second idea: the family's configuration space

2.1 What it is, said precisely

The configuration space of one refinement is \(\theta\): amplitudes, cell, profile, the rest. The "generalised" space adds the chemistry \(c\): which ions on A, B, B′ and X, the fraction \(x\) of a substitution, and the descriptors that summarise them (the ionic radii, the tolerance factor \(t\), the charge difference and the radius ratio of B and B′, which decide the rock-salt order), plus the temperature. A family is a region of \(c\) with one parent and one tree; a series such as Sr₂MSbO₆ (M = Ln) is a path through it; chemical pressure is the path along \(r_{\mathrm A}\) or \(r_{\mathrm B}\); a temperature series is the path along \(T\).

What is learned over the family is not a posterior for one compound. It is the map \(c \mapsto p(\theta \mid c)\): how the amplitudes of \(\Gamma\)₄⁺, X₃⁺ and X₅⁺, the cell, the tilt angles, the space group at a temperature and the transition temperatures depend on the chemistry, with their scatter. Written as a model, that is a hierarchy: each compound's parameters are drawn from the family's distribution, whose centre is a smooth function of the descriptors (a low-order regression, or a Gaussian process when the smoothness is not to be assumed) and whose width is the family's scatter plus each study's error. Its predictive distribution for a new composition \(c_\ast\) is exactly the prior §1.4 asked for; and the new compound's refinement, once done, is one more row of the family and moves the map. The loop closes: the family is the prior of the refinement, the refinement is the update of the family.

2.2 The corpus exists: the Perovskite Atlas

Josu's Perovskite Atlas (~/Claude/Projects/igartua/perovskite-atlas, read 15 October 2026) holds the family as typed records with evidence: 47 materials, 41 structures, 66 transitions, 13 series, 47 samples, 41 datasets, all rock-salt-ordered double perovskites with the Fm-3m aristotype: the Sr₂MSbO₆ and Ca₂MSbO₆ antimonates across the lanthanides, the Sr₂(Co,Mg)TeO₆ and Sr₂(Ni,Mg)TeO₆ substitution series, the SrLnMRuO₆ ruthenates, the ALn₂CuTi₂O₉ family. A structure's record carries its symmetry-mode decomposition against the parent, by irrep, with the k-vector, the isotropy subgroup, the amplitude with its e.s.d. and the convention (amplimodes-cell), and each value points at its source page; a material carries its tolerance factor, its oxidation states and its phase sequence (Sr₂CoTeO₆: Fm-3m → I4/m → I2/m → P2₁/n on cooling, \(t = 0.9758\)). make derive already recomputes tolerance factors, volumes and decompositions from the CIFs. This is P3 of the inverse project done by hand over thirty years, for one family, with the parent unambiguous and recorded: it answers the project's Q3 (which family) better than ABX₃, because the data are the group's own and the tree is short.

2.3 What it would give

• A prediction before synthesis, with its credibility. For Sr₂(Co_0.5Mg_0.5)TeO₆ at 300 K: the probable space group, the amplitudes with their predictive widths, the cell. The refinement then measures how far the compound is from its family's expectation.
• The quantitative version of a known qualitative law. That the tilts grow as \(t\) falls below one, and that the sequence of subgroups follows, is textbook; the function with its scatter, from one group's consistent data, is not written anywhere and is a result on its own.
• Design across the family. Which member of a series, which temperature, which substitution would show the I2/m → I4/m transition, or would separate two subgroups whose evidences the single pattern leaves close: the expected information gain of §1.2, summed over the family's map.
• The family drawn. Explore's subspaces for one refinement become the family's: every compound a point in (\(t\), amplitude) with its region, the new one marked, the series as paths. That figure is both an app view and an Atlas page.

2.4 Where it is weak

1. The size. Forty structures. That rules out anything that learns a function without a stated form, and it is why the model is hierarchical regression on a few descriptors and not a network. It also means the map's uncertainty is large where the family is thin (the Ca antimonates are nine compounds, all P2₁/n at room temperature, with no high-temperature study), and the map must say so rather than interpolate confidently.
2. Heterogeneity. The values come from different instruments, radiations and temperatures, and from refinements whose e.s.d.'s are Rietveld's, hence too small by an unknown factor per study; the amplitudes' normalisation differs between AMPLIMODES and ISODISTORT and between reference cells (the core knows both; the Atlas records the convention). The family model needs a per-study error inflation as a parameter and the amplitudes brought to one convention by the core, not by hand.
3. Selection. The Atlas holds what was made, crystallised and published. The map is of that, and a compound the family could not make is not in it. Said plainly on the page, it is a limitation; unsaid, a bias.
4. Circularity, the real danger. A prior from the family pulls a new compound toward the family. The interesting compounds are the ones that break the trend, and a module that smooths them into it would be worse than none. Two safeguards, both standard output: the information gain of the data over the prior for every parameter (how many prior widths the data moved it), so that a compound that moved far is flagged as new; and prior-predictive and posterior-predictive checks (does the family predict a pattern like this one; does the fitted model reproduce the pattern's features), printed with every result.
5. "Then more families." Each family is a parent, a tree and its descriptors: ABX₃ with the fifteen Glazer systems, the 1:2 and 1:3 ordered perovskites (Howard and Stokes 2004, checked by search: ten and eleven structures), the Ruddlesden–Popper phases. The machinery is family-agnostic if those three things are its inputs and nothing about perovskites is wired in. But the second family comes after the first works end to end, and the first is the one with the data.

3. The shape it would take here

• A crate of its own, drs-infer, beside drs-publish: it depends on the core for the forward model, the modes, the equivalent sets, the normal matrix and the covariance, and holds what the core does not: the priors (as files with provenance), the k-point test, multi-start with Laplace evidences, the folding by equivalence, the sampler, the identifiability maps, the design computation, the family model, and a command line (drs-infer survey, drs-infer family). Why not inside core/: the core's tests are bit-exact against FullProf and its code is deterministic; a sampler is stochastic, seeded and long-running, and its tests are statistical (calibration over many draws). L139's rule, every calculation in the core and none in the app, is kept: the crate is core-side, and the app module draws what it returns. Josu may still prefer a module infer inside the core; the difference is build and test hygiene, not principle.
• A stream of its own, as Explore, Publish and Plots: a folder infer/ for the design, the toy and the requests (infer/REQUESTS.md, IN1 …), a branch infer, a worktree, and later app/ui/modules/infer/. The name is Josu's: Infer is proposed (Survey, Prior and Search set aside: each names one view).
• The Atlas as the family's source, not a copy of it: an export from its computed layer (one row per structure: descriptors, temperature, instrument, amplitudes by irrep in one convention with their e.s.d.'s, the source) that the crate reads as a family file, with the Atlas's commit as its provenance. The family's map goes back to the Atlas as a notebook page. Whether the Atlas becomes a stream of this project or stays its own project with an export is Josu's.
• The app module: a survey view (the subgroups ranked by evidence, with their sensitivity to the prior's width, and the k-point test's verdict), the marginals drawn over Explore's maps (the same axes, the posterior's contour beside the \(\chi^2\) map), the family map with the compound on it, and a design view (candidate measurements ranked by what they would decide). Every number from the crate; the module a viewer.
• The order of work, and what each step must pass before the next:
  1. The diagnostic (the inverse project's P1): the stiffness spectrum and the weak directions in the modes' names from R24 and E5, and the credible region of a chosen pair, printed by drs check. Days; the core alone.
  2. The toy (P2): a synthetic double perovskite from a known P2₁/n structure, patterns simulated with the core's Poisson noise under one instrument, the twelve subgroups surveyed by multi-start least squares and Laplace evidences with the equivalents folded; then the calibration test, the one that decides: over a hundred noise draws, do the 90 % regions hold the truth 90 % of the time, under the tempered likelihood and under a discrepancy term; then the information removed (a shorter range, more noise, X-rays for neutrons) and the regions watched to lengthen. Two weeks. The claim "the region means something" becomes a passing test or a failed one.
  3. drs-infer 0.1: what the toy used, as a crate with its tests and its command line, on case A's real data.
  4. The sampler, for the cases the toy shows the Laplace version gets wrong (the saddle, a ridge, two basins): the linear parameters solved inside, the rest sampled on the core's thread pool.
  5. The family: the Atlas export, the hierarchical model, the prior file, the information gain and the predictive checks. A quarter, with Josu, since the descriptors and the error model are his judgement.
  6. The app module, after the crate has a command line for every view.
  Multi-pattern likelihoods wait for the core's multi-pattern refinement; a general enumeration of isotropy subgroups is set aside.

4. For Josu to decide

1. Go ahead with the diagnostic and the toy, with the rule that the module is built only if the toy's regions are calibrated; or set the ideas aside for now.
2. The home: a crate drs-infer and a stream infer, or a module inside the core.
3. The name of the module: Infer, or another.
4. The likelihood: the tempered \(\chi^2\) first and the discrepancy term if calibration fails; or the discrepancy term from the start.
5. The family: the rock-salt-ordered double perovskites with the Atlas as the source, read by export with its commit as provenance; and whether the Atlas becomes a stream of this project.
6. The inverse-problems project: whether this is its phase 1 (its Q2 answered by diffraction leading, its Q3 by this family), to be put to Iñigo; and whether the toy is logged there as P2 taken up, with its code in that project's p2-toy/ or in this project's infer/.
7. The brief: whether R33 (inference before refinement) and R34 (the family's configuration space) are appended now, or when the toy passes.