diff ref struct Rietveld refinement for the Mac · powder diffraction with X-rays, synchrotron and neutrons · started October 2026

The minimisation itself: what molecular dynamics, DFT relaxations, Monte Carlo and machine learning can lend to the Rietveld engine, with the same results

The minimisation itself: what molecular dynamics, DFT relaxations, Monte Carlo and machine learning can lend to the Rietveld engine, with the same results workshop

I06 · an idea of the seed-place, on its way to the core.

Said
2026-10-19, Josu
Status
workshop
Home
core (where it migrates)
Needs
I05

The idea

Beyond reproducing FullProf and GSAS-II, innovate in the minimisation, not only in the information it uses (as mode crystallography does). The Rietveld method is fixed; is its least-squares minimisation the best one can do, in reliability and in speed? Molecular dynamics and DFT codes (SIESTA, for one) relax structures into energy minima with algorithms of their own; Monte Carlo might approach the minimum of a many-parameter problem faster. The application differs, but think about what could be borrowed, and propose another possible way of refining that still lands on the very same results. And, asked straight after: machine learning, neural networks, physically informed models?

Assessment

1. The first answer, the one that orders everything else: locally, the core already has the better minimiser. A DFT relaxation knows the energy and its gradient (the forces, by Hellmann–Feynman) and almost never the Hessian, which costs a phonon calculation. That is why SIESTA relaxes with conjugate gradients, Broyden's quasi-Newton or FIRE (its MD.TypeOfRun, documented in its tutorials; L-BFGS through its Lua engine): all three build curvature from successive gradients because nothing better is affordable. The Rietveld problem is the opposite case. The core computes the exact Jacobian \(J\) of the pattern by dual numbers (L053), and \(J^{\mathsf T}WJ\) is the Gauss–Newton approximation to the Hessian of \(\chi^2\), exact when the residuals are small; near the minimum Gauss–Newton converges quadratically, which no first-order method and no Monte Carlo can match. A Monte Carlo walk approaches a smooth minimum by diffusion, far more slowly; FIRE or conjugate gradients on \(\chi^2\) would throw away the curvature the core already pays for. So the honest answer to "is there a faster algorithm for the last stretch" is no, not by replacing Levenberg–Marquardt; and that is good news, because it means FullProf's result is kept by the engine as it is and the innovation goes where the time is really lost. A refinement loses its time in six places:

  • (i) the start lies outside the basin: calculated and observed peaks do not overlap, the gradient with respect to the cell or the zero is nearly zero, and nothing moves (the reason every manual says "start close");
  • (ii) false minima: a wrong sign of a tilt or of a mode amplitude, occupancy traded against displacement, two species exchanged, a profile asymmetry absorbed by the zero;
  • (iii) sloppy directions: the eigenvalues of the scaled normal matrix spread over many decades (the core shows them, Explore §2.3), the steps zig-zag along a narrow valley and λ climbs;
  • (iv) large residuals: with \(\chi^2_\nu\) of 3–10, common on laboratory data, the term Gauss–Newton drops, \(\sum_i w_i r_i \nabla^2 r_i\), is no longer small;
  • (v) the human's plan: which parameters to free in which order, the main cost of a refinement in practice and the least formalised;
  • (vi) the cost of each cycle when the parameters are many (sequential, parametric and joint refinements).

For each there is a tool from the neighbouring fields. They are grouped below by what they promise: the same minimum reached more reliably and in fewer cycles (A), the whole landscape and the posterior with the same machinery (B), the physics of the energy as extra information (C), and machine learning where its errors are corrected by the likelihood (D).

A. The same minimum, more reliably and faster. Each of these changes the path, not the end point: the minimum it reaches is the least-squares minimum, so FullProf's result is the test.

  • A1. Variable projection for the linear parameters (Golub and Pereyra 1973). The calculated pattern is linear in the scale factors and in the background coefficients (and, in a Pawley fit, in the intensities). For any value of the non-linear parameters the optimal linear ones are a linear least-squares solve; variable projection eliminates them exactly at every evaluation and minimises over the non-linear parameters alone. Golub and Pereyra proved that the reduced functional has the same minima; in practice it converges in fewer iterations and from worse starts, because the problem is smaller and better conditioned. In Rietveld terms, the classical first step "refine the scale and the background" is done exactly at every evaluation. Not found in FullProf's or GSAS-II's manuals (believed, not searched exhaustively).
  • A2. Geodesic acceleration (Transtrum, Machta and Sethna 2011, Phys. Rev. E 83, 036701; Transtrum and Sethna 2012, arXiv 1201.5885). Levenberg–Marquardt is a first-order (Euler) step along a geodesic of the model manifold; adding the second-order term, which needs only the second directional derivative of the residuals along the step, \(\partial^2 r / \partial v^2\), corrects for parameter-effect curvature. The authors report a large rise in the success rate and the speed on sloppy fits. In the core the directional second derivative is one more evaluation with dual numbers seeded along \(v\) (a second-order dual in one direction), the cost of about one extra column of \(J\). The sloppy-model picture (Gutenkunst et al. 2007) is also the right language for Explore's "directions the data cannot fix".
  • A3. The large-residual term (Dennis, Gay and Welsch 1981, NL2SOL). A secant (quasi-Newton) estimate of \(\sum_i w_i r_i \nabla^2 r_i\) is built from successive Jacobians and added to \(J^{\mathsf T}WJ\) only when \(\chi^2_\nu\) is large; with small residuals it switches itself off. Broyden's update here is the same idea SIESTA's Broyden relaxation and its density mixing use, put where it belongs: on the part of the curvature the core does not compute.
  • A4. Continuation: deform the landscape, not the walker. The DFT and MD counterpart is the soft-core potential and graduated non-convexity (Blake and Zisserman 1987). (a) Resolution continuation: convolve the observed and the calculated patterns with a Gaussian of width \(s\), refine, shrink \(s\), refine again, to \(s = 0\). At large \(s\) the peaks overlap even when the cell is wrong, the gradient is not zero and problem (i) disappears; at \(s = 0\) the functional is Rietveld's and so is the minimum. (b) Range continuation: low angles first, where fewer parameters matter, then the whole pattern. (c) Prior annealing in place of the plan: every parameter free from the first cycle, each held by a Gaussian prior around its start whose width grows cycle after cycle (a Tikhonov path); at the end the priors are gone and the minimum is the unrestrained one. It is the principled form of the human's staged plan (v): the data release each parameter when they can fix it, which the scaled normal matrix measures.
  • A5. Reparameterisation as preconditioning. DFT codes relax in redundant internal coordinates (Pulay and Fogarasi 1992) because Cartesian coordinates are badly conditioned. The crystallographic equivalents are partly in the core (symmetry modes, rigid bodies) and partly not: the displacement tensor as its Cholesky factor (positive definite at every step, never a non-positive-definite \(U\) to stop on), \(\ln B\), occupancies through a logistic or on the simplex, the cell through the metric tensor. The minimum is invariant under a reparameterisation; the esd's transform with its Jacobian; the path and the bounds become smooth.
  • A6. Robust likelihood. An impurity peak or a spike pulls every parameter. A Student-t likelihood is minimised by iteratively reweighted least squares, which is the core's engine with the weights recomputed each cycle (it already recomputes them for Iwg). TOPAS has a robust refinement ("Robust Refinement as Implemented in TOPAS", Mater. Sci. Forum 651, 27; seen by search, not read). On clean data it returns the least-squares result; that is its test.
  • A7. The cost of a cycle. For the gradient of a scalar, reverse mode (the adjoint) costs a few forward passes whatever the number of parameters, where forward-mode duals cost one per parameter; it is what the samplers of B need. For very large problems, a matrix-free solve (LSQR) uses \(Jv\) and \(J^{\mathsf T}u\) without forming the normal matrix. In a sequential series, start each pattern from the extrapolation of the previous two, as MD extrapolates the wavefunctions from step to step: a predictor that the core's sequential can test in a day.

B. The landscape, and the posterior, with the tools of statistical mechanics. Here Monte Carlo and molecular dynamics come in, not to reach a minimum faster but to find all of them and to weigh them.

  • B1. Basin hopping with the core's engine as the relaxer (Wales and Doye 1997; minima hopping, Goedecker 2004). Perturb, relax with Levenberg–Marquardt, accept or reject by a Metropolis test on \(\chi^2\), keep a catalogue of the minima found. This is exactly how energy landscapes of clusters are searched, with the local minimiser doing the work and the Monte Carlo only choosing where to start. The moves should be crystallographic, not random: flip the sign of a mode amplitude, exchange two species, apply an operation of the Euclidean normaliser (which gives the same structure: to be folded, not counted, as L286 already folds the domain-equivalent amplitude sets), move to a neighbouring subgroup's description. Global search exists for structure solution: FOX by simulated annealing and parallel tempering (Favre-Nicolin and Černý 2002; its manual), DASH by simulated annealing (David et al. 2006), TOPAS by simulated annealing in the whole profile (Coelho 2000). What is new here is using it at the end of a refinement to answer "is this the minimum, how many others are there, how far above".
  • B2. The temperature ladder: one engine, three products. Write \(p_\beta(\theta) \propto \pi(\theta)\, e^{-\beta \chi^2(\theta)/2}\). At \(\beta \to \infty\) it concentrates on the least-squares minimum: Rietveld's answer, FullProf's number. At \(\beta = 1\) it is the posterior of L286. At small \(\beta\) it explores. Parallel tempering runs a ladder of \(\beta\) at once and swaps states between rungs; and the same run gives the evidence by thermodynamic integration, \(\ln Z = \int_0^1 \langle -\chi^2/2 \rangle_\beta \, d\beta\) (Gelman and Meng 1998), which L286 needs to compare subgroups. Simulated annealing is the same ladder walked once, downwards. The minimisation, the search and the inference become one calculation, and its cold end is the result FullProf gives.
  • B3. Hamiltonian Monte Carlo is molecular dynamics on \(\chi^2\) (Duane et al. 1987; Neal 2011). Give each parameter a momentum, take \(\chi^2/2 - \ln \pi\) as the potential energy, integrate with the leapfrog (velocity Verlet, the MD integrator), accept by the energy error. Josu's intuition is literally right here: the algorithm that samples a many-parameter posterior most efficiently is an MD code. And the core holds what makes it good: the normal matrix is the Fisher information, the natural metric of the problem; used as the mass matrix (Girolami and Calderhead 2011), stiff and sloppy directions move at comparable speed, which is what Gauss–Newton does for the minimum. Cooling the thermostat is annealing; NUTS (Hoffman and Gelman 2014) chooses the trajectory length. Bayesian powder analysis exists (Fancher et al. 2016, Sci. Rep. 6, 31625, with MCMC; BLAND, Lesniewski et al. 2016), not with the Fisher metric of a refinement engine.
  • B4. A fast surrogate with an exact check. David and Sivia showed that the profile \(\chi^2\) near a Pawley fit equals, to second order, a \(\chi^2\) over the extracted intensities with their covariance matrix; DASH's annealing uses it and is about two orders of magnitude faster per evaluation (DASH's paper, as summarised by search). The core has Pawley fits (pawley.rs). Delayed-acceptance MCMC (Christen and Fox 2005) screens each proposal with the surrogate and confirms it with the exact profile; the chain samples the exact posterior. The hot rungs of B2 run on the surrogate, the cold ones on the profile.
  • B5. The landscape as a picture (for Explore). Two refinements that end in different minima are joined by the minimum-\(\chi^2\) path found by the nudged elastic band (Henkelman, Uberuaga and Jónsson 2000, the DFT tool for reaction barriers): the barrier in \(\Delta\chi^2\) along it says whether the data really separate the two solutions or whether they are one broad basin. The minima of B1 drawn as a disconnectivity graph (Becker and Karplus 1997) show the whole landscape in one figure. Not found for Rietveld refinement (believed; not searched beyond this session).

C. The energy as information, which is what DFT can lend in earnest. Not its minimiser but its landscape. The structure measured at temperature \(T\) is a thermal state, so a physically meaningful prior over the positions is Boltzmann's, \(\pi(\theta) \propto e^{-E(\theta)/k_B T}\), and the posterior is \(\exp[-\chi^2/2 - E/k_B T_{\text{eff}}]\).

  • C1. Which energy. Now: the core's bond-valence table (bv_table.rs) gives a soft bond-valence energy (Adams' softBV and BVSE, believed, not re-read), cheap and with derivatives. Next: the universal machine-learned interatomic potentials (MACE-MP-0, CHGNet and others) give near-DFT energies and forces for any chemistry in milliseconds. The weight: an approximate energy must be tempered, and \(T_{\text{eff}}\) is inferred from the evidence, as MEM's \(\alpha\) is (I05): the same idea in another place.
  • C2. What exists, checked by search on 19 October 2026. Scatter2xyz (Anker, Gardner, Rosset, Goodwin and Deringer, arXiv 2510.05938, October 2025) optimises populations of walkers against powder, small-angle, total scattering and PDF data together with a machine-learned potential's energy and chemical constraints, with automatic differentiation and the Adam optimiser: a unified differentiable loss, aimed at structure solution and autonomous laboratories. TOPAS adds potential-energy penalties (grs_interaction, electrostatic plus user terms; its forum). DFT-D validation of powder structures compares the refined structure with its energy minimum after the fact (van de Streek and Neumann 2010, believed). Hoser and Madsen (2016, 2017, Acta Cryst. A) refine the frequencies of ab initio normal modes against single-crystal data, so that the displacement parameters come from lattice dynamics. Not found: inside a refinement program, the energy as a Boltzmann prior of the posterior with its weight inferred, esd's from that posterior, and the result compared with the unrestrained refinement.
  • C3. Displacement parameters from the Hessian. In the harmonic approximation the mean-square displacements follow from the phonons: \(U = \frac{\hbar}{2N}\sum_{\mathbf q,j} \frac{e e^{\dagger}}{m\,\omega}\coth\frac{\hbar\omega}{2k_BT}\), with the eigenvectors and frequencies of the potential's Hessian. Powder data fix ADPs worst of all parameters; refining one scale per group of modes in place of six numbers per atom (Hoser and Madsen's idea carried to powders and inorganic solids) gives physically consistent ADPs, the test being neutron data with known ADPs.
  • C4. Modes and the Landau potential. For a distorted perovskite the energy over the amplitudes of a few irreps, \(E(Q_1, \dots, Q_k)\), is low-dimensional and can be tabulated once with the potential (frozen phonons in the parent). It is the measured prior over amplitudes that L286 lacks (its named weakness: the evidences depend on the prior's width, Lindley); its trilinear couplings tie amplitudes that the data fix poorly to those they fix well; in a temperature series, the amplitudes follow \(Q(T)\) from the same potential, a parametric refinement with physics in it (parametric.rs).
  • C5. Forces as a diagnostic. After a plain refinement, the potential's forces on the refined atoms flag the positions the energy disputes: a view in Explore, with no change to the result, available as soon as an energy is.
  • The caution. The energy is information the diffraction data do not contain; the result with it is not FullProf's, and must not be presented as if it were. It is shown beside the unrestrained refinement, with the evidence for including it.

D. Machine learning, neural networks and physics-informed models: the rule "the network proposes, the likelihood decides". A network's output carries an error that nobody can bound for a pattern unlike its training set; the exact likelihood, which the core computes, has no such error. So every learned piece goes where its error is corrected by the likelihood (a start, a proposal, a prior with an inferred weight, a model of the misfit) and never at the end point. Where that holds, the result is still the least-squares minimum or the exact posterior, and the network only makes it faster or surer.

  • D1. What exists, checked by search on 19 October 2026. Mun, Nam and Choi (J. Appl. Cryst. 2026, "Automation of Rietveld refinement through machine learning") train a convolutional network on simulated patterns; it predicts the structural and profile parameters in about 2 ms, and those are the start of an ordinary least-squares refinement, with \(R_{wp}\) within 1 % of the manual one on CeO\(_2\), Tb\(_2\)BaCoO\(_5\) and PbSO\(_4\). RADAR-PD (arXiv 2605.12478) identifies and quantifies phases with a network and verifies by Rietveld. Riesel et al. (2024) generate structures from patterns (67 % on RRUFF, L286). Scatter2xyz (C2) puts a learned potential in the loss. All of them use the network before or beside the refinement, none instead of it, and that is right.
  • D2. Amortised inference: a network that has learned the posterior (simulation-based inference; neural posterior estimation, reviewed in "Simulation-Based Inference: A Practical Guide", arXiv 2508.12939). Train, once per family and instrument, a network on pairs (parameters, pattern) generated by the core's own forward model; for a new pattern it returns an approximate posterior in milliseconds. Used alone it is a guess; used as the proposal of importance sampling or of the samplers of B, it is corrected by the exact likelihood and the posterior is exact, which X-ray astronomy already does for spectral fits (arXiv 2512.16709, "deriving exact posteriors with dimension reduction and importance sampling"). For the core it answers problem (i) and part of (ii): many starts in the right basins, at once. Mun et al. is the one-point version of it.
  • D3. A learned geometry for the sampler. A normalising flow trained on the posterior's samples maps it to a near-Gaussian, and HMC runs in the warped space (NeuTra, Hoffman et al. 2019, arXiv 1903.03704, which reports large gains in effective sample size). It changes the speed of B3, never its target. Across a sequential series, the flow of one pattern is the start of the next.
  • D4. The machine-learned potentials are the physics-informed machine learning that matters here (C1): equivariant under rotations and translations by construction, trained on DFT, giving energies, forces and Hessians. They bring physics into the refinement as a prior with an inferred weight, which is where a learned model's error is acceptable.
  • D5. A learned model of the misfit. Rietveld's \(\chi^2\) is not a likelihood because the residuals are structured (L286's first risk). A Gaussian process over \(2\theta\), with a kernel whose length follows the peak widths, models the systematic misfit (Kennedy and O'Hagan 2001); the parameters are then inferred with the misfit integrated out, and their esd's are no longer overconfident. This is machine learning in the statistical sense, small and interpretable, and it is what makes the posterior honest.
  • D6. The core as a differentiable physics layer. "Physics-informed neural networks" in the narrow sense (Raissi, Perdikaris and Karniadakis 2019) put the residual of a differential equation into a network's loss; the refinement has no such equation, so that literal form does not apply. The sense that does apply: the physics in the architecture, not learned. The core's forward model with exact derivatives (and, with A7, its adjoint) is a differentiable layer; a network that outputs mode amplitudes, cell and profile parameters and is trained through the core's pattern and \(\chi^2\) learns on the true physics, not only on simulated pairs. The inductive biases are the crystallographer's: mode amplitudes rather than coordinates, invariance under the Euclidean normaliser, the subgroup tree.
  • D7. Learning the plan. Which parameters to free and when (problem v) can be learned from the bench's runs (step 0), as black-box tuning of refinement settings was (Ozaki et al. 2020, L286). Prior annealing (A4 c) is the principled alternative; the two can be compared on the bench, and a learned plan must be explainable through the normal matrix to be accepted.
  • Where machine learning is not worth it. A neural emulator of the forward model: the core's is exact and takes 5–40 ms, and the Pawley surrogate (B4) is exact to second order; an emulator would add error and gain little. A deep network trained on the Perovskite Atlas: 47 materials are a hierarchical Bayesian model's size, not a network's (L286). A network as the engine: its result cannot be traced to the data, and FullProf's number would no longer be the test.

What is not proposed. Monte Carlo, genetic algorithms or Adam as the local minimiser (slower than Gauss–Newton near the minimum, and their end points depend on settings); a network that maps patterns to structures as the engine (not reproducible, not traceable to the data); any method whose cold limit is not the least-squares minimum, presented as Rietveld refinement.

Proposal

The engine of version 1 stays Levenberg–Marquardt and keeps FullProf's results. Everything below is an option, behind a test that it lands on the same minimum where it claims to.

  • Step 0. The bench, before any algorithm. The five runs in manuals/fullprof/runs/ and Explore's two cases, each started from 100 perturbed starts (inside physical ranges, seeded); measured per method: the fraction that reach FullProf's minimum (\(\chi^2\) within its tolerance, every parameter within 0.01 esd), the cycles, the evaluations, the time. Without it "faster" and "more reliable" are guesses. A finding with the present engine's numbers.
  • Step 1. Variable projection (A1) for the scale factors and the background coefficients, against step 0.
  • Step 2. Geodesic acceleration (A2) with the second directional derivative by dual numbers, against step 0; then the large-residual term (A3) on the laboratory runs.
  • Step 3. Continuation (A4): resolution and prior annealing, measured as the size of the basin of attraction (how far a start can be and still arrive), and prior annealing against FullProf's plans.
  • Step 4. Basin hopping with crystallographic moves (B1) on case A of the mode material: the catalogue of minima, and the nudged elastic band between two of them (B5); requests to Explore for the two views.
  • Step 5. The ladder (B2–B4): parallel tempering with Fisher-metric HMC, delayed acceptance on the Pawley surrogate; the sampler and the evidence of L286, checked by its calibration test (90 % regions hold the truth 90 % of the time) and its cold end checked against FullProf.
  • Step 6. The energy (C): first the bond-valence energy with its weight by the evidence; the machine-learned potential where Josu decides it lives (core or extension, I02); then C5, C3 and C4.
  • Step 7. Robust likelihood (A6) and the reparameterisations (A5), each against step 0.
  • Step 8. The learned pieces (D), after step 5 exists to correct them: the misfit model (D5) first, since L286's calibration depends on it; then amortised starts and proposals (D2) for the double perovskites, trained on the core's forward model and judged by the bench (more starts in the right basin, the same end points); then the flow (D3) if the sampler is the bottleneck. Training happens outside the app; where a trained network is evaluated (core or extension) is Josu's call, as for the potential.

Steps 0–3 are engine work for the core stream, a few versions each; 4 and 5 are research, and a paper (the ladder whose cold end is Rietveld's result and whose evidence chooses the subgroup); 6 is a second paper; 8 joins the first.

The workshop

The idea worked on, note by note, dated; the latest last.

A stream of its own, a bench, and the refinement as a sequence of measured stages

2026-10-19

Josu, thinking out loud the same day: the innovations must not interfere with the work done; perhaps they go as extensions of the app; perhaps a parallel stream for steps 0–2, kept as options; built and proved against the known cases and results; compared on the ease of reaching the result, with gain parameters defined and measured, and the gain looked for not only in the total time but in partial times, with intermediate steps that depend on the case; the aim, a contribution: a proposal for a Rietveld refinement more robust in every respect. Read below as six claims, each taken further, and a critique.

1. "As extensions": the aim kept, the form adjusted. The aim is no friction: nothing done so far may move because of an experiment. An extension in I02's sense, a module installed from outside the app with numbers from outside, does not fit, for two reasons: I02 is not designed yet, so the work would wait on it; and a minimiser is the heart of the calculation, which L139 keeps in Rust under the core's tests. The form that gives the aim without either cost is the one Publish already has: a stream with its own crate (here called drs-lab in lab/; the name is Josu's), its own worktree and branch, depending on the core as a library exactly as drs-publish does, and asking for the rest in lab/REQUESTS.md. It calls the core's forward model and derivatives (Model::calculate, Model::derivatives) and never copies them, so every number it shows comes from the same physics; it never edits core/src/lsq.rs. In the app, its methods sit behind a Lab switch, off by default. A method moves into the core only when the bench shows its gain and Josu decides, which is the seed-place's path from workshop to migrated. It is an extension in spirit (outside, removable, frictionless) and core in substance.

2. The guarantee that removes the friction: the last stage is always the reference engine. Every lab strategy ends by handing its point to the core's Levenberg–Marquardt with FullProf's convergence test (every shift below Eps × esd). The end point is then the reference engine's by construction; the innovation changes the path, never the result. This splits I06's methods in two, and the bench treats the two differently:

  • path methods (A1–A5, A7, the starts of D2, basin hopping used to certify): must land on the reference minimum, every parameter within 0.01 esd, and are judged on the cost and the reliability of getting there;
  • information methods (the energy of C, the robust likelihood A6, the misfit model D5): change the result on purpose, are shown beside the reference result and never instead of it, and are judged on whether they come nearer the truth (synthetic cases) and on the calibration of their esd's.

3. The known cases are already here, and richer than I06 said. manuals/fullprof/runs/ holds about forty FullProf runs; FullProf's fp2k runs headless on the Mac (run_fullprof.py); and converge_runs.py (L052) already refines each tutorial from its own starting file, CODE.pcr.in, to FullProf's convergence. The bench therefore has three kinds of start, one more than I06's step 0:

  • the tutorial's own start, the path a user really takes, with FullProf's converged result as reference;
  • perturbed starts, seeded, at graded distances from the reference;
  • synthetic patterns computed by the core from a known structure, with graded difficulty (peak overlap, pseudo-symmetry, number of parameters, noise): here the truth is known, not only FullProf's answer. And FullProf itself is run from the same perturbed starts: FullProf's own robustness becomes the baseline, measured. Not found published: the IUCr round robins compared results between programs and users (Hill and Cranswick 1994, J. Appl. Cryst. 27, 802; believed, not re-read), not convergence from controlled starts.

4. Partial times, made exact: four phases of a refinement. Josu's "partial times" need phases with operational definitions, or each method will claim the phase it is good at.

  • Capture: from the start to the first iterate from which the reference engine alone reaches the reference minimum. Defined backwards on the bench, by restarting the reference engine from every iterate: costly, bench-only, and exact.
  • Descent: from capture until \(\chi^2 \le 1.01\,\chi^2_{\text{final}}\).
  • Polish: until every shift is below Eps × esd.
  • Certification: the extra cost of showing, within a stated budget of basin-hopping trials, that no lower minimum was found; zero for a method that does not certify, and then said so.

The costs are counted in forward evaluations and Jacobian evaluations separately (a Jacobian by duals costs about as many forward passes as there are parameters; measured, not assumed), with wall time on a fixed machine as secondary, because the Mac's load varies. The gain parameters, per method and case:

  • reach: the success rate against the distance of the start, summarised by \(r_{50}\), the distance at which half the starts arrive; distances per parameter class in physical units (cell in %, coordinates in Å, \(B\) in Å\(^2\)) and in esd's;
  • cost per phase, as above;
  • ease: the user's decisions needed (plan steps written, parameters freed by hand, restarts); a strategy that needs none scores best;
  • tuning-freedom: the success rate over a grid of the settings (λ's start, damping); a robust method needs no tuning;
  • agreement: \(\chi^2\) and every parameter within 0.01 esd of the reference (path methods), the esd's' ratio;
  • failure taxonomy: diverged, singular, false minimum (which one), held at a bound, budget exhausted.

Across the cases, the summaries the optimisation field uses to compare solvers on a test set: performance profiles (Dolan and Moré 2002, Math. Program. 91, 201) and data profiles (Moré and Wild 2009, SIAM J. Optim. 20, 172). Not found used for Rietveld refinement (believed; not searched beyond this session).

5. Intermediate steps that depend on the case: the refinement as a sequence of stages with switching rules. The core already has the plan: steering::PlanStep is a named step with the parameters it frees, a damping, a number of cycles and an Eps. The lab adds two fields to it: the method of the stage, and the trigger that ends it. The triggers are read from what the core already reports each cycle (Cycle: λ, \(\chi^2\), the shifts in esd's, the eigenvalues of the scaled normal matrix), plus one new measure, a peak-overlap index (the fraction of the observed net intensity lying under calculated peaks), which tells capture from descent. An example: continuation while the overlap index is below 0.6; then Levenberg–Marquardt with variable projection and geodesic acceleration while λ is climbing; then the reference engine to polish; then, optionally, certification. Which sequence suits which class of case (laboratory X-rays, neutrons, several phases, refinements by modes) is itself a result of the bench: a table from the class of case to the best sequence, with the gain in each phase. Written by hand first; learned later (D7). The app can then report it in the user's terms: "capture took 14 cycles with continuation, against 40 without (or a false minimum)".

6. The contribution: a Rietveld refinement robust in six senses, each with its measure.

  • to the start: \(r_{50}\);
  • to false minima: certification;
  • to the settings: tuning-freedom;
  • to the user: the decisions needed;
  • to the model's misfit: the calibration of the esd's (A6, D5);
  • reproducible: the same result from any start inside the basin, seeded, every run recorded. Two papers before the ladder (B2) and the energy (C): the bench and its measures, with the baselines of FullProf and of the core (the tutorials' data are distributed with FullProf, so their licence is checked before the bench is published); and the staged, certified minimisation, with its gains per phase.

The critique. (a) Tutorials are chosen because they work; a bench of tutorials alone flatters every method, so the synthetic graded cases and some of Josu's hard cases are needed, the latter kept private and off the site. (b) Wall time on a loaded Mac misleads; evaluations are the primary count. (c) Methods tuned on the bench overfit it; some cases are held out until the end. (d) "The same result" is tested on every parameter, not on \(\chi^2\) alone. (e) The honest outcome may be modest: no gain from the tutorials' starts, a large one from far starts. That is a result, and it says where the method matters. (f) A stage language risks becoming a second control-file dialect; it stays the plan with two fields, inside the lab, until a stage migrates.

Steps, replacing I06's steps 0–2. L0: the stream set up (folder, crate, worktree, branch, requests), with no change to the core. L1: the bench (the three kinds of start, the four phases, the measures), with the core's engine and FullProf measured: the baseline, a finding. L2: variable projection. L3: geodesic acceleration, first with the second directional derivative by a finite difference of \(J\) along the step, then by a request to the core for it in dual numbers. L4: continuation and the overlap index. L5: stages with triggers, and the table by class of case. Each one a finding with its numbers, against L1.

The audit of 21 October: where Bayesian optimisation belongs, the precedents the first assessment missed, three more ways to help the minimisation, and a project of its own

2026-10-21

Josu asked, two days after the first notes, for an audit of I05 and I06 together: can the minimisation be taken further and helped, and how; does Bayesian optimisation, as a primer on hyperparameter search presents it (codecut.ai, "Bayesian optimization for hyperparameter search", read 21 October 2026), make sense here; is there literature, is anyone working on this; and why were the references the session gave in its chat not on the site. Then, while this note was being written: perhaps the whole topic should move to a site of its own, to go deeper first and implement after. Everything below was checked by search and reading on 21 October 2026; the references at the foot of this page say of each source whether it was read in full, in abstract, found by search, or is believed from memory.

1. Bayesian optimisation: what the primer says, and where the method belongs in a refinement. The primer is correct and small. A Gaussian process (a Matérn 5/2 kernel plus white noise) is fitted to the values of a black-box function; the expected improvement says where to evaluate next; a one-dimensional synthetic function is minimised in 13 evaluations where a grid would need hundreds; and it closes with "the principles apply equally to higher-dimensional spaces". That claim comes with the field's own limits (Frazier 2018, the standard tutorial; Shahriari et al. 2016): the objective should be expensive, minutes to hours per evaluation; the domain of fewer than about twenty continuous dimensions; derivatives absent; and the Gaussian process costs the cube of the number of evaluations, so that after a few hundred the surrogate costs more than the function. The Rietveld problem as the core poses it is the opposite on every count: tens to hundreds of parameters, a forward evaluation of 5–40 ms, the exact Jacobian by dual numbers and the Gauss–Newton Hessian with it. Bayesian optimisation over the structural parameters would reach the minimum slower than Levenberg–Marquardt by orders of magnitude and would give no esd's. As the engine of descent and polish it is the wrong tool, for the reason Monte Carlo was (§1 of the assessment).

The field has already found where it does belong, and the first assessment did not name the fact: where the black box is a whole refinement. Three published cases.

  • BBO-Rietveld (Ozaki, Suzuki, Hawai, Saito, Onishi and Ono 2020, npj Comput. Mater. 6, 75). The variables are the recipe: which parameters are refined, in which order, with which settings; each trial is a complete GSAS-II refinement through GSASIIscriptable; the sampler is Optuna's tree-structured Parzen estimator, a Bayesian optimiser over a mixed discrete space; the objective is \(R_{wp}\). It is reported to surpass a human expert and an expert-system automation at about 100 trials, about 30 minutes on an ordinary computer (KEK's release of June 2020; the paper's own numbers not re-read, its page blocked), and the sampler is distributed on OptunaHub as bbo_rietveld (100 trials, 10 of them random, in its example). L286 cited this as "black-box tuning of a refinement's settings"; said exactly, it is Bayesian optimisation of the plan, problem (v) of I06, published six years ago, with Y\(_2\)O\(_3\) as its worked example, the core's own first reference case.
  • CrystalMO-TuRBO (Agada, Wang and Biswas, arXiv 2609.20592, September 2026) and the pcVAE-assisted Bayesian optimisation before it (Agada, McAninch, Day, Tanyu, McCombs, Koohpayeh, Toby, Wang and Biswas, arXiv 2601.00855, December 2025; Toby is GSAS-II's author). Here Bayesian optimisation does take the structural parameters, of a single-crystal pyrochlore Ho\(_2\)Ti\(_2\)O\(_7\) with defects, refined jointly against X-ray and neutron data as two objectives: trust-region Bayesian optimisation (TuRBO, Eriksson et al. 2019) and sparse axis-aligned subspaces for the dimension, a variational autoencoder constrained by the physics to compress thousands of reflections into a few latent coordinates, GSAS-II as the reference, and "improved convergence, robustness and parameter precision" against classical refinement and plain Bayesian optimisation (abstracts read; the numbers not). The case is a rough landscape with many minima, and the method is a global searcher that pays for every evaluation with a full GSAS-II call.
  • Spotlight (Biwer, Feng, Finstad, McDonnell, Knezevic, McKerns, Savage and Vogel 2025, Sci. Rep. 15, 8358, Los Alamos; read in full). An ensemble of optimisers from mystic (Powell's method first, Nelder–Mead, differential evolution, chosen because they "do not require a smooth, differentiable function"), with surrogates of the \(R_{wp}\) surface (radial basis functions, Gaussian processes, neural networks), finds starting lattice parameters and phase fractions for MAUD, GSAS and GSAS-II; the whole refinement is the black box. Its numbers are the price of that: 685,065 pattern evaluations in 2 hours on 160 processors for GSAS-II's Al\(_2\)O\(_3\) tutorial, "more than 7.9 days" in series; 1,632 refinements in 2 hours on one node for PbSO\(_4\); 146,832 MAUD refinements over a 336-point lattice-parameter grid for 437 Ti–6Al–4V histograms, about 17 hours. And it states the rule the bench needs, in one sentence: "deviations of less than 1 % in the lattice parameters from the true value can prevent convergence of the analysis as peaks of the measurement and model do not overlap", which is problem (i); with the authors' own caution that an unknown phase still needs an expert.

What this gives I06. (a) The place of Bayesian optimisation in the stream: over the stage table of §5 and the settings of each stage (the method, the trigger, the schedule of blur, range and prior width, the basin-hopping temperature and move sizes), with the bench's measures as the objective and not \(R_{wp}\): a lower \(R_{wp}\) can be a wrong model, and Ozaki's objective is the one part of his design the bench should not copy. A few dimensions, an expensive objective (a bench run), no derivatives: the textbook case. Also, as one capture method among others, over the handful of parameters that put a start outside the basin (cell, zero, displacement: at most eight), on the variable-projected \(\chi^2\) of A1; whether it beats a Sobol sweep followed by Levenberg–Marquardt, or continuation, is the bench's question, since each evaluation costs 40 ms and the surrogate's cost overtakes it after a few hundred points. Never for descent, polish, or as the engine. (b) External baselines for the bench: Spotlight's Al\(_2\)O\(_3\) and PbSO\(_4\) are GSAS-II's public tutorials and BBO-Rietveld's Y\(_2\)O\(_3\) is the core's first case; the bench runs the same data, so that "capture in \(N\) evaluations" is compared with 685,065. (c) The claim, sharpened: the three open the refinement as a black box because their engine is opaque to them; the core's is open (derivatives, the normal matrix, variable projection, the Pawley surrogate), and I06's thesis becomes measurable: an open engine makes capture cost thousands of times less. Spotlight's numbers are the figure to beat.

2. The precedents the first assessment missed, by the section they enter.

  • The likelihood (A6; L286's first risk). Antoniadis, Berruyer and Filhol 1990 (Acta Cryst. A46, 692) gave the maximum-likelihood refinement for Poisson counts, the correct objective where counts are low; iteratively reweighted least squares reaches it with the engine as it is, the weights computed from the model's counts and not the data's, each cycle, as the core already recomputes weights for Iwg. David 2004 ("Powder diffraction: least-squares and beyond", J. Res. NIST 109, 107; abstract read) says why least squares rules ("because that is the way it has always been done"), gives a robust likelihood with Jeffreys' prior for unknown uncertainties (a logarithmic penalty for large residuals, the alternative to A6's Student t), a Bayesian treatment of an unmodelled impurity that assumes only its positivity, and the cumulative \(\chi^2\) as the diagnostic of where the misfit lies, which Explore can draw at once. The robust-resistant refinement of Prince and Nicholson (1985) is the classical form (believed, not re-read).
  • The misfit model (D5). Gagin and Levin 2015 (J. Appl. Cryst. 48, 1201; GSAS_USE, an extension of GSAS-II, on GitHub): unknown systematic errors in three classes, multiplicative, additive and of the peak shape, modelled as nuisance corrections and marginalised, so that the refined parameters are "probability-weighted averages over all possible error corrections". This is Kennedy and O'Hagan's discrepancy term inside a Rietveld program, done eleven years ago; D5 starts from it, and L286's calibration test (a 90 % region holding the truth 90 % of the time) is what remains to be shown (believed; the paper's tests not read).
  • The engines of the reference programs, named. FullProf: Marquardt damping on the normal equations (its manual). GSAS-II: Levenberg–Marquardt on the analytic Hessian with a singular-value decomposition (Toby and Von Dreele 2013, J. Appl. Cryst. 46, 544). TOPAS: Newton–Raphson with the Marquardt method and a bound-constrained conjugate-gradient solution of the normal equations (Coelho 2005, J. Appl. Cryst. 38, 455; Coelho 2018, J. Appl. Cryst. 51, 210), the one paper on the minimisation algorithm itself written for a Rietveld program in twenty years, and the bench's third baseline where TOPAS is available. And Toby 2024 ("A simple solution to the Rietveld refinement recipe problem", J. Appl. Cryst. 57; abstract read): each parameter is stepped by a small offset up and down, and its derivatives tell a meaningful parameter from numerical noise, so that the program decides which parameters may be turned on. A deterministic answer to problem (v) from inside the engine, and the natural trigger of §5's stages: a parameter is released when its derivative is significant, which the scaled normal matrix the core already reports measures without the extra evaluations.
  • The equivalence of B4, cited exactly. David 2004, J. Appl. Cryst. 37, 621 (DOI 10.1107/S0021889804013184): the Rietveld method and the two-stage Pawley-then-correlated-intensities refinement are equivalent to second order, "extremely useful … where a great many refinements need to be performed very quickly"; Shankland 2004 ("Global Rietveld refinement", J. Res. NIST 109, 143) reports DASH at about 3,500 trial structures per second on that surrogate. B4's "as summarised by search" is now a checked citation.
  • Starts from random points as a measure (ii; §3 of the first note). Kerman, Campbell, Satyavarapu, Stokes, Perselli and Evans 2012 (Acta Cryst. A68, 222) use "the average \(R_{wp}\) over repeated local minimisations from random starting points" as the cost of a mode set in their TOPAS search; and Spotlight's 336-point lattice-parameter grid is convergence measured from controlled starts for one class of parameter. The first note said this had not been published; it had, in these two partial forms. The bench's novelty is narrower and still real: every parameter class, FullProf and the core as baselines from the same starts, the four phases with their costs, and path methods judged on landing on the reference minimum.
  • The landscape (i, ii, D6). Segal, Subramanian, Li, Miller and Gómez-Bombarelli 2025 (arXiv 2512.04036, "The loss landscape of powder X-ray diffraction-based structure optimization is too rough for gradient descent"; abstract read): the common pattern-similarity losses are highly non-convex in the structural parameters, remain so "along certain symmetry axes" even inside the right crystal family, and symmetry constraints raise the recovery rate substantially. It confirms I06's reading of (i) and (ii), supports modes over coordinates as the inductive bias, and warns D6: a network trained end to end through the pattern's \(\chi^2\) walks that landscape.
  • A differentiable engine with calibrated uncertainty. AiMat-DRX (Marcelo Junior, Gomes and Pontes de Deus, Zenodo 10.5281/zenodo.22679432, version 1.2.0 of September 2026, MIT; record read): a Rietveld engine in JAX for quantitative phase analysis, with Laplace uncertainties from the exact Hessian, the Bérar–Lelann inflation, a parametric Poisson bootstrap, conformal calibration, and a "declared domain of validity" computed from the weighted residuals, validated on the IUCr quantitative-phase round robin (Madsen et al. 2001; Scarlett et al. 2002). The nearest neighbour of the core's position (exact derivatives, calibrated esd's) and of L286's calibration test; its domain of validity is a cousin of the bench's failure taxonomy. To be read; where its data are public, run on the bench.
  • Agents on the plan (v, D7). AutoXRD (Wu and Sun, arXiv 2609.00070; XRDBench: ten language models averaged 57.8 of 100, weakest at "coupled-parameter control" and "workflow termination") and AgentBuild (Shin, Bridges, McDonnell and Ferreira da Silva, arXiv 2606.12834, Oak Ridge; GSAS-II through MCP, a rubric-graded curriculum on LLZO data). Their measured failures are what the normal matrix and §5's triggers formalise; a learned or prompted plan (D7) is judged on the bench as any other method, and the bench is what they lack.
  • A different estimator. WPEM (Cao, Zhang, Feng, Zhang, Huang, Weng and Zhang, arXiv 2602.16372, February 2026): expectation–maximisation over the whole pattern as a mixture density with Bragg's law as the constraint; lower \(R_p\) and \(R_{wp}\) than FullProf and TOPAS claimed on PbSO\(_4\) and Tb\(_2\)BaCoO\(_5\). A lower \(R_{wp}\) is not the bench's criterion; the method is a path to measure on the same public data and an estimator to compare with least squares on synthetic truth, as I05's (a) is.
  • Hamiltonian Monte Carlo in scattering (B3). Jiang, Wang, Tirrell, de Pablo and Chen 2022 (J. Synchrotron Rad. 29, DOI 10.1107/S1600577522003034): HMC "with appropriate preconditioning" beats random-walk MCMC on SAXS, reflectivity and fluorescence holography. Not Rietveld, and without the Fisher metric; B3's claim stands, with a neighbour.
  • Optimal transport in diffraction. LaueOT (Kacprzak et al. 2024, arXiv 2404.06478): Sinkhorn optimal transport assigns spots to grains, 1,000 grains in under 30 minutes, robust to 70 % outliers; PARA-X (Cocke, Camacho, Gorske, Faber and Bhattacharya, arXiv 2609.13619, September 2026): "a differentiable optimal transport-type objective" between simulated and measured diffractograms in a physics-aware global refinement for high-energy diffraction microscopy. Optimal transport has entered diffraction; not yet the capture of a powder refinement, which is the first proposal below.

3. Three more ways to help the minimisation, each a path method, each with its test.

  • A4d. An optimal-transport misfit for capture. Problem (i) has an exact twin in seismic full-waveform inversion, "cycle skipping": when the computed and the observed oscillations do not overlap, the \(L_2\) gradient points nowhere. Its cure since Engquist and Froese 2014 (Commun. Math. Sci. 12, 979) and Métivier, Brossier, Mérigot, Oudet and Virieux 2016 (Inverse Problems 32, 115008; Yang, Engquist, Sun and Hamfeldt 2018, Geophysics 83, R43) is the Wasserstein distance, convex with respect to shifts and dilations of the signal, which is how a wrong cell, zero or displacement moves peaks. In one dimension, between two normalised, background-subtracted patterns, \(W_2\) is the \(L_2\) distance between their quantile functions and \(W_1\) the \(L_1\) distance between their cumulative patterns; the squared distance between the cumulative patterns (Cramér's) is a plain least-squares residual, so the core's engine minimises it with dual-number derivatives and no new machinery. The cell's gradient is non-zero at any distance, and at the optimum the two patterns agree, so the hand-over to \(\chi^2\) lands in the basin; per-window transport (several \(2\theta\) windows) keeps each region's scale. Precedents in diffraction are LaueOT and PARA-X; none found for the capture of a powder refinement. Test: \(r_{50}\) on the bench against the plain engine and against resolution continuation (A4a); the end point the reference engine's, by construction. The best ratio of novelty to cost in this note; L4 takes it beside continuation.
  • A6′. The Poisson likelihood as an option of the engine. Antoniadis 1990 by iteratively reweighted least squares, the weights from the calculated counts: on FullProf's runs with FullProf's weights it must reproduce FullProf (the test); on synthetic low-count data (a short synchrotron frame, a weak neutron pattern) it removes the known bias of weighting by the observed counts. Cheap, exact, and part of what L286's likelihood needs anyway.
  • §5′. A parameter released by its derivative, not by hand. Toby's offset test and the core's scaled normal matrix measure the same thing; a stage's trigger "release \(p\)" fires when \(p\)'s diagonal element of the scaled normal matrix and its eigen-component pass a threshold set on the bench: prior annealing (A4c) with the data choosing the schedule. Test: against the tutorials' hand-written plans, the same minimum, fewer decisions (the ease measure).

4. I05, read again beside this. Its references were in the log's sources (L298, L301) and are now on its page. One link to add: the move the two ideas and Bayesian optimisation share is a single Bayesian move, the weight of a prior set by the evidence: MEM's \(\alpha\) (I05), the energy's \(T_{\text{eff}}\) (C1), and a Gaussian process's hyperparameters by the marginal likelihood (the fitting step of Bayesian optimisation, and of D5's misfit model); Gull 1989 and Rasmussen and Williams 2006 (ch. 5) are one argument in two fields. Nothing found contradicts I05; its entropic refinement (a) still has no precedent for powders, and WPEM's mixture-density estimator is the nearest different estimator to compare it with.

5. The references, on the page. Josu noted that the references of the ideas chats were nowhere on the site: the chat of 19 October listed seventeen sources that the page, which cited them by author and year, never linked. From today each idea's page ends with its references, each with a link where there is one and with how it was checked (read, abstract, by search, believed); the seventeen are among I06's. The rule stands as it was written: a claim's evidence is on the page that makes the claim.

6. A project of its own. Josu's second message of the day: perhaps the topic (I05, I06, the sessions they came out of, this audit) should move to another website dedicated to it, to go deeper with all the references, knowing what each claims and how all of them could be taken into account, in an independent project, first for deepening and only after for implementing in the app, if it makes sense. The seed-place is a waiting room, not a library: these two pages already hold more than an idea's page should, and the app's streams are bound by L139 and the FullProf tests, where the deepening needs to read, compare and try estimators that are not Rietveld's. The inverse-problems project of 14 October is the precedent: a line of thought on its own site, worked the same way, code only when a proposal is taken up. The design is idea I07; its scaffold was made the same day. From the day Josu names it, new notes on this topic are written there; these two pages keep the record and the pointer, and their log says when something lands in the core.

Steps, amended. L1 adds Spotlight's and BBO-Rietveld's public cases and a TOPAS baseline where available; L4 adds the optimal-transport capture beside continuation; L5's triggers include the derivative release; the stage table searched by Bayesian optimisation is L6, after L5 exists; A6′ is a small core option, requested when the stream starts. Where these steps are worked, here or in I07's project, is Josu's decision.

For Josu to decide

  • Whether steps 0–2 (the bench, variable projection, geodesic acceleration) go to the core stream now, as engine options that keep the default.
  • Whether prior annealing (A4 c) may be offered in place of a hand-written plan, the plan staying available.
  • Where a machine-learned potential would live: in the core (a large dependency and every calculation in the core, L139) or as an extension (I02); and whether the bond-valence energy is enough to start.
  • Whether a result refined with an energy prior is shown in the app beside the unrestrained one, never instead of it (the session's proposal).
  • Whether Explore takes the landscape views (catalogue of minima, the path between two minima) as requests now or after step 4.
  • Whether the ladder (step 5) is written as a methods paper, and with whom (the inverse-problems project of L286).
  • Whether the rule for machine learning (the network proposes, the likelihood decides; never at the end point) is adopted as a standing rule of the project.
  • Whether Bayesian optimisation is taken for what it is good for here, the stage table and the schedules (L6), and refused as a minimiser of the structural parameters, as a standing rule beside the machine-learning one (note of 21 October, §1).
  • Whether the optimal-transport capture (A4d) enters L4 beside continuation, as the first new path method.
  • Whether the Poisson likelihood (A6′) is requested of the core as an engine option now, since L286's likelihood needs it too.
  • Whether Spotlight's and BBO-Rietveld's public cases (GSAS-II's \(\mathrm{Al_{2}O_{3}}\) and \(\mathrm{PbSO_{4}}\) tutorials; \(\mathrm{Y_{2}O_{3}}\)) join the bench, so that the stream's claims are compared with published numbers.

References

What this page cites, each with how it was checked.

  1. P. I. Frazier, *A tutorial on Bayesian optimization*, arXiv 1807.02811 (2018): expensive objectives, fewer than about 20 dimensions, no derivatives; the Gaussian process's cubic cost. arxiv.org/abs/1807.02811 by search, 21 October 2026
  2. B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, N. de Freitas, *Taking the human out of the loop: a review of Bayesian optimization*, Proc. IEEE 104 (2016) 148–175. believed, not re-read
  3. D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, M. Poloczek, *Scalable global optimization via local Bayesian optimization* (TuRBO), NeurIPS 32 (2019). arxiv.org/abs/1910.01739 believed, not re-read
  4. Y. Ozaki, Y. Suzuki, T. Hawai, K. Saito, M. Onishi, K. Ono, *Automated crystal structure analysis based on blackbox optimisation*, npj Comput. Mater. 6 (2020) 75 (BBO-Rietveld: Optuna's TPE over the recipe of a GSAS-II refinement; \(\mathrm{Y_{2}O_{3}}\)). doi.org/10.1038/s41524-020-0330-9 by search, 21 October 2026 (page blocked; KEK's release and the GitHub README read)
  5. BBO-Rietveld on GitHub (quantumbeam/BBO-Rietveld; GSASIIscriptable, Optuna \(\ge 3.5\), Docker image) and the OptunaHub sampler bbo_rietveld. hub.optuna.org/samplers/bbo_rietveld/ read, 21 October 2026
  6. J. Agada, Y. Wang, A. Biswas, *CrystalMO-TuRBO: multi-objective trust-region Bayesian optimization for high-precision joint crystal structure refinement*, arXiv 2609.20592 (September 2026). arxiv.org/abs/2609.20592 abstract, 21 October 2026
  7. J. O. Agada, A. McAninch, H. Day, Y. Tanyu, E. McCombs, S. M. Koohpayeh, B. H. Toby, Y. Wang, A. Biswas, *Physically-constrained autoencoder-assisted Bayesian optimization for refinement of high-dimensional defect-sensitive single crystalline structure*, arXiv 2601.00855 (December 2025). arxiv.org/abs/2601.00855 abstract, 21 October 2026
  8. C. M. Biwer, Z. Feng, D. Finstad, M. McDonnell, M. Knezevic, M. McKerns, D. J. Savage, S. C. Vogel, *Spotlight: efficient automated global optimization in Rietveld analysis of diffraction data*, Sci. Rep. 15 (2025) 8358. doi.org/10.1038/s41598-025-92452-4 read in full (PMC11897340), 21 October 2026
  9. A. Antoniadis, J. Berruyer, A. Filhol, *Maximum-likelihood methods in powder diffraction refinements*, Acta Cryst. A46 (1990) 692–711. by search, 21 October 2026
  10. W. I. F. David, *Powder diffraction: least-squares and beyond*, J. Res. Natl. Inst. Stand. Technol. 109 (2004) 107–123. doi.org/10.6028/jres.109.008 abstract and summary read, 21 October 2026
  11. W. I. F. David, *On the equivalence of the Rietveld method and the correlated integrated intensities method in powder diffraction*, J. Appl. Cryst. 37 (2004) 621–628. doi.org/10.1107/S0021889804013184 by search, 21 October 2026
  12. K. Shankland, *Global Rietveld refinement*, J. Res. Natl. Inst. Stand. Technol. 109 (2004) 143–154 (DASH, about 3,500 trial structures per second). doi.org/10.6028/jres.109.011 summary read, 21 October 2026
  13. E. Prince, W. L. Nicholson, *Influence of individual reflections on the precision of parameter estimates in least squares refinement*, in Structure and Statistics in Crystallography (Adenine, 1985); W. L. Nicholson, E. Prince, J. Buchanan, P. Tucker, *A robust/resistant technique for crystal structure refinement* (1982). believed, not re-read
  14. A. Gagin, I. Levin, *Accounting for unknown systematic errors in Rietveld refinements: a Bayesian statistics approach*, J. Appl. Cryst. 48 (2015) 1201–1211; GSAS_USE (NIST, GitHub AntonGagin/GSAS_USE). github.com/AntonGagin/GSAS_USE by search and NIST's page, 21 October 2026
  15. B. H. Toby, R. B. Von Dreele, *GSAS-II: the genesis of a modern open-source all purpose crystallography software package*, J. Appl. Cryst. 46 (2013) 544–549. doi.org/10.1107/S0021889813003531 believed, not re-read
  16. A. A. Coelho, *A bound constrained conjugate gradient solution method as applied to crystallographic refinement problems*, J. Appl. Cryst. 38 (2005) 455–461; A. A. Coelho, *TOPAS and TOPAS-Academic: an optimization program integrating computer algebra and crystallographic objects written in C++*, J. Appl. Cryst. 51 (2018) 210–218. by search, 21 October 2026 (the 2005 paper through its citation in the 2018 one; its page blocked)
  17. B. H. Toby, *A simple solution to the Rietveld refinement recipe problem*, J. Appl. Cryst. 57 (2024). doi.org/10.1107/S1600576723011032 abstract, 21 October 2026
  18. B. J. Kerman, B. J. Campbell, K. R. Satyavarapu, H. T. Stokes, F. Perselli, J. S. O. Evans, *The superstructure determination of displacive distortions via symmetry-mode analysis*, Acta Cryst. A68 (2012) 222–234. iso.byu.edu/2012%20Kerman.pdf by search, 21 October 2026
  19. N. Segal, A. Subramanian, M. Li, B. K. Miller, R. Gómez-Bombarelli, *The loss landscape of powder X-ray diffraction-based structure optimization is too rough for gradient descent*, arXiv 2512.04036 (2025). arxiv.org/abs/2512.04036 abstract, 21 October 2026
  20. A. de O. Marcelo Junior, A. B. Gomes, E. Pontes de Deus, *AiMat-DRX: a differentiable Rietveld engine with calibrated uncertainty and a declared domain of validity*, Zenodo, version 1.2.0 (September 2026). doi.org/10.5281/zenodo.22679432 record read, 21 October 2026
  21. I. C. Madsen et al., *Outcomes of the IUCr CPD round robin on quantitative phase analysis: samples 1a to 1h*, J. Appl. Cryst. 34 (2001) 409–426; N. V. Y. Scarlett et al., *… samples 2, 3, 4, synthetic bauxite, natural granodiorite and pharmaceuticals*, J. Appl. Cryst. 35 (2002) 383–400. believed, not re-read
  22. Y. Wu, M. Sun, *AutoXRD: autonomous LLM agents and comprehensive evaluation for powder diffraction analysis*, arXiv 2609.00070 (2026). arxiv.org/abs/2609.00070 abstract, 21 October 2026
  23. W. Shin, C. A. Bridges, M. T. McDonnell, R. Ferreira da Silva, *Fantastic scientific agents and how to build them: AgentBuild for Rietveld refinement*, arXiv 2606.12834 (2026). arxiv.org/abs/2606.12834 abstract, 21 October 2026
  24. B. Cao, Q. Zhang, Z. Feng, T. Zhang, J. Huang, L.-T. Weng, T.-Y. Zhang, *AI-driven structure refinement of X-ray diffraction* (WPEM), arXiv 2602.16372 (2026). arxiv.org/abs/2602.16372 abstract, 21 October 2026
  25. Z. Jiang, J. Wang, M. V. Tirrell, J. J. de Pablo, W. Chen, *Parameter estimation for X-ray scattering analysis with Hamiltonian Markov chain Monte Carlo*, J. Synchrotron Rad. 29 (2022). doi.org/10.1107/S1600577522003034 abstract, 21 October 2026
  26. T. Kacprzak, S. Samothrakitis, C. B. Larsen, J. Kopeček, M. Strobl, E. Polatidis, G. Obozinski, *Laue indexing with optimal transport*, arXiv 2404.06478 (2024). arxiv.org/abs/2404.06478 abstract, 21 October 2026
  27. C. K. Cocke, E. Camacho, S. F. Gorske, K. T. Faber, K. Bhattacharya, *Physics-aware global Rietveld refinement for high-energy X-ray diffraction microscopy* (PARA-X), arXiv 2609.13619 (2026). arxiv.org/abs/2609.13619 abstract, 21 October 2026
  28. B. Engquist, B. D. Froese, *Application of the Wasserstein metric to seismic signals*, Commun. Math. Sci. 12 (2014) 979–988; L. Métivier, R. Brossier, Q. Mérigot, E. Oudet, J. Virieux, *An optimal transport approach for seismic tomography*, Inverse Problems 32 (2016) 115008; Y. Yang, B. Engquist, J. Sun, B. F. Hamfeldt, *Application of optimal transport and the quadratic Wasserstein metric to full-waveform inversion*, Geophysics 83 (2018) R43–R62. arxiv.org/abs/1612.05075 by search, 21 October 2026
  29. G. H. Golub, V. Pereyra, *The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate*, SIAM J. Numer. Anal. 10 (1973) 413–432; and *Variable projection algorithms: theoretical insights and a novel approach for problems with large residual*, arXiv 2402.13865 (2024). arxiv.org/abs/2402.13865 by search, 19 October 2026
  30. M. K. Transtrum, B. B. Machta, J. P. Sethna, *Geometry of nonlinear least squares with applications to sloppy models and optimization*, Phys. Rev. E 83 (2011) 036701; M. K. Transtrum, J. P. Sethna, *Improvements to the Levenberg–Marquardt algorithm for nonlinear least-squares minimization*, arXiv 1201.5885 (2012); *Why are nonlinear fits to data so challenging?*, Phys. Rev. Lett. 104 (2010) 060201. arxiv.org/abs/1201.5885 by search, 19 October 2026; Sethna's PDFs (SloppyGeometry, ChallengingFits) read in part
  31. R. N. Gutenkunst, J. J. Waterfall, F. P. Casey, K. S. Brown, C. R. Myers, J. P. Sethna, *Universally sloppy parameter sensitivities in systems biology models*, PLoS Comput. Biol. 3 (2007) e189. believed, not re-read
  32. J. E. Dennis, D. M. Gay, R. E. Welsch, *An adaptive nonlinear least-squares algorithm* (NL2SOL), ACM Trans. Math. Softw. 7 (1981) 348–368. believed, not re-read
  33. A. Blake, A. Zisserman, *Visual Reconstruction* (MIT Press, 1987): graduated non-convexity. believed, not re-read
  34. P. Pulay, G. Fogarasi, *Geometry optimization in redundant internal coordinates*, J. Chem. Phys. 96 (1992) 2856–2860. believed, not re-read
  35. SIESTA's tutorials on structure optimisation (MD.TypeOfRun: CG, Broyden, FIRE) and molecular dynamics. docs.siesta-project.org/projects/siesta/en/5.2/tutorials/basic/structure-optimization/ read, 19 October 2026
  36. *Robust refinement as implemented in TOPAS*, Mater. Sci. Forum 651 (2010) 27; TOPAS forum thread on grs_interaction penalties. www.scientific.net/MSF.651.27 by search, 19 October 2026
  37. D. J. Wales, J. P. K. Doye, *Global optimization by basin-hopping*, J. Phys. Chem. A 101 (1997) 5111–5116; S. Goedecker, *Minima hopping*, J. Chem. Phys. 120 (2004) 9911–9917. believed, not re-read
  38. V. Favre-Nicolin, R. Černý, *FOX, free objects for crystallography*, J. Appl. Cryst. 35 (2002) 734–743; FOX's manual on its global-optimisation algorithms; W. I. F. David, K. Shankland, J. van de Streek, E. Pidcock, W. D. S. Motherwell, J. C. Cole, *DASH*, J. Appl. Cryst. 39 (2006) 910–915; A. A. Coelho, *Whole-profile structure solution from powder diffraction data using simulated annealing*, J. Appl. Cryst. 33 (2000) 899–908. objcryst-fox.readthedocs.io/en/latest/manual-algorithms.html FOX's manual read, 19 October 2026; the papers by search
  39. The Rietveld mailing list, 1998, *Monte Carlo, genetic algorithms (…) for powder structure*: David and Sivia on Pawley intensities with their covariance. www.cristal.org/DU-SDPD/rietveld_l/1998/00000051.htm read, 19 October 2026
  40. A. Gelman, X.-L. Meng, *Simulating normalizing constants: from importance sampling to bridge sampling to path sampling*, Stat. Sci. 13 (1998) 163–185. believed, not re-read
  41. S. Duane, A. D. Kennedy, B. J. Pendleton, D. Roweth, *Hybrid Monte Carlo*, Phys. Lett. B 195 (1987) 216–222; R. M. Neal, *MCMC using Hamiltonian dynamics*, in Handbook of Markov Chain Monte Carlo (2011); M. Girolami, B. Calderhead, *Riemann manifold Langevin and Hamiltonian Monte Carlo methods*, J. R. Stat. Soc. B 73 (2011) 123–214; M. D. Hoffman, A. Gelman, *The No-U-Turn sampler*, J. Mach. Learn. Res. 15 (2014) 1593–1623. believed, not re-read
  42. C. M. Fancher, Z. Han, I. Levin, K. Page, B. J. Reich, R. C. Smith, A. G. Wilson, J. L. Jones, *Use of Bayesian inference in crystallographic structure refinement via full diffraction profile analysis*, Sci. Rep. 6 (2016) 31625. doi.org/10.1038/srep31625 by search, 15 and 19 October 2026
  43. J. P. Lesniewski, D. M. Pearce, I. Levin, *BLAND: a Bayesian method for the analysis of diffraction patterns*, J. Appl. Cryst. 49 (2016) 2201–2209. www.nist.gov/publications/bayesian-method-analysis-diffraction-patterns-using-bland by search, 19 October 2026
  44. J. A. Christen, C. Fox, *Markov chain Monte Carlo using an approximation*, J. Comput. Graph. Stat. 14 (2005) 795–810. believed, not re-read
  45. G. Henkelman, B. P. Uberuaga, H. Jónsson, *A climbing image nudged elastic band method*, J. Chem. Phys. 113 (2000) 9901–9904; O. M. Becker, M. Karplus, *The topology of multidimensional potential energy surfaces*, J. Chem. Phys. 106 (1997) 1495–1517. believed, not re-read
  46. A. S. Anker, J. Gardner, E. Rosset, A. L. Goodwin, V. L. Deringer, *Scatter2xyz*, arXiv 2510.05938 (October 2025). arxiv.org/abs/2510.05938 read, 19 October 2026
  47. J. van de Streek, M. A. Neumann, *Validation of experimental molecular crystal structures with dispersion-corrected density functional theory calculations*, Acta Cryst. B66 (2010) 544–558. believed, not re-read
  48. A. A. Hoser, A. Ø. Madsen, *Dynamic quantum crystallography: lattice-dynamical models refined against diffraction data*, Acta Cryst. A72 (2016) 206–214 and A73 (2017) 102–114. researchprofiles.ku.dk/en/publications/dynamic-quantum-crystallography-lattice-dynamical-models-refined-/ by search, 19 October 2026
  49. Machine-learned interatomic potentials, surveyed: H. Yu et al., *Systematic assessment of various universal machine-learning interatomic potentials*, Mater. Genome Eng. Adv. (2024); Park et al., *Machine learning interatomic potentials for energy materials*, Adv. Energy Mater. (2026). doi.org/10.1002/mgea.58 by search, 19 October 2026
  50. J. Mun, K. Nam, S. Choi, *Automation of Rietveld refinement through machine learning*, J. Appl. Cryst. 59 (2026). journals.iucr.org/j/issues/2026/02/00/yr5164/ by search, 19 October 2026 (page blocked on 21 October)
  51. RADAR-PD, *Automated multiphase identification and refinement in powder diffraction using mismatch-tolerant machine learning*, arXiv 2605.12478 (2026). arxiv.org/abs/2605.12478 read, 19 October 2026
  52. E. A. Riesel et al., *Crystal structure determination from powder diffraction patterns with generative machine learning*, J. Am. Chem. Soc. 146 (2024); *Powder diffraction crystal structure determination using generative models*, Nat. Commun. 16 (2025). doi.org/10.1038/s41467-025-62708-8 by search, 15 and 21 October 2026
  53. *Simulation-based inference: a practical guide*, arXiv 2508.12939 (2025); *Simulation-based inference with neural posterior estimation applied to X-ray spectral fitting, III: deriving exact posteriors with dimension reduction and importance sampling*, arXiv 2512.16709 (2025); M. D. Hoffman et al., *NeuTra-lizing bad geometry in Hamiltonian Monte Carlo using neural transport*, arXiv 1903.03704 (2019). arxiv.org/abs/2508.12939 read in part, 19 October 2026
  54. M. C. Kennedy, A. O'Hagan, *Bayesian calibration of computer models*, J. R. Stat. Soc. B 63 (2001) 425–464. believed, not re-read
  55. M. Raissi, P. Perdikaris, G. E. Karniadakis, *Physics-informed neural networks*, J. Comput. Phys. 378 (2019) 686–707. believed, not re-read
  56. R. J. Hill, L. M. D. Cranswick, *International Union of Crystallography Commission on Powder Diffraction Rietveld refinement round robin, II*, J. Appl. Cryst. 27 (1994) 802–844. believed, not re-read
  57. E. D. Dolan, J. J. Moré, *Benchmarking optimization software with performance profiles*, Math. Program. 91 (2002) 201–213; J. J. Moré, S. M. Wild, *Benchmarking derivative-free optimization algorithms*, SIAM J. Optim. 20 (2009) 172–191. believed, not re-read
  58. C. E. Rasmussen, C. K. I. Williams, *Gaussian Processes for Machine Learning* (MIT Press, 2006), ch. 5. gaussianprocess.org/gpml/ believed, not re-read
  59. codecut.ai, *Bayesian optimization for hyperparameter search* (the primer Josu asked about): a Gaussian process with a Matérn 5/2 kernel, expected improvement, a one-dimensional synthetic function minimised in 13 evaluations. codecut.ai/bayesian-optimization-hyperameter-search/ read, 21 October 2026

The idea's log

  • 2026-10-19 · said

    Said by Josu in the ideas chat: beyond reproducing FullProf and GSAS-II, innovate in the minimisation itself; could algorithms from molecular dynamics, DFT relaxations (SIESTA) or Monte Carlo make the refinement more reliable or faster, with the same results; think further than mode crystallography's extra information. Then: and machine learning, neural networks, physically informed models?

  • 2026-10-19 · discussion

    Assessed with the core's engine and the literature: locally, Gauss-Newton with exact derivatives beats the DFT relaxers and Monte Carlo, so the default stays; the time is lost in six other places, each with a borrowed tool: variable projection, geodesic acceleration, continuation, basin hopping, a temperature ladder with Fisher-metric HMC whose cold end is the Rietveld minimum, and the energy as a Boltzmann prior; machine learning only where the exact likelihood corrects it (amortised starts and proposals, a learned sampler geometry, a misfit model, the potentials). Closest precedents: Scatter2xyz (Anker et al. 2025) and Mun, Nam and Choi (2026). A bench of perturbed starts comes first. L358

  • 2026-10-19 · discussion

    Josu, thinking out loud: the innovations must not interfere; as extensions, or a parallel stream for steps 0-2 kept as options; proved on the known cases; gains measured, also in partial times, with case-dependent intermediate steps; the aim a more robust Rietveld refinement. Worked as the first workshop note: a stream with its own crate (as Publish), the reference engine as every strategy's last stage, the bench with FullProf itself as baseline, four phases with operational definitions, stages with switching rules, six senses of robustness. Status to workshop. L363

  • 2026-10-21 · discussion

    Audited with I05 at Josu's request, with the Bayesian-optimisation primer he sent: Bayesian optimisation belongs where a whole refinement is the black box (BBO-Rietveld 2020, the Agada–Biswas papers of 2025–26, Spotlight 2025), never as the minimiser; twelve precedents the first assessment missed, by section (Antoniadis 1990, David 2004 twice, Gagin–Levin 2015, Coelho 2005, Toby 2024, Kerman 2012, Segal 2025, AiMat-DRX, the agents, WPEM, HMC in scattering, optimal transport in diffraction); three more path methods (an optimal-transport misfit for capture, the Poisson likelihood, a parameter released by its derivative); the references on the page; a project of its own proposed as I07. Second workshop note. L393