The collapsed configuration space

Iñigo's ideas (T01–T11), sorted into one argument and pushed a step further. Where the text goes beyond the brief it says so. First version, 14 October 2026 (L005); to be argued over.

1. The claim, in one paragraph

A system lives in a configuration space: the set of all values of all the parameters of the model that governs it. An experiment does not look at that space; it looks at a function of it — a diffractogram, an extinction spectrum, a reflectance curve. The function is known: that is what "I know the theory" means (T01). But the function is a projection (T04): it sends a space of many dimensions onto a curve, and in doing so it folds whole regions of the configuration space onto the same measured response, or onto responses that differ by less than the noise. Inverting the projection from the measurement alone is then not merely hard. Over the folded regions it is not defined (T05). What makes it possible is always something that is not in the measurement: a different set of coordinates in which the fold is visible, a prior that says which of the folded configurations is plausible, a second experiment that projects along another direction, or knowledge of the system written down once and applied at scale (T07–T10). The rest of this page says what kinds of fold there are, why the symmetry-mode coordinates of crystallography undo one of them, what the same move looks like in two other problems, and what a machine learned on massive, scripted data can and cannot add.

2. Three things the word "underdetermined" covers

The brief uses one word; this development splits it into three, because the three need different tools. The vocabulary is that of the identifiability literature (Raue et al. 2009), of the sloppy-models literature (Gutenkunst et al. 2007; Transtrum et al. 2015) and of the model-discrepancy literature (Kennedy and O'Hagan 2001).

The three are routinely confused in practice, because all three present the same symptom: "the fit is compatible with many solutions". But a null space is removed only by another experiment or an exact constraint; a sloppy valley is shortened by better data in the right place, by a prior, or by a change of coordinates that aligns it with an axis; and model inadequacy is cured by none of these, only by a better model or by an explicit discrepancy term that keeps the inadequacy from being absorbed into the parameters. A programme that does not say which of the three it is facing will apply the wrong remedy.

3. The projection, written down

Let \( \theta \in \mathcal{C} \subset \mathbb{R}^n \) be the configuration — all the parameters of the model — and \( F : \mathcal{C} \to \mathcal{D} \subset \mathbb{R}^m \) the forward map that gives the response the instrument would record. The measurement is \( d = F(\theta^\ast) + \eta \), with noise \( \eta \) of covariance \( \Gamma \). Three objects carry the whole discussion.

• The fibre \( F^{-1}(d) = \{\theta : F(\theta) = d\} \): every configuration that gives exactly the response measured. A structural non-identifiability is a fibre of positive dimension.
• The tube \( \{\theta : (F(\theta) - d)^{\mathsf T}\Gamma^{-1}(F(\theta) - d) \le \chi^2_{\text{crit}}\} \): every configuration whose response is within noise of the measurement. This is what a refinement is really choosing from. Its cross-section is what the covariance matrix describes, locally and in the coordinates used; its length along the sloppy directions is what the covariance matrix hides when only the diagonal is read.
• The model manifold \( F(\mathcal{C}) \subset \mathcal{D} \): the set of all responses the model can produce. Transtrum et al. call it that and show it is typically a thin, curved ribbon with boundaries. Model inadequacy is the statement that \( d - \eta \) is not on it.

Locally, the structure of the tube is read from the weighted Jacobian \( J = \Gamma^{-1/2}\,\partial F/\partial\theta \) at the solution. Its singular value decomposition \( J = U\Sigma V^{\mathsf T} \) gives the stiffness spectrum \( \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_n \): a singular value of zero is a structural non-identifiability, a singular value much smaller than one is a practical one, and the columns of \( V \) are the directions in configuration space those singular values belong to. The point for what follows: the right singular vectors of \( J \) are the coordinates the experiment actually sees. They are combinations of the model's parameters, not the parameters themselves, and they change from one solution to another. A refinement program computes \( J \) at every cycle; it reports the correlation matrix, which almost nobody reads, and the diagonal of \( (J^{\mathsf T}J)^{-1} \), which everybody reads and which answers the wrong question.

The counting that makes the brief's "too many parameters" (T02) precise is also here. A diffractogram has thousands of points, but its effective dimension is far smaller: the number of resolved peaks, each carrying a position, an integrated intensity and a width, plus a smooth background of a few parameters. Everything that is not in that count is noise or redundancy. The configuration space has as many dimensions as the model has parameters: coordinates, occupancies, displacement parameters, cell, profile, corrections. The experiment is underdetermined whenever the second count exceeds the first in the directions the Jacobian can see, and that qualification is the whole subject.

4. Why symmetry-adapted modes work, stated so that it can be carried elsewhere

T06 and T07 are Iñigo's. The reading of them as a change of basis chosen for the shape of the prior is this development's; it is also, in other words, what Perez-Mato, Orobengoa and Aroyo (2010) describe as the purpose of the symmetry-mode parameterisation: detection of hidden structural correlations, identification of fundamental and marginal degrees of freedom, reduction of the effective structural parameters.

A distorted structure of pseudo-symmetric type is written as a parent structure of higher symmetry plus a displacement field \( \mathbf{u} \) of its atoms. The mode decomposition writes that same field in a different orthonormal basis: \( \mathbf{u} = \sum_k A_k\,\boldsymbol{\epsilon}_k \), where each \( \boldsymbol{\epsilon}_k \) is a displacement pattern that transforms as an irreducible representation of the parent space group and \( A_k \) is its amplitude (Perez-Mato et al. 2010; Orobengoa et al. 2009; Campbell et al. 2006). Nothing is lost: the configuration space is the same space, the number of coordinates the same number, the map between coordinates and amplitudes linear and invertible. So the question is why the same space, in these coordinates, is explorable (T07) when in atomic coordinates it is not. Three reasons, and each of them is a general principle in disguise.

1. The prior is sparse in this basis and not in the other. Landau's picture of a displacive transition says the distortion is dominated by one, sometimes two, primary modes, with the secondary modes small and induced. In atomic coordinates that knowledge cannot even be written down: every atom moves, by a little, and nothing says which little is which. In mode coordinates it reads "most amplitudes are near zero", which is a prior a refinement can use: release the primary amplitudes, freeze the marginal ones, and the configuration space is cut to its physical dimension before a single least-squares cycle. The general principle: choose the coordinates in which what the physics expects is simple to state.
2. The correlations the data cannot break become single coordinates. When two atoms move together by symmetry, a powder pattern sees their joint displacement and not the difference; in atomic coordinates that is a sloppy valley between two parameters, in mode coordinates it is one amplitude that is well determined and one that is small and may be fixed. The general principle: choose the coordinates in which the near-null space of the Jacobian is aligned with the axes, so that the practical non-identifiability is seen as such instead of being smeared over parameters that look independent.
3. The hierarchy of amplitudes is a measurement of which degrees of freedom matter. A refinement in modes returns not only a structure but an ordered list: this amplitude large and determined, that one small and undetermined. That list is the dimension reduction Perez-Mato et al. speak of, obtained from the data, with a physical meaning attached to each retained direction. The general principle: the output of the analysis should be the ranking of directions, not only the point.

And one thing the modes do not do, which must be said because the programme depends on it: the parent structure is not in the data. It is a hypothesis — a group–supergroup relation the crystallographer brings from chemistry, from a phase diagram, from a related compound. The symmetry search of T07 searches within that hypothesis. So the external tool has two parts: a piece of exact mathematics that anyone can script (the decomposition, given the parent), and a piece of judgement that cannot be scripted away (the choice of parent). The programme must record the second as an input with its provenance, not hide it in the first.

5. The same move in the other two cases

If the principle is "find the basis in which the prior is simple and the null space is explicit, then bring in what the data cannot supply", it should name its analogues in problems that have no space groups. The Cases pages work the three side by side; this is the summary.

• Drude parameters from an extinction spectrum (Daun et al. 2019). The configuration is \( (\omega_p, \tau) \), a scale factor that absorbs the particle number density, a size distribution and a temperature. The projection is the extinction cross-section integrated along a line of sight and over the sizes; in the visible, where the spectrum is a smooth decay, it sees mainly one combination of \( \omega_p \) and \( \tau \). The external tools the paper uses are other experiments (ellipsometry; bulk density and resistivity through \( \omega_p^2 = N_e e^2/\epsilon_0 m_e \) and \( \tau = m_e\sigma_{\text{dc}}/N_e e^2 \)), a better source that reaches the ultraviolet where the surface-plasmon resonance sits, and an explicit model-error term for the approximate Mie theory after Kaipio and Somersalo. The "sparse basis" here is not a symmetry; it is the choice of which spectral region to measure, and the paper finds it by trying two lamps. The programme proposes to compute it instead (P6): the Fisher information per wavelength band says in advance which measurement would shorten the valley.
• Dielectric functions from infrared reflectance and emittance (the FOCUS–difes case). The configuration is a list of oscillators, three or four parameters each, plus \( \epsilon_\infty \); the projection is the Fresnel reflectance \( R = |\tilde r|^2 \) or the opaque emittance \( 1 - R \), measured on a finite range. Two folds are known to everyone who has done it: reflectance alone gives the modulus of \( \tilde r \) and not its phase, and a smooth \( \epsilon(\omega) \) can be written as a sum of overlapping Lorentzians in many ways. The exact constraints that play the part of symmetry are causality — the Kramers–Kronig relations that fix the phase from the modulus over an infinite range — and the sum rules; the finite range is what makes them an approximation that needs anchoring (Kuzmenko 2005). The "sparse prior" is continuity across a series: the parameters of a mode measured at twenty temperatures must move smoothly, and a fit that satisfies each spectrum but jumps between neighbours has found a valley, not a structure. difes already carries the expert's steps as recipes, text files of bench verbs; that is T10 before the fact.

Across the three the pattern holds: a known forward map; a projection that folds; a point returned by a fit that looks like an answer; and a way out that always comes from outside the measurement — exact mathematics (symmetry, causality), another projection (another radiation, another lamp, another temperature), or a prior (sparsity in the right basis, smoothness along a series).

6. Where the expert's knowledge goes

T08–T10 are Iñigo's. The description of what the output of the scripts must be, and of what the learning can and cannot do, is this development's.

Today the knowledge is applied by hand, once per data set, and lost. The expert looks at one pattern, decides the parent, decides which parameters to release in which order, sets the restraints, judges the residual, and moves on; what remains is a structure in a paper and nothing of the path that led there. Iñigo's proposal (T09, T10) is to move the knowledge one level up: from the hands to the scripts that treat the data. The symmetry analysis, the release strategy, the constraints, the sanity checks — written once, applied to everything. Three consequences follow that the brief does not spell out.

1. What the script must keep is the tube, not the point. If every treatment stores only the refined structure, the corpus repeats the mistake of the hand analysis at scale. Each treatment should keep the full result: the solution, the Jacobian's singular spectrum and vectors at it, the alternative minima found from other starting points, the mode amplitudes with their uncertainties, the residual. That is the data over which anything can be learned about identifiability; the point alone teaches nothing about it.
2. The knowledge in the script is also the knowledge in the training set. The published successes of machine learning on powder diffraction are built exactly this way: Park et al. (2017) classify crystal system, extinction group and space group from about 150 000 simulated patterns; Lee et al. (2020) identify phases in mixtures from 1.8 million synthetic patterns combinatorially mixed from 170 compounds; Szymanski et al. (2021) train on simulated spectra augmented with physics-informed perturbations and return probabilities from an ensemble. In every case the expertise is in the generator — which structures, which instrument model, which artefacts — and not in the labelling of real data one by one. T10 is already how the field's best results are produced; the brief names the principle the practice follows.
3. The symmetry is what keeps the synthetic data physical. A generator that samples atomic coordinates at random produces structures that do not exist; one that samples mode amplitudes around a parent, with the primary ones large and the secondary ones small, produces the structures a transition actually makes. The corpus of T09 is credible only to the degree that the script of T10 encodes constraints of this kind. The two theses are one.

7. What a machine learned on that corpus can and cannot do

The limit first, because it decides the form of everything else. Learning cannot recreate what the projection destroyed. If two configurations give the same response, no training set breaks the tie; a network asked for a point will return one of them, or an average of them, with a confidence that reflects nothing. For an underdetermined problem the only honest output is a distribution over the fibre, and a method that returns a point should be rejected on that ground alone. Within that limit, five things are possible, and each is a proposal of the programme.

• Amortised inference: the whole tube, at the cost of a lookup. Simulation-based inference (Cranmer, Brehmer and Louppe 2020) trains a network on pairs (configuration, simulated response) to return, for any new response, the posterior over configurations. A refinement gives one point and a local covariance; this gives the fibre with its curvature and its disconnected branches, for every pattern of a measurement campaign in the time a single refinement takes. It is the formal version of "explore the configuration space" in T07, done once for a family of problems instead of once per data set.
• Priors learned from what exists. The databases of known structures, of measured dielectric functions, of published Drude parameters are the distribution of configurations nature actually produces. A prior estimated from them is "chemical sense" made explicit and auditable, and it is a legitimate external tool in the sense of T05 precisely because it is not in the measurement.
• Surrogates for the forward map that remove a trade-off. Daun et al. use an approximation to Mie theory because the exact Riccati–Bessel sums are too slow for a million MCMC steps, and then pay for it with a model-error term. A forward model learned once from the exact theory is both fast and exact to within a known tolerance, and takes that term away.
• Model inadequacy detected at scale. One structured residual is a bad fit; the same structured residual across a thousand treatments of a family is a wrong model. A corpus is the only place where that distinction can be made, and it is where the third kind of underdetermination becomes visible instead of being absorbed into the parameters.
• Experimental design: which measurement cuts the fibre. Given the tube of the present data, the expected information gain of a candidate measurement — a second radiation, an ultraviolet source, a lower temperature — is computable from the Jacobian. The two lamps of Daun et al. were a design experiment done by hand; the corpus can rank the candidates before anyone goes to the instrument.

8. Objections, taken seriously

• "This is Bayesian inference with another name." In form, yes: the tube is a posterior, the prior is a prior. What the brief adds is not the inference but where the knowledge enters: in the basis (modes, causality) and in the scripts that generate and treat the corpus, rather than in the hands at each fit; and the insistence that the result is the ranking of directions and the region, not the point. Daun et al. do the inference by hand on one spectrum; the proposal is the same inference scripted over thousands, with the basis chosen first.
• "Networks trained on synthetic data do not transfer." The known failure, and the reason point 3 of §6 matters. Two answers. The generator is where the expertise lives, so the quality of transfer is a measure of the quality of the script, which is the right thing to be measuring. And the primary targets proposed here — which directions of configuration space a given measurement determines — are properties of the forward map and the instrument, not of the sample; they transfer by construction, because they were never about the sample.
• "Mode crystallography already does this." For displacive distortions, with a chosen parent, one structure at a time: yes, and that is why it is the model case. What is not done is the generalisation of the move to problems without space groups, and its automation over a corpus with the tube kept.
• "Someone still chooses the parent." Yes. That is the irreducible expert input, and the programme asks that it be recorded as an input with its provenance rather than hidden in a script that appears to need none. The same holds for the number of oscillators in an infrared fit and for the choice of Drude as the dielectric model: these are hypotheses, and the corpus should carry them as fields, so that the learning can tell which of them the data could have contradicted.

9. The words, fixed

• Configuration space \( \mathcal{C} \): all values of all parameters of the model. Data space \( \mathcal{D} \): all responses the instrument can record. Forward map \( F \): the theory, from the one to the other. Projection: the brief's word for \( F \) when it folds.
• Fibre: the configurations that give exactly a response. Tube: those that give it within noise. Model manifold: all responses the model can produce.
• Structural non-identifiability: a fibre of positive dimension. Practical non-identifiability: a tube long in some direction at the instrument's noise. Sloppiness: the typical case of the latter, with singular values spread over decades. Model inadequacy: the truth is off the manifold.
• External tool (T05): anything not in the measurement that cuts the tube — exact constraints, another projection, a prior, a basis in which the prior is simple.
• Amortised inference: the map from response to posterior learned once from simulations and applied to any measurement.