diff ref struct Rietveld refinement for the Mac · powder diffraction with X-rays, synchrotron and neutrons · started October 2026

Maximum entropy: what became of it, and what of it belongs in the inference

Maximum entropy: what became of it, and what of it belongs in the inference workshop

I05 · an idea of the seed-place, on its way to the core.

Said
2026-10-16, Josu
Status
workshop
Home
core (where it migrates)
Needs
none

The idea

Some twenty-five years ago the maximum entropy method was a promising approach to solving and refining crystal structures. Did it succeed, how far was it adopted or abandoned, was it really for refinement; could it be resurrected, or derived into something like the Bayesian inference and the exploration of configuration space?

Assessment

What it was, and what became of it, checked by search on 16 October 2026. The maximum entropy method (MEM) entered crystallography with Collins (1982, Nature): reconstruct the density in the cell as the most uniform one (maximum entropy) that reproduces the measured structure factors within their errors. Sakata and Sato (1990, Acta Cryst. A46) gave the algorithm the field used, and Takata, Sakata and Nishibori made it a method for powders: the MEM/Rietveld cycle, where a Rietveld fit partitions the overlapped intensities, MEM draws the density from them, and the density guides the next model (Takata 2008, Acta Cryst. A64, a review). Its public success was the Y atom seen inside the C\(_{82}\) cage from synchrotron powder data (Takata et al. 1995, Nature 377, 46). It did not die: it lives as a tool after the refinement, in three programs still maintained or distributed, Dysnomia (Momma, Ikeda, Belik and Izumi 2013; version 1.0 of 2018, the successor of PRIMA, with RIETAN-FP and VESTA), BayMEM (van Smaalen's group, Bayreuth, version 2.01 of 2010, with its superspace extension, van Smaalen, Palatinus and Schneider 2003, Acta Cryst. A59, 459) and Jana2020's interface to it. It was never the refinement engine: the structural parameters come from least squares; MEM gives the picture of the density that the parametric model leaves unexplained.

  • Where it was abandoned. As a method of solving structures from powders, the maximum-entropy and likelihood route of Bricogne and Gilmore (MICE, 1990s) was overtaken by charge flipping (Oszlányi and Sütő 2004; Superflip, Palatinus and Chapuis 2007), which needs neither atomicity nor symmetry and solves modulated structures and quasicrystals; the field moved. Bricogne's wider "Bayesian programme" for structure determination (his 1997 CCP4 lecture) became BUSTER, which is today a macromolecular refinement system (Global Phasing), the one place where that programme reached practice.
  • Where it was discredited, in part. The non-nuclear maxima MEM showed in the Si–Si and Be–Be bonds were shown to be artefacts of the method with a flat prior (de Vries, Briels and Feil 1996, Phys. Rev. Lett. 77, 1719): MEM applied to structure factors computed from a known density produced maxima the density did not have. The remedy was a non-uniform prior (a procrystal density) and better constraints: the generalized F constraint of Palatinus and van Smaalen (2002, Acta Cryst. A58, 559) and the weighting of the low-order reflections that Dysnomia's authors still discuss. The lesson is the one L286 drew for the credible regions: the result is only as honest as the prior and the error model.
  • Where it is alive in magnetism. Papoular and Gillon (1990, Europhys. Lett.) reconstruct the magnetisation density from polarised-neutron flipping ratios by MEM, and the method is still the standard for that ill-posed inverse problem; it is a tool of single crystals, not powders.

What it is, said in L286's language. MEM is a Bayesian estimate: the maximum of a posterior whose prior is the entropy of the density relative to a reference and whose likelihood is the \(\chi^2\) of the structure factors. It lives in density space, with a few hundred thousand voxels as unknowns, where L286's inference lives in parameter space, with tens of structural parameters. The two are not rivals; they are the two halves of one model. The parametric part says where the atoms are, with symmetry and a region for each parameter; the density part says what the atoms do not explain: disorder, anharmonicity, a guest in a cage, the magnetisation's shape. Dysnomia's "MEM-based pattern fitting" is already the loop between them, run by hand.

What could be derived from it, for this project. Three things, in order of worth.

  • The misfit as density, in the configuration-space work. Explore shows the residual pattern; MEM would show the residual density, the model-free image of what the parametric model leaves, computed from the partitioned intensities the core already writes (I obs partitioned by the model, L100). That is the MEM/ Rietveld cycle inside the app, with the core doing the entropy maximisation and the app drawing the map.
  • The prior for the inference. L286 needs a prior over the amplitudes and the cell and gets it from the family (I04 from MAGNDATA). Where no family exists, the entropic prior relative to the parent structure's density is the principled "stay as close to the parent as the data allow": it is the Bayesian form of "start near the parent", and it penalises a distortion by the information it adds, not by a tolerance.
  • Not a resurrection as a refinement engine. The parametric model, with its symmetry, its restraints and its dictionary, is what makes a refinement a statement about a structure; MEM has no atoms in it. Replacing least squares by entropy maximisation would give a density, not a structure, and the field's thirty years say so.

Cost. The entropy maximisation over a grid is a few hundred lines of Rust with an FFT (the Sakata–Sato iteration or a conjugate-gradient Cambridge-type algorithm), tested against Dysnomia's and BayMEM's published densities; the hard part is the error model of the partitioned intensities, which is the same hard part as L286's likelihood. It is a module of the core, drs mem, that Explore and Publish draw.

Proposal

  • Step 1. A note with the evidence above, kept on this page, so that the question "what became of MEM" is answered once. Done.
  • Step 2. When Explore's residual tools are in place: drs mem on the core's partitioned intensities, against a Dysnomia density of the same data (Dysnomia is free of charge and runs on the Mac); the de Vries test as the core's test: structure factors computed from a known density must not produce maxima the density lacks, or the test says how large they are.
  • Step 3. The entropic prior in L286's inference, compared on the toy against the family prior of I04: which calibrates.

The workshop

The idea worked on, note by note, dated; the latest last.

The method itself, and a refinement built on it

2026-10-16

Josu's second question, taken apart: not MEM as it was used, but the method, its basis, and whether a refinement can be built on it with the inference of L286, or coupled to it. The answer below has three parts: what the method is when said exactly; why it never became a refinement and what one move makes it one; and a proposal with three uses, in increasing novelty, each with the test that would decide it.

1. The basis: entropy counts arrangements. A density on \(N\) voxels is a positive additive distribution: it can be thought of as a large number of equal quanta of scattering matter thrown into the voxels. The number of ways \(W\) to throw them so that voxel \(i\) receives \(f_i\) is a multinomial, and \(\ln W\) is, to within a constant, \(S(f \mid m) = \sum_i \left[ f_i - m_i - f_i \ln (f_i/m_i) \right]\), where \(m\) is the default: the density the map goes to when the data say nothing about a voxel. Skilling (1988) showed this is the only form a consistent "preference among maps" can take, from four requirements (subset independence, coordinate invariance, system independence, scaling); Jaynes (1957) had made it the rule of inference, Gull and Daniell (1978) made images with it, Collins (1982) made the first crystal density. The Bayesian reading (Skilling 1989, Gull 1989, both checked by search on 16 October 2026) is the one that matters here: \(\exp[\alpha S(f \mid m)]\) is the only consistent prior over positive additive maps; the posterior is \(\exp[\alpha S - \chi^2/2]\); the MEM map is its maximum; and \(\alpha\), the prior's strength, is itself inferred from the evidence \(P(d \mid \alpha)\). At that optimum \(-2\alpha S\) equals the number of measurements the data determine beyond the prior (Gull's "good" measurements). So MEM, said exactly, is a maximum-posterior estimate in density space, with a prior whose weight the data set. Three properties follow that the proposal uses: the posterior in \(f\) is strictly concave for given phases, so it has one maximum and no local ones; the gradient of \(S\) is \(-\ln(f_i/m_i)\), a logarithmic pull of every voxel toward the default, which is why the map has no series-termination ripples; and the map is a probability object, so it has an uncertainty, which the field almost never reported.

Where crystallography departed from the basis. Sakata and Sato (1990) stop at the constraint \(C = 1\), one \(\chi^2\) unit per reflection, instead of the evidence-optimal \(\alpha\): a stopping rule, not an inference, and one that under- or over-fits by the accident of the error estimates. The default was flat, \(m_i = \) const: the prior that put the maxima in the Si–Si bond (de Vries, Briels and Feil 1996); the Bayreuth school's procrystal prior and generalized F constraint (Palatinus and van Smaalen 2002) are the return to a proper \(m\) and a proper likelihood. And the phases came from the model, so the map was conditioned on the model and biased toward it; the REMEDY cycles of MEM-based pattern fitting (Izumi, Dysnomia) alternate fit and map to reduce the bias by hand. Gilmore's review (1996, Acta Cryst. A52, 561) said most of this at the time. None of it is a fault of the method; all of it is a departure from its Bayesian form, and the proposal below is first of all a return to that form.

2. Why it was never a refinement, and the one move that makes it one. A refinement is a statement about parameters \(\theta\): positions, occupancies, displacements, a cell, under a symmetry, named in the dictionary. MEM's unknowns are voxels; it has no \(\theta\), so it could show but never state. The move: make the parametric model the default. Write the total density as \(f\) and set \(m = \rho(\theta)\), the density the model predicts. Then \(S(f \mid \rho(\theta))\) measures exactly the information the data carry beyond the model: zero when the model explains everything, negative in proportion to what it misses. Bricogne's school made a cousin of this move for proteins: the atoms not yet modelled are a maximum-entropy distribution of random scatterers, and the modelled part is refined against a likelihood that knows the missing part's variance (Roversi, Blanc, Vonrhein, Evans and Bricogne 2000, Acta Cryst. D56; BUSTER). For a powder refinement with a complete model it has not been done, as far as search shows.

3. The proposal: one posterior over the parameters and the density. \(p(\theta, f, \alpha \mid y) \propto L(y \mid \theta, f)\; \exp[\alpha S(f \mid \rho(\theta))]\; \pi(\theta)\; \pi(\alpha)\), where \(y\) is the pattern, not a list of partitioned intensities: the structure factors enter the pattern model from \(f\) by a Fourier transform, the profile, background, scale and cell stay in \(\theta\), and overlapping reflections need no partition. That collapses MEM/Rietveld's two programs and the REMEDY loop into one posterior. Three things can be done with it.

  • (a) Refinement with the model as default. Maximise over \(\theta\) with \(f\) integrated out. The integral is tractable: the Hessian of \(\alpha S\) in \(f\) is diagonal, \(\alpha / f_i\), so Laplace's approximation costs Fourier transforms. The result is an effective likelihood for \(\theta\): the \(\chi^2\) of the model plus the entropy deficit, the information the data demanded beyond it. Its gradient in \(\theta\) pulls the model toward where the residual density is, through \(\partial \rho / \partial \theta\), using the whole map and not the derivative of \(\chi^2\) alone. This is a different estimator from least squares, and the difference is the point: where the model is wrong (a missing guest atom, a disorder, an anharmonic site), least squares bends \(\theta\) to compensate, the displacement parameters most of all; here \(f\) absorbs the misfit and \(\theta\) is left to describe what it can. The test that decides it is synthetic: a structure refined by both estimators from data with a deliberate fault in the model, and the bias of each \(\theta\) measured.
  • (b) The deficit as the agreement figure, and model selection. At the evidence-optimal \(\alpha\), \(D = -2\alpha S\) counts the measurements the model does not explain: a dimensionless, information-theoretic misfit that separates noise from model error, which \(R_{wp}\) and \(\chi^2\) cannot. Reported beside \(\chi^2\) in the results file, it says "the data contain \(D\) things this model does not". Between two models, the evidence with \(f\) integrated out prefers the one whose default leaves less for \(f\): a subgroup comparison in L286's sense, with the entropy as the Occam term.
  • (c) The distortion as a map, projected onto the modes. Take the parent structure as the default, \(m = \rho(\theta_{\text{parent}})\), and compute \(f\) in the lowest candidate symmetry. Then \(f - m\) is the distortion, model-free, read from the data. Project it onto the symmetry-adapted displacement fields of each irrep, which the core computes without tables (L263): the projection gives a first amplitude for every irrep at once, with its sign, before any parametric refinement. This meets L286's first obstacle head-on: a least-squares refinement cannot start at the parent, because the superstructure intensities have zero derivative there and the matrix is singular (Explore §6.2); the entropic map starts there by construction, since the entropy relative to the parent is maximal at the parent and only the data move it. The ranking of irreps by projected amplitude, each with the uncertainty of the posterior over \(f\), is the first evidence ranking, and the inference of L286 then refines the top candidates in parameter space. For I04 the same projection on the two-channel magnetisation map (positive and negative densities, needed also for nuclear maps with negative scattering lengths; believed, not re-read) gives the magnetic amplitudes.

4. Where it can fail, and the tests. The default's own smearing: if \(\rho(\theta)\) carries the displacement parameters, a sharper or broader \(f\) says something about them, and the estimator must not let \(f\) relearn the thermal motion the model should state; the synthetic test of (a) decides where the line goes. Identifiability: with \(\alpha\) free, \(f\) could absorb everything; the evidence fixes \(\alpha\), and the calibration test of L286 (a 90 % region holds the truth 90 % of the time) is the acceptance test, here over \(\theta\) and over the map. Powder resolution: the superstructure reflections of (c) are weak and overlapped, so the projected amplitudes come with wide posteriors at first; that is the honest answer and it still ranks. The de Vries test is the core's test for the map itself: structure factors computed from a known density must give back no feature the density lacks beyond the stated uncertainty. Cost: grids of \(64^3\) to \(128^3\), the pattern's derivative with respect to every voxel by the adjoint transform, all in the core (L139); sampling \(f\) when Laplace is not enough is natural in logarithmic coordinates, where the entropy is quadratic-like and Hamiltonian Monte Carlo moves well. The author of the method (Skilling) is the author of nested sampling, the tool L286 names for the evidence.

5. Steps, replacing the proposal above. Each one a test before the next.

  • Step 1: drs mem in its Bayesian form: the map from structure factors with the model's phases, a non-flat default, \(\alpha\) by the evidence; against Dysnomia's map of the same data; the de Vries test.
  • Step 2: the pattern likelihood: the same map from the pattern, with no partition; against MEM-based pattern fitting on one of FullProf's runs.
  • Step 3: estimator (a) against least squares on synthetic data with a deliberate fault; the bias of each parameter measured. A finding either way.
  • Step 4: (c) on case A of the mode material (L089, L263): the parent as default, the projection on its irreps, against the amplitudes FullProf and AMPLIMODES give.
  • Step 5: the deficit \(D\) in the results file and in Explore, beside \(\chi^2\).
  • Step 6: the magnetic channel, with I04.

For Josu to decide

  • Whether the entropic refinement (use a) is pursued as a research line beside the least-squares engine: it is a paper before it is a feature, and the engine of version 1 stays least squares either way.
  • Whether the distortion map projected on the irreps (use c) becomes the starting step of L286's toy, in place of "start near the parent".
  • Whether the residual density as a view of Explore is wanted now, as the first visible piece.
  • The name of the deficit \(D\) in the results file and the dictionary.

References

What this page cites, each with how it was checked.

  1. M. W. Collins, *Entropy maximization on crystal structure factors*, Nature 298 (1982) 49–51. doi.org/10.1038/298049a0 by search, 16 October 2026
  2. M. Sakata, M. Sato, *Accurate structure analysis by the maximum-entropy method*, Acta Cryst. A46 (1990) 263–270. by search, 16 October 2026
  3. M. Takata, B. Umeda, E. Nishibori, M. Sakata, Y. Saito, M. Ohno, H. Shinohara, *Confirmation by X-ray diffraction of the endohedral nature of the metallofullerene Y@C82*, Nature 377 (1995) 46–49. doi.org/10.1038/377046a0 by search, 16 October 2026
  4. M. Takata, *The MEM/Rietveld method with nano-applications: accurate charge-density studies of nano-structured materials by synchrotron-radiation powder diffraction*, Acta Cryst. A64 (2008) 232–245. by search, 16 October 2026
  5. K. Momma, T. Ikeda, A. A. Belik, F. Izumi, *Dysnomia, a computer program for maximum-entropy method (MEM) analysis and its performance in the MEM-based pattern fitting*, Powder Diffr. 28 (2013) 184–193; version 1.0 of 2018. jp-minerals.org/dysnomia/en/ by search, 16 October 2026
  6. S. van Smaalen, L. Palatinus, M. Schneider, *The maximum-entropy method in superspace*, Acta Cryst. A59 (2003) 459–469 (BayMEM, version 2.01 of 2010). www.crystal.uni-bayreuth.de/en/baymem/ by search, 16 October 2026
  7. L. Palatinus, S. van Smaalen, *The generalized F constraint in the maximum-entropy method: a study on simulated data*, Acta Cryst. A58 (2002) 559–567. by search, 16 October 2026
  8. R. Y. de Vries, W. J. Briels, D. Feil, *Critical analysis of non-nuclear electron-density maxima and the maximum entropy method*, Phys. Rev. Lett. 77 (1996) 1719–1722. doi.org/10.1103/PhysRevLett.77.1719 by search, 16 October 2026
  9. C. J. Gilmore, *Maximum entropy and Bayesian statistics in crystallography: a review of practical applications*, Acta Cryst. A52 (1996) 561–589. by search, 16 October 2026
  10. G. Bricogne, *Bayesian statistical viewpoint on structure determination: basic concepts and examples*, Methods Enzymol. 276 (1997) 361–423; and the CCP4 study-weekend lecture of 1997. by search, 16 October 2026; the volume and pages believed
  11. P. Roversi, E. Blanc, C. Vonrhein, G. Evans, G. Bricogne, *Modelling prior distributions of atoms for macromolecular refinement and completion*, Acta Cryst. D56 (2000) 1316–1323 (BUSTER). by search, 16 October 2026
  12. G. Oszlányi, A. Sütő, *Ab initio structure solution by charge flipping*, Acta Cryst. A60 (2004) 134–141; L. Palatinus, G. Chapuis, *SUPERFLIP*, J. Appl. Cryst. 40 (2007) 786–790. by search, 16 October 2026
  13. R. J. Papoular, B. Gillon, *Maximum entropy reconstruction of spin density maps in crystals from polarized neutron diffraction data*, Europhys. Lett. 13 (1990) 429–434. by search, 16 October 2026
  14. E. T. Jaynes, *Information theory and statistical mechanics*, Phys. Rev. 106 (1957) 620–630; S. F. Gull, G. J. Daniell, *Image reconstruction from incomplete and noisy data*, Nature 272 (1978) 686–690. believed, not re-read
  15. J. Skilling, *The axioms of maximum entropy*, in Maximum-Entropy and Bayesian Methods in Science and Engineering, vol. 1 (Kluwer, 1988) 173–187. by search, 16 October 2026
  16. J. Skilling, *Classic maximum entropy*, and S. F. Gull, *Developments in maximum entropy data analysis*, both in Maximum Entropy and Bayesian Methods (Cambridge 1988; Kluwer, 1989) 45–52 and 53–71: the entropic prior, the posterior exp(αS − \(\chi^2\)/2), α by the evidence, the 'good' measurements. by search, 16 October 2026
  17. J. Skilling, *Nested sampling for general Bayesian computation*, Bayesian Analysis 1 (2006) 833–859. doi.org/10.1214/06-BA127 believed, not re-read
  18. C. E. Rasmussen, C. K. I. Williams, *Gaussian Processes for Machine Learning* (MIT Press, 2006), ch. 5: hyperparameters by the marginal likelihood, the same move as α by the evidence. gaussianprocess.org/gpml/ believed, not re-read; added 21 October 2026

The idea's log

  • 2026-10-16 · said

    Said by Josu in the ideas chat: what became of the maximum entropy method of twenty-five years ago; was it for refinement; could it be resurrected or derived into the Bayesian inference and the configuration-space exploration.

  • 2026-10-16 · discussion

    Traced by search: alive as a density tool after refinement (Dysnomia, BayMEM, Jana2020), displaced by charge flipping for structure solution, its artefacts traced to the flat prior; never a refinement engine. Derivable as the residual density in Explore and as the entropic prior of the inference; not as the engine. L298

  • 2026-10-16 · work

    Josu asked for the method itself: its basis analysed (entropy as counting, the entropic prior, alpha by evidence, where the field departed from it) and a refinement coupled to the inference proposed: one posterior over parameters and density with the model as the default; the entropy deficit as agreement figure; the distortion map projected on the irreps as the start of L286's inference. Status to workshop. L301

  • 2026-10-21 · discussion

    Audited with I06 (its workshop note of 21 October, §4): nothing found contradicts it; the entropic refinement still has no precedent for powders; WPEM (2026) is the nearest different estimator to compare with. Its references, until now only in the log's sources, are on this page with how each was checked. Josu proposes a project of its own for the topic (I07); this page keeps the record and points there. L393