Data operations: the settings

What each operation of the Data module does to a scan and what each of its settings means (app request 16, L088): the unit, the range the core accepts, the default, how a value is proposed, what raising or lowering it does, and the definition it follows. The app's dialogs, drs data settings and this page are generated from the same file, as the parameter dictionary is.

10 operations and 2 analyses of the Data module, each setting with its unit, meaning, the range the core accepts, its default, how a value is proposed, what raising and lowering it does, and the definition it follows. Operations change the scan the refinement sees and are kept in the control file; analyses return results and leave the scan as it is. Generated from data/operations.yml; do not edit by hand.

Operations

Mask

mask

Points that stay in the scan but carry no weight in the refinement: a range of x, or a list of points.

σ: \(\sigma\) = ∞ at the masked points (no weight); the others unchanged.

Definition: the core's own (R21, L066); test data_operations

`from` From	The first x of the range, included. Unit: the scan's x (° 2\(\theta\), Å for d, Å⁻¹ for Q). Range: any number; with to, from \(\le\) x \(\le\) to. Default: required (unless points is given). Proposed: the left end of the range selected in the plot.
`to` To	The last x of the range, included. Unit: the scan's x. Range: any number. Default: required (unless points is given). Proposed: the right end of the range selected in the plot.
`points` Points	A list of points, each by its x, matched to \(10^{-9}\) of x. Unit: the scan's x. Range: a list of numbers; an x matched to no point does nothing. Default: none (from and to are used). Proposed: the points picked in the plot.

Delete

delete

Points removed from the scan: a range of x, or a list of points.

σ: the points that stay are unchanged.

Definition: the core's own (R21, L066); test data_operations

`from` From	The first x of the range, included. Unit: the scan's x. Range: any number. Default: required (unless points is given). Proposed: the left end of the range selected in the plot.
`to` To	The last x of the range, included. Unit: the scan's x. Range: any number. Default: required (unless points is given). Proposed: the right end of the range selected in the plot.
`points` Points	A list of points, each by its x, matched to \(10^{-9}\) of x. Unit: the scan's x. Range: a list of numbers. Default: none (from and to are used). Proposed: the points picked in the plot.

Smooth (Savitzky–Golay)

smooth

Each point replaced by a polynomial least-squares fitted over a window of points centred on it; at each end, the polynomial of the first (last) window evaluated at its first (last) half.

σ: \(\sigma\)² of a smoothed point is Σ c_k\(^{2}\) \(\sigma\)_k\(^{2}\) over its coefficients c_k.

Definition: A. Savitzky & M. J. E. Golay (1964), Anal. Chem. 36, 1627–1639; as SciPy's savgol_filter(mode = "interp"), against which test data_operations checks it on four real files

`points` Window	The number of points the polynomial is fitted over. Unit: points. Range: odd, 3 or more, more than order, at most the scan's points; the step must be constant to 1 %. Default: required. Proposed: 5. Raising it: a smoother curve; peaks narrower than the window flattened and broadened. Lowering it: closer to the data, less noise removed. SciPy savgol_filter, window_length
`order` Order	The degree of the polynomial. Unit: none. Range: 0 or more, less than points. Default: required. Proposed: 2. Raising it: the peaks' shape kept better, less noise removed. Lowering it: more noise removed, peaks flattened (order 0 is a moving average). SciPy savgol_filter, polyorder

Rebin

rebin

Points combined into fewer: by a factor (each n consecutive points become one at their mean x; the last points that do not make n are dropped), or to a new step (each point's interval, between the midpoints to its neighbours, shared among the new intervals in proportion to the overlap).

σ: mode sum: \(\sigma\)² = Σ (share·\(\sigma\))²; mode average: that divided by the factor (or by the new step over the old).

Definition: the core's own (R21, L066); test data_operations (counts conserved in mode sum, exact identities)

`factor` Factor	How many consecutive points become one. Unit: points. Range: a whole number, 1 or more. Default: none (step is used). Proposed: 2. Raising it: fewer, broader points; more counts per point. Lowering it: closer to the measured step.
`step` New step	The step of the new points. Unit: the scan's x. Range: positive; the scan must have two points or more. Default: required when factor is absent. Proposed: twice the scan's step. Raising it: fewer points. Lowering it: more points (below the measured step, the counts are shared out, not measured).
`mode` Mode	sum: the counts are conserved (Σy before = Σy after); average: the sum divided by the factor (or the new step over the old), so that the counts per point stay on the same scale; keep (a factor only): the first point of each group kept as it is, the others left out, as FullProf's Ste (manual p. 64, L267). Unit: sum, average or keep. Range: sum, average, keep. Default: sum. Proposed: sum.

Normalise

normalize

y and \(\sigma\) multiplied by one factor: a value over the scan's monitor, its counting time or its maximum; or a factor given.

σ: \(\sigma\) times the same factor.

Definition: the core's own (R21, L066); test data_operations

`by` By	What the scan is divided by: its monitor, its counting time or its maximum (from the file's metadata or the data), or nothing (a factor given). Unit: monitor, time, maximum or factor. Range: monitor, time, maximum, factor; monitor and time need the file to state them. Default: required. Proposed: monitor when the file states one, else time, else maximum.
`to` To	The value the monitor, the counting time or the maximum is brought to: y · to / monitor (time, maximum). Unit: counts (or the scan's y). Range: any number; with by = monitor, time or maximum. Default: required with by = monitor, time or maximum. Proposed: the file's own monitor (or time), so that the counts keep their scale; 1000 for the maximum.
`factor` Factor	The factor y and \(\sigma\) are multiplied by. Unit: none. Range: any number. Default: required with by = factor. Proposed: 1.

Add a scan

add

Another file's scan, times scale, interpolated linearly at this scan's points and added; points outside the other scan are removed.

σ: \(\sigma\)² = \(\sigma\)_a\(^{2}\) + scale\(^{2}\) \(\sigma\)_b,interpolated\(^{2}\), \(\sigma\)² of an interpolated point (1 − t)²\(\sigma\)₁² + t\(^{2}\)\(\sigma\)₂².

Definition: the core's own (R21, L066); test data_operations

`file` File	The other file, relative to the control file; read by the reader registry. Unit: path. Range: a file the registry reads. Default: required. Proposed: a file chosen in the project.
`reader` Reader	The format to read the other file as, when the registry cannot tell it alone. Unit: a reader's id. Range: an id of `drs data formats`. Default: none (the registry recognises the file). Proposed: the format the registry recognises.
`scale` Scale	The factor the other scan is multiplied by before it is added. Unit: none. Range: any number. Default: 1. Proposed: 1.

Subtract a scan

subtract

Another file's scan (an empty holder, a measured background), times scale, interpolated linearly at this scan's points and subtracted; points outside the other scan are removed.

σ: \(\sigma\)² = \(\sigma\)_a\(^{2}\) + scale\(^{2}\) \(\sigma\)_b,interpolated\(^{2}\).

Definition: the core's own (R21, L066); test data_operations

`file` File	The other file, relative to the control file; read by the reader registry. Unit: path. Range: a file the registry reads. Default: required. Proposed: a file chosen in the project.
`reader` Reader	The format to read the other file as, when the registry cannot tell it alone. Unit: a reader's id. Range: an id of `drs data formats`. Default: none (the registry recognises the file). Proposed: the format the registry recognises.
`scale` Scale	The factor the other scan is multiplied by before it is subtracted (the ratio of the counting times or monitors). Unit: none. Range: any number. Default: 1. Proposed: the ratio of the two files' monitors (or counting times) when both state them, else 1. Raising it: more of the other scan removed. Lowering it: less removed.

Merge a scan

merge

The points of another scan joined to these; points at the same x (to \(10^{-6}\) of the step) become their mean weighted by 1/\(\sigma\)².

σ: \(\sigma\) = (Σ 1/\(\sigma\)²)^(−1/2) at the merged points.

Definition: the core's own (R21, L066); test data_operations

`file` File	The other file, relative to the control file; read by the reader registry. Unit: path. Range: a file the registry reads. Default: required. Proposed: a file chosen in the project.
`reader` Reader	The format to read the other file as, when the registry cannot tell it alone. Unit: a reader's id. Range: an id of `drs data formats`. Default: none (the registry recognises the file). Proposed: the format the registry recognises.
`scale` Scale	The factor the other scan is multiplied by before it is merged. Unit: none. Range: any number. Default: 1. Proposed: 1.

Convert the axis

convert

x converted to 2\(\theta\), d or Q (d = λ/(2 sin \(\theta\)), Q = 4π sin \(\theta\)/λ = 2π/d); y and \(\sigma\) kept; the points kept in increasing x.

σ: unchanged (the axis changes, not the counts).

Definition: Bragg's law; test data_operations (exact identities)

`to` To	The quantity on the new axis. Unit: 2theta, d or q. Range: 2theta, d, q; from 2\(\theta\), d or Q only (time of flight and energy need the instrument's constants). Default: required. Proposed: d.
`wavelength` Wavelength	λ of the conversion. Unit: Å. Range: positive. Default: the scan's λ1. Proposed: the scan's λ1.

Another wavelength

wavelength

2\(\theta\) recalculated for another wavelength, sin \(\theta\)′ = (λ′/λ) sin \(\theta\); points with |sin \(\theta\)′| > 1 removed.

σ: unchanged.

Definition: Bragg's law; test data_operations

to
To The new wavelength λ′. Unit: Å. Range: positive; the scan's x must be 2\(\theta\), and its λ known. Default: required. Proposed: Cu Kα1, 1.5406 Å, to compare with laboratory data. Raising it: the peaks move to higher 2\(\theta\), the high-angle points past 180° are removed. Lowering it: the peaks move to lower 2\(\theta\).

from
From The wavelength the scan was measured with, for a scan that states none (a control file's λ1, W7's Use λ from); a scan that states its own is converted from it, and a different from is refused. Unit: Å. Range: positive. Default: the scan's own λ. Proposed: the control file's λ1. Raising it: the same scan read as measured at a longer wavelength: the peaks move to lower 2\(\theta\). Lowering it: the peaks move to higher 2\(\theta\). app request 34

`to` To	The new wavelength λ′. Unit: Å. Range: positive; the scan's x must be 2\(\theta\), and its λ known. Default: required. Proposed: Cu Kα1, 1.5406 Å, to compare with laboratory data. Raising it: the peaks move to higher 2\(\theta\), the high-angle points past 180° are removed. Lowering it: the peaks move to lower 2\(\theta\).
`from` From	The wavelength the scan was measured with, for a scan that states none (a control file's λ1, W7's Use λ from); a scan that states its own is converted from it, and a different from is refused. Unit: Å. Range: positive. Default: the scan's own λ. Proposed: the control file's λ1. Raising it: the same scan read as measured at a longer wavelength: the peaks move to lower 2\(\theta\). Lowering it: the peaks move to higher 2\(\theta\). app request 34

Analyses

Peak search

peak_search

The local maxima of the scan with their prominence (height above the higher of the two bases), height and width at half the prominence; kept when the prominence is at least min_prominence and at least min_significance times the peak's \(\sigma\).

σ: the scan is unchanged; \(\sigma\) decides which peaks are significant.

Definition: SciPy's find_peaks and peak_widths(rel_height = 0.5), against which test data_operations checks it; GSAS-II's own candidates (gsas2_peak_candidates) are given beside it

`min_prominence` Least prominence	A peak is kept only if it stands at least this high above the higher of its two bases. Unit: counts (the scan's y). Range: 0 or more. Default: 0. Proposed: 0: the significance alone decides, as it scales with the counts. Raising it: fewer peaks: only the stronger ones. Lowering it: more peaks, the weak ones and noise among them. SciPy find_peaks, prominence
`min_significance` Least significance	A peak is kept only if its prominence is at least this many times its own \(\sigma\). Unit: \(\sigma\). Range: 0 or more. Default: 3. Proposed: 3: a peak three standard deviations above its bases. Raising it: fewer peaks, the weak ones dropped. Lowering it: more peaks, noise among them. the core's own (R21, L066)

Automatic background

auto_background

A smooth curve under the peaks by asymmetrically reweighted penalised least squares (arPLS): a Whittaker smoother whose weights drop where the data stand above the curve; with points every N/100 of the scan, as GSAS-II takes them for its fixed points.

σ: the scan is unchanged.

Definition: S.-J. Baek, A. Park, Y.-J. Ahn & J. Choo (2015), Analyst 140, 250–257; as pybaselines' whittaker.arpls, which GSAS-II uses (GSASIIpwd.py autoBkgCalc); test data_operations against both

log_lambda
Smoothness (log₁₀ λ) log₁₀ of the smoother's λ, the weight of the curve's second differences against its distance to the data. Unit: none. Range: any number (GSAS-II offers 0 to 10). Default: GSAS-II's for N points: min(10, (int(10·log₁₀(N)^1.5) − 9.5)/10). Proposed: the default for the scan's number of points. Raising it: a stiffer curve that passes under broad features. Lowering it: a curve that follows the data, rising into broad peaks. GSAS-II GSASIIpwd.py autoBkgCalc, logLam

max_iter
Iterations How many times the weights are updated after the first solve (it stops sooner when they change by less than \(10^{-3}\)). Unit: none. Range: 0 or more. Default: 10. Proposed: 10. Raising it: the weights settle further where they have not converged. Lowering it: a curve closer to the plain smoother, higher under the peaks. pybaselines whittaker.arpls, max_iter; GSAS-II's autoBkgCalc stops after 10

`log_lambda` Smoothness (log₁₀ λ)	log₁₀ of the smoother's λ, the weight of the curve's second differences against its distance to the data. Unit: none. Range: any number (GSAS-II offers 0 to 10). Default: GSAS-II's for N points: min(10, (int(10·log₁₀(N)^1.5) − 9.5)/10). Proposed: the default for the scan's number of points. Raising it: a stiffer curve that passes under broad features. Lowering it: a curve that follows the data, rising into broad peaks. GSAS-II GSASIIpwd.py autoBkgCalc, logLam
`max_iter` Iterations	How many times the weights are updated after the first solve (it stops sooner when they change by less than \(10^{-3}\)). Unit: none. Range: 0 or more. Default: 10. Proposed: 10. Raising it: the weights settle further where they have not converged. Lowering it: a curve closer to the plain smoother, higher under the peaks. pybaselines whittaker.arpls, max_iter; GSAS-II's autoBkgCalc stops after 10