Data reference notes

Missing values and zero

Zero is an observed numeric value. Missing means no usable value was supplied. Treating missing data as zero can distort totals and averages. Decide on exclusion, explicit imputation or rejection based on the meaning of the measurement.

Denominators

A rate needs a clearly defined numerator and denominator. Before comparing conversion rates, error rates or retention, verify that the opportunity counted in the denominator is the same in both groups.

Weighted means

For values x with non-negative weights w and positive total weight, the weighted mean is sum(w*x)/sum(w). Averaging group percentages equally changes their influence when group sizes differ. Use the appropriate denominator as the weight when combining compatible rates.

Mean and median

The arithmetic mean adds observations and divides by their count. The median is the middle ordered observation, or the average of the two central observations for an even count. Extreme observations can move the mean more strongly than the median.

Primary reference: www.itl.nist.gov

Percentage points

Moving from 20 percent to 25 percent is a rise of 5 percentage points and a relative increase of 25 percent. State which comparison is intended. Relative change divides by the baseline, so a zero baseline needs separate treatment.

Population and sample variation

Population variance describes the supplied population using its count as divisor. A common sample variance estimator uses count minus one. Specify which quantity is being reported; the sample form is not defined for a single observation.

Correlation and causation

An association does not by itself identify the cause. Consider common causes, reverse direction, selection effects and measurement artifacts. A causal claim needs a design or assumptions that address plausible competing explanations.

Data provenance

Keep a record of source, collection time, transformations and ownership. Derived outputs should be traceable to the inputs that produced them. Provenance helps investigate disagreement but does not automatically make a source correct.

Data leakage

Information unavailable at real prediction time must not accidentally enter training or evaluation inputs. Separate cases by the real deployment boundary, such as time or entity, when random row splitting would share information across that boundary.

Measurement drift

A change in a metric can result from changed instrumentation, filtering, population or definitions. Check those possibilities before concluding that the underlying behavior changed. Preserve metric definitions alongside historical values.