Why Early Rounding Distorts Mean, Variance, and Standard Deviation

A student computes the mean of three measurements, 10.1, 10.3, 10.2, and rounds the result to 10 before calculating deviations. That one-digit truncation changes the variance from 0.01 to 0.07, a sevenfold error. Understanding why early rounding distorts mean, variance, and standard deviation starts with a single rule: retain at least two extra decimal places through every intermediate step, and round only the final reported statistic.

NIST SP 811 and ISO 5725-1 both recommend carrying one or two additional digits during intermediate calculations to prevent rounding error from propagating through subtraction, squaring, and averaging. The mean is the sum of all data points divided by the count. Variance is the average of squared deviations from that mean. Standard deviation is the square root of variance. Each of those operations can amplify a small rounding error if the input has already been truncated.

Take three values: 10.1, 10.3, 10.2. Each has three significant figures. The sum is 30.6. The mean is 30.6 / 3 = 10.2. For intermediate work, retain 10.2000. Deviations: 10.1-10.2 =, 0.1, 10.3-10.2 = 0.1, 10.2-10.2 = 0.0. Squared deviations: 0.01, 0.01, 0.00. Sum of squared deviations = 0.02. Sample variance = 0.02 / (3-1) = 0.01. Standard deviation = √0.01 = 0.1. Report mean = 10.2 and s = 0.1, both at three significant figures.

Now repeat the same calculation but round the mean to 10 (two significant figures) before computing deviations. Deviations become 0.1, 0.3, 0.2. Squared deviations become 0.01, 0.09, 0.04. Sum = 0.14. Variance = 0.07. Standard deviation = 0.26. The variance is seven times larger and the standard deviation is 2.6 times larger than the correct values. The error comes entirely from one early rounding step.

The example above is not a hypothetical edge case. It uses real data with three significant figures, the kind a chemistry student or lab technician handles every day. The failure occurs because variance depends on squared differences from the mean. Squaring amplifies small deviations. If the mean itself is off by even 0.2, each deviation shifts by that same 0.2, and after squaring the error grows.

Consider the same data set with one extra digit of precision: 10.05, 10.35, 10.20. The true mean is 10.20. Rounding the mean to 10.2 before calculating deviations produces the same correct result as before. Rounding it to 10 produces the same sevenfold error. The problem is not the data; it is the decision to round before the calculation is complete.

The calculation itself has three stages: sum and divide for the mean, subtract and square for deviations, then average and square root for variance and standard deviation. Each stage multiplies the error from the previous one.

When you need to compute the next step, you reach for a statistics calculator. That tool will give the correct result only if you feed it unrounded intermediate values. If you input a rounded mean, the calculator has no way to recover the lost digits.

The rule from NIST SP 811 section 7.5 is unambiguous: carry at least two extra decimal places through every intermediate step. For a final report to one decimal place, retain three decimal places during calculation. For a report to three significant figures, retain four or five significant figures in intermediate work.

Rounding error propagates through subtraction and squaring because those operations are not linear. A small error in the mean shifts every deviation in the same direction. When those deviations are squared, the error grows as the square of the shift. Variance, being the average of those squared deviations, inherits the amplified error. Standard deviation, the square root, inherits about half the error in percentage terms, but half a large error is still a large error.

The mechanism is consistent across data sets. For a set of five values with a true variance of 0.05, rounding the mean to the nearest integer can produce a variance of 0.20 or more. The error is not random; it systematically inflates the spread of the data.

The most common mistake is rounding the mean to the same number of digits as the raw data, then computing deviations. The failure case above, rounding 10.2 to 10, is the simplest version. A more subtle version occurs when the data themselves are rounded before entering the calculation. If the original measurements are 10.1, 10.3, and 10.2 but were recorded as 10, 10, and 10, the variance becomes zero.

This is not a data-entry error; it is a rounding error that destroys all information about spread. The guide for users who need to avoid this is simple: never round a data point until the final report. If intermediate rounding is unavoidable, retain at least two extra digits.

Students in grades 3 through 5 learning to round whole numbers and decimals under Common Core standards 3.NBT.A.1, 4.NBT.A.3, and 5.NBT.A.4 are suited for this content.

It also suits high school chemistry and physics students applying significant-figure rules from OpenStax Chemistry 2e, and parents checking homework who need a quick reference.

Office and finance users who need to understand how Excel's ROUND, ROUNDUP, ROUNDDOWN, and MROUND behave will find the tie-breaking rules relevant. Programmers and data analysts who work with Python's round(), JavaScript's Math.round(), or .NET's Math.Round will benefit from knowing that the default rounding mode varies by language.

This content excludes philosophical background on approximation, advanced error propagation for Monte Carlo simulations, and jurisdiction-specific currency rounding for tax compliance. Those topics require separate resources.