Precision
For numbers, how finely a value is recorded: 3.14159 carries more precision than 3.1. In matching and error detection the word has a second meaning: of everything that was flagged, how much was flagged correctly.
Numeric precision is not accuracy. A weight recorded to six decimal places can still be wrong, and digits produced by a calculation can suggest a finer measurement than was ever made.
Where it comes up
- Rounding and precision lossDigits were dropped when a value was displayed, exported, or converted, and the file now holds the shortened number.
See also
- RoundingReplacing a number with a nearby one that has fewer digits, such as 2.675 with 2.68.
- Floating pointThe usual way computers store numbers with a fractional part, using a fixed number of binary digits. Most decimal fractions cannot be stored exactly, so 0.1 plus 0.2 comes out as 0.30000000000000004.
- AccuracyHow closely a recorded value matches the real thing it describes. A birth date that is a well-formed date and still not the person's birth date is inaccurate.