Measurement fundamentalsReading time 13 min

Resolution Is Not Measurement Capability

A display that reads 0.001 mm does not prove that a 0.01 mm tolerance can be judged reliably. Separate resolution, repeatability and uncertainty before choosing a gauge or accepting a result.

01Four measurement terms

Term	Definition	In plain language
Resolution	The smallest increment the instrument can display	How much one step on the screen is worth
Accuracy	How close a reading is to the true value	Is it systematically off
Precision / repeatability	The spread of repeated readings under the same conditions	Will ten readings agree
Uncertainty	A quantified statement of the interval you can trust	How far can you believe this number

Display resolution is only one specification; it does not establish total measurement uncertainty.

A digital caliper may display 0.01 mm increments, yet its result also depends on measuring force, alignment, temperature and the operator. Those effects are not visible on the screen. Before accepting a close-to-limit result, compare the total measurement uncertainty with the tolerance.

The Abbe principle

Abbe's principle favors a measuring scale aligned with the dimension being checked. Caliper jaws sit away from their scale, so jaw tilt can add error. A micrometer's geometry reduces that particular effect, though its suitability still depends on the feature and measurement range.

02What GR&R measures

Gauge R&R separates variation from repeated readings by one operator and variation between operators. Use it to see whether the measurement method can distinguish real part differences from measurement noise.

%GR&R	Verdict	Action
< 10%	Acceptable	Fit for process control
10% ～ 30%	Conditionally acceptable	Decide on the importance of the application and the cost of improvement
> 30%	Not acceptable	The measurement system must be improved

Gauge R&R is interpreted against the study's chosen tolerance or process variation and the applicable acceptance criteria.

03Matching gauge to feature

Instrument	Typical resolution	Typical uncertainty	Typical use
Vernier caliper	0.02 / 0.01 mm	±0.03 mm	Quick checks on the floor, coarse tolerances
External micrometer	0.001 mm	±0.004 mm	Outside diameters, plate thickness, IT7 and looser
Lever dial indicator	0.001 mm	±0.003 mm (comparative)	Runout, parallelism, setting up
Gauge blocks (grade 0)	—	±0.1 μm class	Calibration reference
Optical comparator / vision system	0.001 mm	±0.002 mm	Two-dimensional profiles, many features at once
Coordinate measuring machine	0.0001 mm	MPE varies by model	Three-dimensional features, geometric tolerances
Roundness tester	0.01 μm	Sub-micron	Roundness, cylindricity

Instrument performance varies with model, range, setup and environment.

Contact and non-contact

Contact methods are unaffected by surface colour or gloss and have mature traceability, but every point has to be touched, complex profiles are slow, and measuring force deflects thin or soft parts.
Vision systems capture hundreds of dimensions at once and suit sheet and stamped parts, but they cannot see height, and edge detection depends on surface condition and lighting.
Laser and confocal reach into deep holes and cope with high-contrast surfaces, but distort on steep slopes and transparent materials.

04Temperature effects

Dimensional metrology commonly uses 20 °C as a reference temperature. Materials expand at different rates, so a long aluminum part can shift noticeably between a warm shop floor and a controlled room. Record temperature when the tolerance makes it relevant.

Materials	Expansion coefficient (μm/m·K)	Change on a 500 mm part, 8 °C rise
Carbon steel	11.7	47 μm
Stainless 304	17.3	69 μm
Aluminium alloys	23.1	92 μm
Brass	19.0	76 μm
Cast iron	10.5	42 μm

Let the part reach the reference environment before judging a tight tolerance.

05Choosing a method

Start with the feature. Two-dimensional profile or three-dimensional form? Are there geometric tolerances that need datums?
Then look at the tolerance band. Aim for measurement variation below one tenth of the band.
Then throughput. Sampling or 100% inspection? How many a day? What is the takt?
Finally traceability. Does the customer want an inspection report, or a calibration certificate?

Match the method to the characteristic and production flow. A CMM suits complex three-dimensional geometry; vision may support high-volume two-dimensional checks; calipers and micrometers remain useful when their uncertainty is small enough for the tolerance.