Cosine error
Cosine error
Why a tilted instrument distorts the measurement.
Cosine error is one of the most common, and most underestimated, errors in dimensional measurement. It appears when the direction along which we measure is not aligned with the true direction of the dimension, that is, when the instrument sits at an angle. It affects rules, calipers, laser interferometers, linear displacement sensors and, most notably, dial test indicators.
What cosine error is
Every length measurement assumes we measure along the correct direction. If the instrument is turned by an angle θ relative to that direction, we no longer measure the true dimension but its projection onto the wrong axis. Because the relationship between true and measured value runs through the cosine of the angle (cos θ), the error is called cosine error.
Its key feature is that the error is second order in the angle: for small angles it stays very small, but it grows quickly as the angle increases. This sets it apart from the Abbe error, which is first order.
The general principle and the formula
In general terms, an angular deviation θ introduces a factor cos θ between the measured and the true value. The sign (whether the instrument over-reads or under-reads) depends on the geometry of the setup.
Laser interferometer
If the beam is not parallel to the axis of motion, the measured displacement comes out smaller than the true one: measured ≈ true × cos θ.
Dial test indicator
Because the stylus moves along an arc, the reading comes out larger than the true value: true = reading × cos θ.
Order of the error
A useful approximation for the size of the error (regardless of sign) is error ≈ travel × (1 − cos θ). Since (1 − cos θ) is about θ²/2 for small angles (θ in radians), it is clear why the error is second order.
The dial test indicator
In dial test indicators the operation is based on the movement of the stylus along a circular arc around a pivot point. The scale is correctly calibrated only for a specific stylus length and only when the stylus is practically parallel to the surface of the workpiece. When the stylus forms an angle θ with the direction of measurement, the contact point travels a greater distance than the actual height difference, and the instrument over-reads.
The relationship that describes it is H = S · cos θ, where H = actual difference, S = indicated difference, θ = inclination angle. In other words, to correct the reading you multiply by the compensation factor cos θ.
Compensation factor
The table below shows, per inclination angle, the compensation factor (cos θ) and the percentage error of the reading for a dial test indicator. Rule of thumb: if the angle is kept within 15°, the error stays below ~4%.
Cosine error vs Abbe error
The two concepts are often confused because both arise from poor alignment, yet they behave differently. The Abbe error is first order in the angle (it grows linearly, error ≈ offset × θ) and is due to the scale being offset from the line of measurement, which is why it creeps in easily even at small angles. The cosine error is second order (error ≈ travel × θ²/2) and stays negligible for small angles. In practice: Abbe demands care in the design and setup of the instrument, whereas cosine mainly demands correct alignment and parallelism during measurement.
Summary
Cosine error arises from an angular deviation θ between the direction of measurement and the true dimension, and relates the measured value to the true one through cos θ. It is second order, so small for small angles but growing quickly. In dial test indicators H = S · cos θ holds (the instrument over-reads), and the solution is proper parallelism of the stylus or applying the compensation factor. Together with the Abbe error, attention to alignment is one of the most effective steps toward reliable measurements.