The advantage of a laser interferometer is in the high resolution it provides, especially in extremely sensitive nanometrology. The laser interferometer plays a crucial role in ultra‐precision measurement areas. However, the measurement accuracy is limited by a nonlinearity error. The nonlinearity error of the heterodyne laser interferometer needs to be compensated for. An H ∞ filter is applied to compensate for the nonlinearity error caused by the elliptic polarisation, two non‐orthogonal beams and an imperfect optical alignment. The proposed compensation method using an H ∞ filter can reduce the nonlinearity error. The experimental results show an improved accuracy and reliability.