Answer
Use nearby fixed references on the ground or horizon, compare the aircraft against them, and scan more than one reference rather than judging motion against empty sky.
Explanation
Drift is easier to detect when the eye has a stable reference. Empty backgrounds can hide slow movement.