Never really noticed, but i dont spend a helluvalotta time in elevators. Here guys would look away mostly to avoid the appearance of staring at the lady. Elevators (lifts) are close quarters, making it hard to observe standard social conventions regarding invading others personal space.
Notions of what qualifies as an appropriate distance to maintain so as not to invade personal space differ from one society to the next. Ive been told this has occasionally led to problems when our military is stationed in other cultures where it is common protocol to get virtually nose to nose while conversing... the result being GIs cold-cocking some Local guy who didnt realize he was perceived as "getting up in their face", or why that was even an issue.
Here getting nose-to-nose with another male is the same as staring a silverback gorilla in the eyes... a challenge.
[What can I say... We're both bi-pedal primates]