Both Men and Women should lower their gaze because that look that you might frame as 'simple' can lead to so much more.

"Tell the believing men to lower their gaze (from looking at forbidden things)” [al-Noor 24:30]

"And tell the believing women to lower their gaze (from looking at forbidden things), and protect their private parts..." [al-Noor 24:31]

Striving with your self and training yourself to lower your gaze and be patient in doing so, and not giving up. Allah says (interpretation of the meaning:

“As for those who strive hard in Us (Our Cause), We will surely guide them to Our paths (i.e. Allaah’s religion — Islamic Monotheism)” [al-‘Ankaboot 29:69]