Why do some have an issue with someone with a less than attractive person....?

Why I ask this is we say look inside the heart for more because beauty is skin deep!. Ok but when you see an attractive person with a less than attractive person you hear.."Ewww why is he/she with them they could do better" Now what's your take? I thought the inside matters more or does it not...?