Even ignoring the fundamental subjectivity of the question, your reasoning only works if all distributions relating attractiveness to some feature are "simple", having only one peak. If instead, for example, the distribution is bimodal, then the average is going to be in a region that almost nobody finds attractive.