If you've grown a beard for religious reasons, why would you care what other people think?

But generally, if you look like Osama Bin Laden, people's perceptions of you (1st impressions) might be "terrorist", and depending on where you are, you might get into trouble. The reason I say this is that people react to things based on their perceptions and if they are not open-minded, that could endanger your personal safety (or as you are probably more concerned about, women may find it a turn-off). You could change the way you dress, if you think you must, hide the beard or something like that.

/If you're here in Qatar, no issue.

//To answer your question - I personally do not find a man with too much facial hair attractive.
(And I am not a man, contrary to my avatar. Can't change it yet. Did try but no luck)

*****************************************
Life is short. Live it to the fullest!