I have seen many Football matches in Qatar. I am 99.99% sure that there will be no problem. I have gone with my friends and my own kids. For me Qatar is among the few safest place on earth with/without family.

Eversince we are here (last 5 years) neither me nor my wife have ever been commented privately and publicly. This is the beauty of Qatar. Here there are more moral, social, ethical, and cultural values then anywhere in the world. I have worked and lived in 5 different countries. If I were you, I would have gone with my 3 year old kid and wife. If you are that worried then go by yourself this time and when you are confident enough then go with your family next time.
Have Fun!!!