I can't tell you the exact but surely the average number of people whisper about every person around them.They never know the exact thing why he is doing this,why Mr Thomson comes late at night.They only would say,he is wasting his money and time he is late out night so he is at bar or party,but who knows that Mr Thomson is working late night or doing an extra job to earn some more money.
Whatever you do people would never tolerate any thing they will give an opinion because its their job for which they are not even getting paid.They will think negative or positive about you,only you know the reality what you are doing and why you are doing.If a performer on stage get discouraged with the audience booing him,he will never get to the top or may be not able to perform next time because he is taking opinions as final decision.
You if you want to be something and you have a positive and useful goal then don't quit until completing it,at least don't let opinions ruin it.
Again the final decision lies in your own opinion about yourself.