This is very important to remember. I used to always try to explain myself when at one point I got tired because people won't actually try to understand me. That's when I realized that I actually don't have to explain myself that much because at the end of the day, no one else is living your life.
Only I know why I did what I did and why I do what I'm doing.
You shouldn't be afraid to do what you want but of course, don't do much at the expense of consideration for others. We still live in a society and I'm sure you don't want to end up in prison or mental hospital.
Seek what is in harmony with your own nature and strive towards this, even if it brings no reputation. For every man is allowed to seek his own good. - Marcus Aurelius