Being wrongly accused cause a lot of pain and hurt on it own, but that hurt doubles when it's someone close, someone you can call family, that's making the accusations. The sudden awareness that even these people do not hold you in high esteem. And just like you said, no amount of apology can really undo that.