You are right to exam yourself. Self-improvement is an important part of self actualization. However, in that vein, receiving pleasure from sources other that yourself is valuable also. All sources should be considered, then discarded if necessary. Just because something is conditioned does not autumatically make it wrong. You decide after deliberation.
For myself, I don't think getting pleasure from others diminishes my self worth. It enhances it. Love is the answer to lots of things.