There's nothing wrong in saying flat out.."no " or even if you have to sugar coat it by saying, "oh I'm sorry I can't. I have no time or ability" etc.
If that person gets mad at you and stops talking to you, Im afraid to say it but, that person was probably not your friend and from the beginning.
RE: Lo difícil que es decir "NO" Spa / Eng