I'm agnostic to IP laws with a bias towards your libertarian argument, but sometimes it's good to play Devil's advocate:

"When the coercive powers of the state are invoked to benefit some and to restrain others, the creative processes will always suffer and, as a consequence, so will the vibrancy of a civilization."
***We should add "restrain others from conducting peaceful activities" to rule out the obvious harm of not restraining people who intend to do harm. The next question is whether stealing your idea without due compensation falls into that category?

I'm not convinced that IP (e.g. patents) really drive innovation; most innovation occurs informally or is taken directly and immediately to market. However, here's an interesting finance paper that links patents and citations to expected stock returns:
"Innovative Efficiency and Stock Returns"
http://www.sciencedirect.com/science/article/pii/S0304405X12001961

Nonetheless, even if IP drives innovation, that's merely a utilitarian wealth maximization argument; it needs to be evaluated wrt the philosophical argument of whether human beings retain full rights to their own lives, and then whether government imposition of IP laws violate those rights. As you point out, since IP laws do not bind bc of voluntary contractual relationships, the answer is likely that they interfere with basic human rights.