To understand this line of thinking, it's very important to get the difference between private property (ownership) and personal property (possession).

It's of course not ok to take your TV, your clothes, your house - as long as you use them. Taking them away from you while you utilize them is an attack on your personal freedoms, which can't be tolerated. But taking an abandoned house to live in there doesn't harm the owner personally. He shouldn't be able to kick you out to let it stay empty again.