This is quite interesting. Many people are genuinely of the opinion that their daily struggles are enough as exercise. Like you said, you really don't know the amount of research that went into the American's Heart Association claim. What I can say is that there are counter claims.
RE: Why You Need Exercise in Your Manual Labor Job (AHA Guidelines)