I think below reasons all together prove your point!

At least it's not unlikely, since ancient buildings like this are known to be much more stable than more recently built houses, and did obviously last already much longer than modern buildings will probably. Moreover, depending on the reasons for this difference in stability and longevity, it is very likely that these ancient buildings - not only in Rome, but worldwide (if people don't destroy it, like you say) - will still last even longer from now on than modern buildings will, by far.
While this fact does seem fairly obvious even to the average person without further knowledge about materials, architecture and engineering, I read/heard about this from various independent knowledgeable sources; however, while none doubted the fact, various sources cited slightly different reasons for this difference in building longevity:

• one reason is of the obvious statistical kind, akin to the so-called survivorship bias: we don't know about all ancient buildings, and tend to know those which are still fine today
• the sand historically used for concrete was better suited for concrete buildings as various kinds of sand - those suitable for concrete - were more abundant, and nowadays we even see sand becoming a rare and increasingly expensive resource (since desert sand is too round to be used, smoothed by wind), possibly one of the most critical resources we'll miss in future decades.
  • actually, I just stumbled upon the concrete reason ;) for roman concrete to be so durable as it has it's own WP article!
• concrete does continue to become harder and does not ever totally stop curing AFAIK, so the buildings which made it already to these days will have more durable concrete the older they are already.

Originally, I just wanted to approve of the pantheon as it is my favorite building/place in rome as well, just so maybe since I like ancient mythology and the building is pretty polished and generally in a good shape while still looking somewhat authentic :)