That can be it, these accounts are indeed not (yet) following.

You may need to reconsider your approach a bit, the blockchain will only increase. If Steem really takes off the delays could be considerable.

An approach would be to have a separate database (mongo or mysql) where you store all relevant info you want to show. You could maintain this database in the background, store a record up till which block you processed and only traverse the blockchain from that block and newer.
For the rest of the info you can use the 2nd database.