That can be it, these accounts are indeed not (yet) following.
You may need to reconsider your approach a bit, the blockchain will only increase. If Steem really takes off the delays could be considerable.
An approach would be to have a separate database (mongo or mysql) where you store all relevant info you want to show. You could maintain this database in the background, store a record up till which block you processed and only traverse the blockchain from that block and newer.
For the rest of the info you can use the 2nd database.
RE: Who are your Most Loyal/Ghost/Dead/Most Influential Followers on Steemit? And who upvoted you with how much?