SteemData now has new, multi-threaded blockchain indexers. This is an long-overdue update, bringing better scalability, performance (one order of magnitude increase) and consistency guarantees.
Further, SteemData has been re-synced from scratch to address past inconsistencies.
Syncing Status
- Operations (100%)
- Accounts (100%)
- Account Operations (99.9%)
- Posts and Comments (93%)
The database is currently @ manageable 150GB. You can see the status of the indexers and other basic stats here.
Real-Time Updates
The last open concern is state change propagation in near-real time.
For example, a single vote operation should trigger cascading updates of all affected Account, AccountOperations and Posts/Comments entries. This is because the vote affects the voters state (voting power), voters history, and the post/comment the vote was cast on.
As the Steem Blockchain usage grows, the amount of dependent state that needs to be updated with each block grows exponentially. Scaling SteemData infrastructure to support this naive model might be tricky in the future. I am currently evaluating 2 options: Using an indexer, which is more efficient and offers better consistency guarantees, vs using a distributed task queue which offers higher capacity (scalability). I hope to have some time this weekend to reach closure on the matter.