I wrote this article about 16 hours ago when Steem was down. But I wasn't able to post it via busy because it was also down since the whole blockchain was taken down. I am posting it now from the point of view of when Steem was down (about 16 hours ago).
Steem blockchain’s biggest web interface, steemit.com has been down for a few hours. Trying to visit the website steemit.com shows the following error message:
Sorry.
Looks like something went wrong on our end.
Head back to Steemit homepage.
I tried visiting different links to the site but all of them show error messages. Visiting profile links show the following error:
Sorry! This page doesn’t exist.
Not to worry. You can head back to our homepage, or check out some great posts.
After asking around, I learned that Steem blockchain crushed due to an incompatible block when some witness nodes / block producers updated to version 20 of Steem the software.
This new version added the ability to upvote posts in the last 12 hours of a post payout. A user reportedly upvoted a post in that last period which resulted in an incompatible block. Steem witness neoxian said in the witness channel of Steem chat:
Someone flagged something at the last 12 hours, then seraph defended it with an upvote…and then boom, you have an incompatible block.
Another Steem witness, timcliff stated:
The consensus that was decided by the top witnesses is to rollback to 19.x Everyone should switch to 19.6 / 19.12.
– If you are running any nodes with 20, you will need to stop the node.
– The 20.0 block_log will not be compatible with the 19.x version.
He further said:
Basically the best thing to do right now is hang tight. There is a bug in V19 and V20, and the chain has stopped until a solution can be found. It is being worked on and a solution will be posted once it’s known.
In order to fix this, Steem witnesses will have to revert back to Steem version 19.6 / 19.12 and start the chain with a newly released version of Steem with current bugs fixed. Several witnesses / block producers said that Steem Inc is working on releasing a new fixed version.