As I've posted about many times, I'm a big fan of DPoS and on chain governance. I've given talks at conferences about DPoS and plenty of interviews, some of which you can find here, which include DPoS discussions. This governance model allows token holders to support witnesses directly and through them, vote for or against proposed code changes.
In my opinion, this is a much better governance approach than many other consensus algorithms. Bitcoin, which uses PoW (Proof of Work), only has a user activated hard/soft fork concept which, in a sense, holds the block producers hostage using full nodes. We're much more civil here. :)
That said, I wanted to explore an often mentioned criticism I hear when I attend conferences or talk with people who are deeply entrenched in other cryptocurrency projects (something I often call crypto tribalism). They talk about Steem, EOS, and BitShares being centralized (which is essentially a critique on DPoS itself). Given how many independent block producers and witnesses involved in creating consensus compared to a limited number of mining pools on other chains, I don't think this is a valid criticism.
What may be a valid criticism is evaluating where the code comes from.
When it comes to technology, whoever ships the code defines the technology. When there's a disagreement about something, the code wins. Usually, the concept of Code is Law rules the land (meaning, whatever the code does is the law, even if it was a bug). EOS is exploring ideas such as intent of code is law via Ricardian Contracts and arbitration, but that's still at very early stages and makes many in the blockchain space very uncomfortable.
Here on Steem, most of the major code changes come from one organization: Steemit, Inc. They are the for-profit entity behind this blockchain. In a sense, they are a centralized creator of the code which defines the blockchain. EOS has similar centralizations with Block One creating the code (or contracting firms such as Object Computing to create the code). In situations like this, when a hard fork is proposed to the community, witnesses and token holders have a limited number of options:
Do not upgrade in support of the Hard Fork until changes are made.
In the short term, this essentially maintains the status quo and does not improve the blockchain for future growth and performance. As you can imagine, this is usually a very unpopular decision for a witness to make and it can only be made effectively if there is a lot of support and consensus by token holders to keep that witness active. The fun thing is, this actually happened with Hard Fork 17. It had changes not everyone agreed to and was postponed until Hard Fork 18.
Test everything you can, do a code review, voice your concerns (if any), and go forward with the planned upgrade.
This approach, it seems, is the most common one. Does it mean witnesses are "rubber stamping" anything Steem, Inc. creates? Does it take anything away from the effective governance of Steem DPoS? Do we have any guidelines for what testing is done and how much code review is enough?
When witnesses go with option 2 and things break (complex software almost always breaks, it's just a matter of how often and how badly), it can be tempting to blame Steemit, Inc for putting too much into one change. It can be tempting to blame witnesses for not going with the very unpopular option 1. I've been talking with a number of top witnesses over the last few days about this hard fork and there are certainly some concerns.
The new resource credit system I mentioned in my last video is a huge change and requires some time to reach equilibrium. As the latest blog from Steemit, Inc. explains, we may have to adjust our expectations.
There's been some concern about the lack of tooling in place for witnesses to set the new properties needed to adjust the resource credit system (the functions are not available within cli_wallet). Thankfully, literally hours ago, Beem for Steem created and maintained by
figured out a way to properly updated the parameters.
The Steem Testnet is a huge win for the Steem ecosystem! That said, I get the impression not that many witnesses have really interacted with it extensively. The account creation fee is currently set to 0 which makes testing certain aspects of the resource credit system related to account creation difficult. It also started with some distorted resource measurements due to the way the content from the live chain was ported over, making testing of the resource credit system under real-world conditions quite difficult.
There are a lot of changes in Hard Fork 20. A lot of changes. This is a tricky balance because in order for a vote for or against a hardfork to matter, it has to be a vote on a single issue. If, for example, 10 important things are included, but 1 is contentious, it's difficult to hold back innovation on the 9 things just to make a strong point about the 1 thing. Some argue we should have more frequent hard forks specific to individual issues. Others have concerns about too many hardforks causing too much maintenance demands on exchanges who then would have to upgrade their Steem nodes on a regular basis. If that's too much work, they may consider it easier to drop support for Steem/SBD.
Are these concerns enough to prevent us from moving forward with Hard Fork 20 and working through any issues we find along the way?
I don't think so, which is why I'm currently signing on a v0.20.2 node. I have a backup on v0.19.12 just in case everything breaks.
This is probably a good time to outline my standards for upgrading to code supplied by Steemit, Inc. (or any other developer, for that matter). Ideally, I'd like to have more contributors to the code than just those employed by a single company and thankfully people like 
Code review the changes being deployed on Github. Look for anything obvious/confusing/concerning and ask questions of the team as appropriate. As you can see with this comparison between v0.19.12 and v0.20.2, that can be quite difficult to do all at once as this change has 2,657 changed files with 569,910 additions and 31,139 deletions:
Run the code on a seed node for a period of time and monitor the logs closely. Are there any changes, issues, or unexpected problems? If so, discuss with the other witnesses, community, and Steemit, Inc. developers and create Github issues as needed.
Run the code on a backup witness node that is not actively signing blocks. Again, monitor the logs and performance closely.
Now that we have a testnet, I'm also including running some tests for existing functionality against the testnet via my own node and through the testnet condensor. For Hard Fork 20, I also created a new account lukestokes.test, configured it as a witness, and signed some blocks.
Test new functionality on the testnet (such as the new rc_api plugin in HF20):
Ask application developers if they've tested their applications against the changes, and if they are comfortable with those changes. I could certainly do more of this and if you're a Steem application developer reading this, please know I'm always available to you on Discord, Telegram, Steem.chat, and more.
Discuss with other technical witnesses their own testing and confidence level with the proposed changes. Since DPoS requires consensus and we all want whats best for Steem, it can be helpful to get multiple perspectives and insights to inform your own opinion.
Ask questions of the Steemit, Inc. development team to determine their confidence level and figure out if there are specific areas of the code which they may be most concerned about (and which I should focus my testing).
So those are my standards as of right now, and they adjust based on the complexity of the code changes. Are they enough to catch every possible potential problem? Of course not. I've been doing software development since 1996, and I know, there is no such thing as perfect code. Sometimes you do the best you can, deploy, iterate, and improve.
I'm posting this right as Hard Fork 20 goes live in the code. That means a whole bunch of code previously hidden behind an "if HF20 = true" check will now run. What will happen? Let's find out.
Edit: Right as I tried to post this entry at 10:00am Central Time, I was greeted with this message:
Checking my account, it seems I have negative resource credits!
curl -s --data '{"jsonrpc":"2.0","id":1,"method":"rc_api.find_rc_accounts","params":{"accounts":["lukestokes.mhth"]}}' localhost:8090 | jq -r
{
"jsonrpc": "2.0",
"result": {
"rc_accounts": [
{
"account": "lukestokes",
"rc_manabar": {
"current_mana": -1384119289406338,
"last_update_time": 1537881810
},
"max_rc_creation_adjustment": {
"amount": "10000",
"precision": 3,
"nai": "@@000000021"
},
"max_rc": "57193602393"
}
]
},
"id": 1
}
It's going to be an interesting day. I'm trying to post again from my witness account.
