I am really happy with the outcome of hardfork 20. That is obviously a crazy thing to say after all the chaos that occurred, but let me explain.

Witnesses' Role in Hardfork Approvals

Prior to HF20, there was very little focus put on the witnesses' responsibility to test and review code before accepting a hardfork. There has also not been an effective way for witnesses to advocate/push for change without standing in the way of the progress that the development team is making.

After the events of HF20, the conversation around witnesses' role in the approval of hardforks has dramatically shifted. This is important, because the next big hardfork (SMTs) is going to be 10x as complicated as HF20, with a lot more things that can go wrong.

Also, the stakes are going to be much higher.

It is great that HF20 brought attention to many of the issues that were there, so we can have a conversation about how to fix them.

SMTs

SMTs are going to be getting a lot of attention from people outside, and if the launch of SMTs is not successful - a lot of people are going to notice. Developers may end up taking their business elsewhere (to another blockchain), and they definitely won't be telling their friends to use Steem.

If SMTs are successful though, it has the opportunity to draw in a lot of new businesses that are interested in building on Steem. These businesses will have the potential to draw a lot of capital and new users to Steem.

We want the launch of SMTs to be a huge success, and in order for that to happen - changes are needed.

Hardfork Approval Standards v0.1

This post is following the lead of the many other witnesses who have started the conversation about what changes are needed in order to prevent another incident like we had with hardfork 20.

Below is a draft of the standards that I plan to use for deciding whether to approve or deny a hardfork going forward. While no set of standards will 100% prevent another issue from ever happening, I believe that by following these standards we can significantly reduce our risk of another failure by increasing our chances of uncovering issues before they reach the mainnet.

Keep in mind, these standards are an initial draft. I am looking for feedback. I plan to use the feedback I receive to create my "version 1.0" standards that I will actually follow for approving the next hardfork.

Hardfork Proposal

The development team should create a detailed proposal describing the changes they are planning to make. The proposal should clearly explain what they are planning to change, and the reasoning behind it.

The stakeholders should discuss the proposal and express their views for or against it. Note: All users who hold SP (even a small amount) are considered stakeholders.

As a witness, I will evaluate the proposal and feedback from the stakeholders. I will publicly express my views for or against the proposal.

Note: It is important to make the distinction that my support for the proposal does not mean that I will ultimately accept the hardfork. There are many other factors that may ultimately lead to my rejection of the hardfork before the final vote is taken.

If I am fundamentally opposed to the changes being proposed, I will make it very clear that I will not be accepting the hardfork and provide my reasons why. I will do this as early as possible, so that:

1. Developers have the opportunity to adjust their proposal if needed.
2. Developers do not spend time coding something if there is not consensus on it's approval.
3. Stakeholders can adjust their witness votes if they disagree with my position.

No Surprise Changes

If additional functional changes are to be included in the hardfork which were not part of the original proposal, or if changes that were part of the original proposal are significantly changed or removed, the development team should communicate the changes as a proposal amendment. The amendment should follow the same review process as the original proposal.

Development Tracking

There should be an issue in GitHub for every change that is being made. The issue should have an appropriate user story that explains what is changing and why the change is being made. All pull requests should be linked to an issue. Witnesses should be keeping up with the development as it progresses in GitHub.

The development team should also be able to provide the witnesses a list of all the issues in the hardfork, so that witnesses can do an appropriate comprehensive review of all the changes included in the hardfork.

Questions and Concerns Addressed

Witnesses and stakeholders should have the opportunity to ask questions about specific changes via their issues in GitHub. The development team should respond to all reasonable questions and concerns that are documented in the GitHub issues.

Automated Tests

The Steem blockchain code already has extensive automated testing in place (which the development team maintains), that verifies updates work the way they are intended, and new changes don't break existing functionality.

These automated tests should continue to be updated to account for the new cases that need to be verified as functionality is changed. Witnesses and stakeholders should open issues if there are test cases that are not accounted for by the automated tests when new changes are checked in.

Test Environment

It is critical that the community be provided with a test environment to sufficiently verify all of the changes before they go live in the mainnet. There are many aspects to this, which I will outline below. It also may be necessary to run multiple testnets in parallel in order to provide sufficient coverage of all the different scenarios in the allotted time.

MVP Verification

Testers must be able to verify the "minimum viable product" (MVP) functionality of all changes included in the hardfork on the testnet.

Testnet Tool Infrastructure

This is one of the most crucial aspects of being able to properly test on the testnet. We need our infrastructure of tools on the testnet to match to what is there on the mainnet.

• This includes a condenser (steemit.com) instance, a block explorer (i.e. testnet.steemd.com), cli_wallet capabilities, and SteemConnect (i.e. testnet.steemconnect.com).
• There should also be instructions provided on how to use all of the developer libraries (Steem Python, Steem-JS, Beem, etc.) to connect and interface with the testnet.
• If there are new blockchain API methods that are exposed, tools and/or libraries to access and use those methods should be provided.
• Ideally all third-party websites and tools (SteemPeak, Steem Monsters, Vessel, Voting Bots, SteemAuto, etc.) should create a testnet or sandbox version of their products that users can use to experiment with on the testnet.

Test Plan

In the steem.chat witness channel, Hive account@ned asked:

My proposal for this is that the development team should be responsible for providing the community with a test plan. This test plan should be a documented series of tests that they performed prior to the release of code in order to verify the proper functioning of every change. The test plan should be executable by community testers on the testnet.

Auditors / testers should be able to verify the proper functionality of the hardfork by executing the test steps in the test plan on the testnet.

Community members should also be expected to perform additional tests if they think of use cases that were not covered by the documented test steps.

Pre-Fork Parallel Operation Test

There is a period of time where the new hardfork code is installed on some nodes, but not all, and the hardfork time has not occured yet. Testing this scenario should be included in the test plan in order to ensure that the "parallel operations" period does not cause any unexpected issues, including unplanned forks.

Simulate "Real World" Conditions

As much as possible, the testnet should simulate the real world conditions that will take place on the mainnet. If there are real world conditions that cannot be accurately replicated on the testnet in order to fully verify the proper operation of a change before it goes live, the limitations should be properly communicated to the community ahead of time, along with the risks involved.

Sufficient Time to Test

The testnet should run for a sufficient amount of time in order for testers to be able to verify all of the changes. For changes that require time for the test to play out (such as verifying a new post receives proper payout seven days after it is created) an instance of the testnet must run uninterrupted for long enough to verify the end-to-end functionality of the change.

If patches are released that invalidate the results of previous tests, additional time should be given in order to re-verify the functionality that needs to be re-tested.

Testers have a responsibility to use the time that is given to test optimally. (i.e. do not wait until the night before the hardfork to start testing and complain that there is not enough time to test.)

Witness Participation in Testnet

Witnesses should be expected to participate in the testnet by running a block producing node. This will allow them to verify the fork occurs as expected on their node, as well as verify any witness-specific functionality included in the fork. Along with this, witnesses should be expected to supply a price feed and submit reasonable witness parameters to facilitate testing. Stakeholders should vote in the witnesses on the testnet who are running testnet witness nodes.

Stable Build Running for 14 Days

The testnet should have a stable build running for at least 14 days prior to the hardfork with no critical issues found. If a critical issue is found that requires a patch to be deployed, the 14 day countdown should be restarted.

Documentation Updates

It should be expected that the documentation for the Steem blockchain be kept up to date along with any changes that are made.

This includes:

• New API methods are documented in developer portal.
• Changes to existing API methods are documented in developer portal.
• New steemd parameters are documented and explained.
• All changes to the config.ini file are documented and explained.
• Any updates to the build process are explained in the release notes.
• Issues are created to update the whitepaper, bluepaper, and steemit.com FAQ as needed.
• Recipes are provided in the developer portal for common expected tasks (example: how to calculate the RC cost of a transaction).
• Complicated new functionality with lots of moving parts (such as the new RC system) should have a wiki article or Steem blog post which explains how it works (example: RC Bandwidth System).

Community Expectations

The Steemit dev team has an important/crucial role to play in making this a success, but the witnesses and stakeholders in the community have a large responsibility in this as well. There are several things that WE should take responsibility for, including:

• EVERYONE should be expected to test on the testnet. Testing is a team effort. The more people we have trying various things in the test environment, the more scenarios we will cover, and the more issues we will find.
• Stakeholders should be expected to create issues in the appropriate GitHub repository if they find something wrong. If creating an issue is not something they are comfortable/capable of doing, then they should reach out to a witness or other community member who can create one on their behalf. If an issue is critical, the the severity should be properly communicated in the GitHub issue.
• Witnesses need to let the development team know loudly, clearly, and as early as possible if something is a showstopper. In other words if there is an issue that needs to be fixed in order for a witness to approve the hardfork, they need to make it very clear that they are not voting for the hardfork until the issue is fixed. Both Steemit and the witnesses should have a clear picture of all the issues that exist based on whatever has been reported in GitHub (see above).

Final Veto Authority

Ultimately witnesses have final veto authority on all hardforks. Even if all of their documented standards are met, if they uncover something that they feel presents a threat to the network/stakeholders which was not covered by their standards, they can always deny or delay the hardfork by voting 'no'.

Witness voting standards are not meant to inhibit this ability, but witnesses should do their best to make it clear ahead of time (via their standards) under what grounds they will reject a HF.

IMO, communication is the most important thing. If a witness is going to vote 'no' on a hardfork, they have a responsibility to make their position known as early as possible, so that the development team and stakeholders can adjust accordingly with the least amount of wasted resources.

Evolving Standards

It is expected that these standards will continue to evolve over time. While the list I have above is not going to prevent every possible issue, it is a big step forward from where we currently are. Most importantly, they are standards that I believe are possible to meet by the time we have the next hardfork.

Feedback Requested

Whether the stakeholders are on board with supporting witnesses' standards, and whether the development team is willing to adapt to the standards witnesses present, are two critical components to making this a success. Hopefully we can find the right balance in order to get the right parties on board.

Please provide your feedback in the comments below.