One of my primary tasks at EOS BlockSmith is node maintenance and assisting with the spinning up of new EOS.IO chains. I think that we can all agree that once IBC is properly implemented, horizontal scaling of the EOS ecosystem will become more mainstream. That being said, it is good to see that more code forks are being spun up to fit the numerous business cases out there.
I'm not here today to praise or call out any specific chain. Rather, I hope to raise a few specific things that hopefuls should consider when they decide to implement their own business models by deploying a code fork. As such, the target audience of this post will be the workflow designers, rather than the users of the chains in question.
Coordination and Communication
As with any coordination effort, communication is key. While the levels of publicity for various projects can differ from one business model to the next, the organizations who are behind the spinning up the new code forks need to ensure that end-to-end communication is well thought-out and properly organized. There are three types of communication channels that should be considered.
Intra-Team Communication
Teams should only have one social media platform on which major decisions are made. The choice of platform should be decided based on a) the quality of the platform and b) the percentage of the target audience that is using the platform. For example: chains that have a base that largely comprises of gamers should consider platforms like Discord, while chains that have a business model more closely geared to security should look into using Keybase. There are several to choose from that are used across several chains already--Keybase, Discord, Telegram, and Slack primarily. The important thing is that there should only be one where decisions are made so that the community doesn't split among the different platforms.
Public Announcements / Discussion
Unlike Intra-Team Communication, public announcements and discussion should be spread across as many social media platforms as possible (if the chain is supposed to go public). In addition to the platforms mentioned above, organizers should consider consistently posting on Medium, Steemit, Reddit, and even Facebook and Twitter. Chains that wish to onboard block producers should make steps to join the chains easily available directly on a responsive website. Dedicated PR personnel are key to making a good impression in an ecosystem that is so community-based.
Developer Resources / Documentation
A separate platform specifically geared to resources and documentation is a must. This can be as simple as a public account on Bitbucket or GitHub (for you Microsoft-lovers) that hosts the code fork, various onboarding scripts, and documentation. Developers and systems administrators need easy access to peer lists, genesis JSON files, and the modified EOS codebase. All of these resources need to be easily discoverable... and social media platforms are not conducive to the discovery of these resources. A link should be readily available on the website.
Testing
One of the true tests as to whether a chain launch is going to go off without a major hitch is the extent that the code fork was tested before launch. All chains that have undergone any code change, or ever plan to undergo a code change, should have a testnet. Nearly every chain that has failed to do this has failed to launch on time or has one or more false starts. "Internal testing" doesn't cut it--interaction from "clients" (in this case, block producers and users) is mandatory to determine if a chain is going to fly or fail. Murphy's Law is wide-spread in the realm of Software Engineering; to presume that it doesn't apply to EOS.IO is simply hubris.
Launching the Chain
The overall organization of a team is indicated by how smoothly a code fork launches. As mentioned before, it is paramount that the chain is tested before launch. That being said, proper coordination on launch day is just as important. Block producers are international and it can be hard for all parties to adhere to a hastily-made time schedule. Here are some tips to ensure that the chain can kick off according to the previously-defined timetable.
- Action items should be scheduled up to a week in advance. Day-of plans are unacceptable except for in the case of an emergency. Most emergencies can be avoided if procrastination is eliminated.
- The main organization launching the chain should consider launching 21 (or an appropriate number) of temporary block producers so that the chain can run without losing consensus.
- The 'eosio' producer (or, renamed master producer, if applicable) can be paused. The chain should go live before the public launch date and be paused to allow all other block producers to sync up without making "catching up" necessary. Resuming the chain should be planned in advance so that all block producers can register and be added to the schedule (and then the temporary block producers mentioned above can be phased out). Failure to perform these steps can easily lead to a lack in consensus.
Security Considerations
Best practices need to be implemented from the get-go. I'm not really going to cover the low-level security practices that systems administrator need to consider, but organizations launching code forks should consider the following:
- Initial block producers should run under a VPN. Some code forks have utilized Wireguard for their VPN solution. It's incredibly new, but it seems to have worked quite well for some of the more popular forks.
- Initial block producers should consider DDoS-mitigation services like CloudFlare and patroneos, and load balancers like haproxy.
- Best practices in wallet security should be practiced. It should be encouraged that block producers adhere to standard EOS.IO security practices, like ensuring that "owner" and "active" keys are different, ensuring that there is a separate "claim" permission, and ensuring that block producers don't use or keep their wallets on the machines containing their nodeos instances.
Final Thoughts
I'm sure that the above points would appear common knowledge to most who are deep in the EOS.IO space, and certainly I only skim the surface as far as best practices go, particularly in the realm of security. That being said, one might be surprised in how disorganized certain chains are run. I hope that these notes are considered by any who plan to launch code forks in the future.