Running a Witness: Is it as Easy as it Sounds? 🤔

It's been a month and a half since I set up my witness node. I can tell you one thing, it's not as easy as it may sound. I've seen numerous posts encouraging people to set up a witness server, with the promise of making big money out of it. The truth of the matter is, running a witness is not a plug-n-play process. It's a responsibility, not a game. It requires personal financial investment, dedication, and most importantly technical skills. Let's go into some details to clear out some of the misconceptions.

This is not what being a witness is all about
(we wish it was, but it's not)

This is

The Hosting

To run a witness, you need to have your servers hosted somewhere (VPS, dedicated, colocation, whatever suits your budget). Yes, servers, we'll get to that in a minute. The steem daemon is a very finicky process, it requires lots of RAM, low network latencies, and preferably no other running process on the box. The first server to set up is the witness node. It is the main powerhouse, with two roles: host the blockchain and create the blocks (the steem transactions). The witness IP is kept secret, for security reasons. The second server is the seed node, it's a publicly available server with the role of distributing the blocks. The more seeds spread across the world, the better, because they help reduce the latencies. The third type of server is the full API node; witnesses are not required to run one, but if they chose to, this monster requires much more RAM (or a fast RAID SSDs), i.e. more expensive.

The second consideration for a successful setup, is to have a fail-over mechanism, so in case something goes wrong with the node, you can have a backup server to keep generating blocks for you, or you will be missing blocks, which is bad. Of course, there's the option of disabling the node's signing key in your steem account when things go sour, but the downtime can cost you precious blocks.

As you may know, witnesses are rewarded with Steem Power for every block they generate. It's around 1 SP (last time I checked it was 0.948 SP). The Top 20 witnesses are rewarded 1/5th of that amount, so ~0.190 SP, because they generate more blocks per day. I won't get into the details of the witness economics, there's plenty of other posts that covered that. Needless to say, to be successful, you need to generate enough rewards to cover your monthly expenses, or you will operate at a loss.

And finally, securing your network goes without saying 🔒. In fact, it's important to secure it. You don't want to be hacked (the configs contain private keys) and you don't want a brute force attack on your port 22 that brings down your server, do you? In fact, changing it helps reduce missed blocks.

The Votes

Running the server(s) is one thing, the other crucial consideration is the rank. When people vote for your witness, their VESTS (Steem Power) are taken into account. The amount of votes you receive is irrelevant. For example, 1 person's vote with 1000 VESTS is equivalent to 1000 votes with 1 VEST each. So if you have a thousand minnow friends ready to support you, the guy next to you with a few whale votes will eclipse your rank. The rank is important, the higher it is, the more probability you have to generate a block on the next scheduled round. For example, I'm at rank 56, which gives me about 15 blocks/day, while someone in the Top 20 generates ~1371 blocks/day (with 1/5th of the reward). As you can imagine, people at the top have way more pressure to run a reliable setup than those at the bottom. At worst, I could miss 15 blocks/day, but for someone at the top 20, that would be 1371 blocks if there's no solid backup(s) and fail-over mechanisms.

The Dedication

After you set up the server(s), and got the votes to be in a comfortable rank position, what next? Well, depending on your projects and involvements, keep it up. However, blogging and running projects can eat a lot of one's time. You also must assume your witness role by hanging out with the other witnesses! If there's a generalized issue occuring (like the bandwidth problem) and an update coming up (like 0.19.1), you need to be in touch and be up to date with the rest, not spending your time on a cruise ship. Reaction time is a factor, if you can't react when needed, you'll lag behind, possibly affect the rest of the network, and ultimately lose your rank.

Of course, reaction time is also important when managing your setup. For example, last week my witness server crashed for some mysterious reason, it was on the provider's end, because a few other witnesses crashed too. I was quick to react and manage things, which prompted me to improve my set up a few days later. Other witnesses couldn't do that, for a lack in the things I discussed. Surprisingly, a top 20 witness also experienced simultaneous failures and a lot of missed blocks, which demonstrates that nobody is immune to server problems. If it can happen to the best, imagine what can happen to a non-knowledgeable witness.

Part of the dedication is the constant learning. Therefore reading, interacting with others and understanding the system's operation is part of the witness job. Paying attention to the other witnesses concerns and listening to the more experienced witnesses' wisdom is a source of knowledge that prevents future problems.

Conclusion

A lot of the new witnesses jump onto the witness bandwagon without being aware of the important requirements I mentioned, because nobody tells them about it. It's possible to run only the witness server, but if you do, you risk more when bad things happen. I threw a lot of technical words on purpose because this wasn't meant to be a newbie guide. If you don't understand what I was talking about or don't know how to change dir on linux, then you should probably reconsider running a witness. Some people choose to hire a tech to manage their witness, which is fine, but it adds to your expenses. Reaching the top 100 can be done with enough votes for a newbie witness, banking 1-2 blocks a day. Aiming for the top 50 or the top 20 (even harder), doesn't and won't happen overnight. It takes time, patience and hard work. Are you up for the task?

Witnesses are the backbone of Steem, they host the blockchain in the background, and I'm happy to be part of that.