alt

With the recent update to steemd version 19.3, I had to do a lot of full node replaying, which can take anywhere from 1-30 days depending on the hardware and plugins enabled.

The full nodes I manage typically take somewhere between 2-3 days to fully replay with account history and all other plugins. The hardware isn't cheap either, it is about the best you can get without building your own configuration and self-hosting.

Full Node Configuration

AMD EPYC 24 Core CPU
512GB Ram
2x 960GB NVMe Gen 3 Data Center Drives
1Gbit/s Internet

This means anything that causes a replay is extremely time consuming and frustrating to resolve. Luckily we have two similar servers in a cluster when one fails I can easily remove it and replay it while the other picks up the load. If both have to be upgraded, they can be staggered and upgraded one at a time without downtime.

In the case of the recent emergency patch, both full nodes had to be replayed and this causes a lot of difficulties when trying to maintain service but also patch quickly. Not only do both nodes have to be updated quickly, most public full nodes are doing the same thing at the same time. A common practice with full nodes is to redirect queries to another node during a replay, this maintains service while allowing you to pull your node out of use. This is effective during failure or when you have other nodes you maintain, but when all full nodes are patching at the same time, this becomes ineffective.

During this wave of replays, I tested two new tricks that dramatically sped up my replay times and allowed me to patch and bring all four full nodes I maintain online much quicker than normal.

Follow Feed Calculation

The first is a trick I picked up from Hive account@gtg (thanks!) and it involves specifying a start time for steemd to start populating follow plugin feeds. Follow feeds are primarily used by a condenser (front ends like steemit.com) and are not used by most services that use public full nodes.

In my case, upon replaying my nodes, I told steemd not to process feeds earlier than 7 days ago. This is done with the --follow-start-feeds= parameter to steemd. It takes a unixtime stamp as a parameter, which is easily calculated using UnixTimeStamp.com.

Example

~/bin/steemd -d data --replay-blockchain --follow-start-feeds=1524355200

Combined with the next suggestion, my replay times went from around 2.5 days to about 18-19 hours. While replays are still extremely painful, this is a huge improvement in recovery time from a node failure or patch that requires a replay. Every day that passes increases the potential replay time of a full node (including witness and seed nodes).

alt

Remove block_log from OS cache

The other trick is a suggestion from Github user theoreticalbts is to drop the block_log file from the OS cache, giving more ram to the shared memory file. The block_log is currently 101GB and growing very rapidly. This eats up a significant portion of ram even on a 512GB server.

There are two ways to accomplish this, both have different implications. Further testing needs to be done to see which one is better overall.

During a replay, run the following script to purge the block_log from the OS cache. This will maintain the maximum amount of ram for steemd and it's shared memory file.

while :
do
   dd if=blockchain/block_log iflag=nocache count=0
   sleep 60
done

You will need to adjust the path to your block_log if you do not run it from the blockchain directory. Leave this script running while you do a replay. I am not entirely sure how much of an improvement this step alone made as further testing is needed to isolate the improvements by this trick alone.

There is another approach Hive account@bhuz uses to accomplish the same thing, but in addition to purging the block_log this will purge the shared memory file. This may provide additional benefits of clearing dirty pages even outside of replays, but further testing is needed.

This can be done with the following command:

echo 3 | sudo tee /proc/sys/vm/drop_caches

Conclusion and failures

In the recent replays, I used both tricks to drastically reduce how long replay took.

Unfortunately, I screwed up and accidentally built some of the nodes using low memory (setting I only use on witness nodes) as I was upgrading 9 nodes during the course of 24 hours. Upon realizing this, I fixed it and restarted the replay.

What I didn't realize, is low memory setting persists even after turning it off with a new build. (Thanks Hive account@anyx for pointing this out) Which caused some weird issues with the node until I realized it was in low memory mode even though I built without low memory. The solution for this is to delete the CMakeCache.txt prior to doing a rebuild.

It's been a trying few days, but learned a few interesting things and have a few new tools to dramatically speed up recovery time in future replays. While most people this isn't very interesting, those running full nodes may save some sanity with this advice.

Thanks again Hive account@gtg, Hive account@anyx, Hive account@bhuz, and Github user: theoreticalbts (I suspect there is a Steem user behind this account).

_{_X48EJ}