Last year I wrote a script to monitor my witness nodes and then updated it again 10 months ago. Today I updated my producing node and my seed node to v0.19.10 while also pulling in the latest changes from @someguy123's steem-docker v0.19.10 branch. One of the changes I noticed and appreciated was the addition of -f to the logs command so it would continuously follow the logs instead of just spitting out the last 30 entries. This is something I've done manually many times, so it was a welcome addition to the script.
Unfortunately, I forgot my monitoring scripts were also using that command and expecting it to exit each time instead of continually follow the logs. My monitoring script is on a cronjob to run every minute which means new processes were being spun up every minute. Yeah, my servers were not to happy with me today, but thankfully I was able to find and fix the problem (while having to kill off a ridiculous amount of processes) without missing any blocks or melting down my servers.
If you're using my monitoring script, you can get the latest version here as a Github gist. The main changes:
last_saved_block=$(< last_block.txt)
source .env
latest_log_entry=$(docker logs --tail=1 $DOCKER_NAME)
The block number position also changed to 10 instead of 11:
latest_block=$(echo $latest_log_entry | awk '{print $10;}')
And at the bottom:
echo $latest_log_entry >> notification_sent.txt
As I mentioned yesterday, v0.19.10 has just been released. There have been a few witnesses testing appbase for some time now and working through bugs, so I felt confident enough to test it out myself. After letting it run for a while on my non-producing node and my seed, I decided to start producing on it as you can see here:
My secondary node which isn't producing right now is still running on v0.19.5 in case any unexpected issues come up. Once more witnesses feel comfortable with v0.19.10, I'll upgrade that one also.
This post is mainly for other witnesses, but if you want to know more about what witnesses do, please check out this post I wrote last year.
If you're using my notification script, please make sure to update it before pulling down the latest steem in a box. If you're a witness, you really should have some scripts letting you know about the health of the chain. When the Steem chain froze on July 2nd, I was one of the first to know about it and respond thanks to my little script. :)
