Witness Monitoring Bash Script (Update 2)

I've been working on some improvements to my witness monitoring script which I first introduced here and then updated here. The script relied on the log output which has gotten a bit chatty since HF20. I started testing out v0.20.6rc1, and it also has some resource credit related log output which can lead to false positives if the logs are parsed for increasing block numbers (as the previous script did). I know you can adjust the logging levels if needed, but I also like to see what's going on there.

The updated script now uses the JSON RPC to get the last_irreversible_block_num directly and ensure it's increasing. The nice thing about this approach is it would also alert you if there was a consensus issue on chain, even if your node is working correctly. I also added an improvement to resend the text message every thirty minutes. Previously, if the notification_sent.txt file existed, it would just exit without another notice. This approach also cleans up after itself as soon as the node is working properly.

This script does require the json_rpc plugin, but with the new resource credit system, I think it's helpful to have that turned on anyway so you can monitor things there (as there is no cli_wallet support for that data). Using Steem-in-a-box, you can keep those RPC ports unexposed to even the host machine, so they can only be used within the docker image.

If you were using my script before and noticing false-positive notifications, this update may be of use to you.

#!/usr/bin/env bash
if [ ! -f last_block.txt ]; then
    echo "0" > last_block.txt
fi
if [ ! -f skip_count.txt ]; then
    echo "0" > skip_count.txt
fi
send_notice=false
last_saved_block=$(< last_block.txt)
skip_count=$(< skip_count.txt)
source .env
# Note: if running without cron, add in -it after exec
latest_block=$(docker exec $DOCKER_NAME curl -s http://localhost:8090 -d '{"jsonrpc":"2.0", "method":"condenser_api.get_dynamic_global_properties", "params":[], "id":1}' | jq '.result.last_irreversible_block_num')
# to test, uncomment this next line:
#latest_block="failtest"
if (( latest_block > last_saved_block )); then
  echo "We're good. Latest block is... $latest_block"
  echo $latest_block > last_block.txt
  echo "0" > skip_count.txt
  if [ -f notification_sent.txt ]; then
      rm notification_sent.txt
  fi
  exit 1
fi

if [ -f notification_sent.txt ]; then
    send_notice=false
    skip_count=$((skip_count+1))
    echo "$skip_count" > skip_count.txt
    echo "Skip count modulus: $(($skip_count % 30))"
    # Send a text every 30 minutes
    if (( $skip_count % 30 == 0 )); then
       send_notice=true
    fi
else
    send_notice=true
    echo -e "Something went wrong. Here's what we got for the latest block: |$latest_block| \r\n" > notification_sent.txt
    latest_log_entry=$(docker logs --tail=10 $DOCKER_NAME)
    echo $latest_log_entry >> notification_sent.txt
fi

if $send_notice; then
    date >> logs.txt
    cat notification_sent.txt >> logs.txt
    echo "There seems to be a problem."
    echo "Last saved block: $last_saved_block"
    echo "Latest block: $latest_block"
    python sendnotice.py
fi