As some of you may have noticed, my witness has been down for a number of days. There was a server seizure by the police at my hosting provider, which completely took down my witness node. I decided to take matters into my own hands...
First I would like to apologize for my lack of communication. I thought I would have the witness back up and running long ago, and would do an update after things were back to normal. Second, I would like to thank the individuals out there who care enough about their witness voting to see my disabled witness and use their votes to keep the network secure. If everyone on Hive were like you guys, this place would be better off. And now I would like to tell you the journey I've been on, so others can enjoy a laugh at my misery or maybe even learn some lessons.
VPS Hosting problems
As I said above, the server my VPS was hosted on had been seized by the police. I assume someone else on the machine was doing some illegal stuff, but as the physical hardware is all shared, they can't leave my machine and only take theirs. This compounded with other problems many other customers were having regarding drive failures and networking issues led me to consider other options.
Self Hosting problems
I have an old server that I once used as a witness back in the St**m days. It's a dusty old R710 with tons of RAM and an ancient Xeon processor that uses as much energy as it does time to get the job done, but it gets the job done. However, after booting it up for the first time in years, it was not recognizing the HDD array. I first thought maybe it was my old Perc6 RAID controller, and proceeded to change that out for the original one I got with the server. After more fighting and fussing, I discovered 3 of my 6 HDD drives were completely shot. As they were bought used, and all the same age, I went ahead and ordered 6 brand new SAS HDD drives to replace them. Nothing too fancy, but will get the job done. Finally I got ESXi to boot, provisioned a VM, and started to get to work on getting a witness server up and running.
This is where I ran into a series of problems that took me days to figure out. As I'm not very smart when it comes to server hardware, I thought I was doing something wrong on the software side when my replays errored out. Long story short, it appears that the CPU instructions are too outdated to run this code reliably, or at least that's my best guess. So I decided to buy a new(er) server...
Shipping Hell
I found a decent server on Ebay that fit the bill. A bit pricey, but solid specs and no HDDs, which as mentioned above I already had bought. I clenched my wallet and had it shipped immediately. When it arrived, it looked like the UPS guy had put it through a meat grinder. The seller had properly packed it with expanding foam and a proper box, but the shipping company left me with a disaster to deal with. I declined to accept the package in its current state, and am now dealing with Ebay and the seller.
As that was the only server the seller had in that exact spec/price point, he can't send me another one. Which leaves me on the search again to start this process all over again. If I was a consensus witness, I would already have a backup machine going to keep the network secure, but as a backup witness with many others reliably witnessing blocks, it doesn't hurt the network to stay down while also saving me money to afford this expense.
A Question for the community
I'm going to resolve this very quickly, but since I'm in this situation I figured I'd ask for some insight. If anyone reading this can link me a good deal on suitable hardware (ECC preferred) to self host a witness server, please do so in the comments. Thanks for reading.