It was bound to happen...
Things were pretty smooth. A hiccup every now and again, but generally, we've been enjoying smooth sailing on the blockchain for the past few years of building on it.
Then...Last week happened!
We'll go over a few of the issues, some fixes that are in place and how we are going to grow and learn from this but let's start with the elephant in the room:
The Hive Engine dev team released an update for light nodes and domains (v 1.7.2 to 1.7.3), so we decided to be proactive and get our servers up to date with the changes. Things didn't work out that smoothly. And for the better part of the weekend up until a few hours ago, our stuff basically went offline.
Blain has been working around the clock while in the middle of a family move to try to get things sorted and as soon as we found the fix, other witness servers started mentioning there were issues. Hey let's just say we are ahead of the game...But the damage was done.
ListNerds Was Offline
This was a glaring issue that most people noticed when LISTNERDS rewards stopped being sent. There are a few days of rewards that should be sent out tonight or tomorrow and things will be back to normal soon. The good news, we were able to do work with the server to improve deliverability and speed of the queue. But still, the site just wasn't working properly over the weekend.
HivePay Swap Had Issues
This is the biggest issue. Because of the issues with our server, the SWAP feature of HivePay basically stopped working and people sent SWAP but did not get refunded immediately. And then once our server got back online, it actually double paid the refunds.
Which drained HivePay Swap's liquidity. We are sorting through all the data, but from what we can tell, from around block # 65413973 SWAP basically double refunded customers and the pool was drained. It's our bad, and we will make sure we get liquidity back, but we are going to tag everyone that SWAP refunded over the past few hours and will ask for anyone that may have been double refunded to send the SWAP or Hive back to
We know this isn't ideal and we do apologize for this, but if we can recoup a chunk of the liquidity it will help Hive Pay Swap get back to normal and functioning again.
As for future issues, Blain has found it and put the fix in place so the double refunds don't happen again. Regardless of what happens, the CTP Team will make sure liquidity is back in the SWAP!
Apologies in advance for tagging so many people, but we just want to make sure you check to see if the funds got sent in error:
Again, apologies for tagging everyone, but we hope to see if we can get some help to recover some of these funds.
So all in all....
Growing pains. And a massive inconvenience for our members this past weekend. Which we cannot apologize enough for. We are confident the fixes in place will make sure any updates by Hive Engine in the future will not hurt our services and platforms.
Thank you so much for your patience and understanding and we hope to put this behind us and continue to grow on the blockchain!
Jon, Blain and Eliana
The CTP SWARM
P.S. If you have any questions at all, or need clarification, be sure to join our Discord server here! We are also transitioning all our Telegram communications into Discord going forward, which will help us serve our community much better!