Witness Update - Acknowledgement of error and changes to @curie witness operation structure (April 2nd, 2018)

This witness update is an open acknowledgement of a major error made on the Hive account@curie witness and an outline of the changes in our witness operation structure that have been put in place to ensure there is not a repeat.

• On April 1st, 2018 an error was made by Hive account@curie witness operator Hive account@locikll while upgrading to a new witness server. The Hive account@curie witness began double producing and causing block collision.
• This event highlighted a failing in the witness monitoring protocols in place for the Hive account@curie witness. The initial error was a major mistake, and it was compounded by the fact it took some hours before the mistake was fixed and Hive account@curie witness recovered.
• We take this matter very seriously; operating a top 20 witness on the Steem blockchain is a serious responsibility and we openly acknowledge failing to live up to this responsibility. The following steps have been taken to ensure this does not happen again:

1. Hive account@locikll has been replaced as primary witness operator. Hive account@markangeltrueman (UTC time zone) will take over as primary witness operator. Mark is Curie's lead developer for several ongoing development projects and in his full time job heads up a team of third line support engineers for a large payments company after spending 15 years as a software engineer.
2. Hive account@locikll will continue as backup witness operator to provide additional monitoring and coverage as needed; primary and backup witness operator will coordinate schedules so at least one is providing coverage on every day. Hive account@locikll will continue to manage the RPC node and other non-witness infrastructure etc.
3. A "four eyes" QA process will be added to any changes that are to be made to the witness node outside of emergency incidents. No changes to witness node config etc. will be made by either primary or backup witness operator without the other operator checking it first.
4. Automated monitoring will be implemented to notify primary and backup witness operators via SMS when blocks are missed.
5. A backup monitor position has been created with direct phone contact to the primary and backup witness operators. Hive account@carlgnash is assuming the duties of backup witness monitor; this gives Curie witness 24/7 human monitor coverage as Hive account@carlgnash is Pacific coast US time, with waking hours to overlap UTC night time. This is intended as a final stopgap to ensure the fastest possible response time at all hours.

• For those new to Curie, please follow Hive account@curie, and join us on Discord: https://discord.gg/jQtWbfj
  For all witness related queries, there is a dedicated channel on the Discord server: #witness-enquiries channel. Please address any Hive account@curie witness related questions or concerns in this channel.

• To learn more about Curie operations, please read the Curie Whitepaper at curiesteem.com

• Follow Hive account@curie's votes to support the authors. Please consider following our trail and voting for curated authors. If you are a SteemAuto user, Hive account@curie is an available trail to follow.