Thank you for this detailed write-up. This is the kind of post that saves people hours of frustration.
Your breakdown resonates with something I experienced just yesterday. I uploaded and updated a file on my server via SSH. Even though that file didn't have many active connections, a pm2 restart caused my entire server to crash. All functions stopped, and I couldn't even log in. I stayed up until 3 AM getting everything back online, and MongoDB was right in the middle of the mess.
What’s interesting is that when I talked to some friends in this space the next morning, around 90% of them had hit a similar issue that day especially those who had pushed some kind of update.
Your point about distinguishing between "the triggering event" and "the actual root cause" is something I wish I had understood before I started panic-repairing things. I assumed data corruption, when the real problem might have been in the service environment the whole time.
The two-step test you laid out, manual foreground start vs. systemd, is something I’m saving permanently:
✅ If it runs fine manually, it’s an environment problem.
❌ If it dies under systemd, check your unit file before touching the data.
The GLIBC_TUNABLES finding is particularly valuable. It’s a great reminder that a single environment variable in a service definition can be the difference between a stable node and a guaranteed segfault after every restart.
I’m going to audit my mongod.service unit today.
I appreciate you taking the time to document this so thoroughly. This is what makes this community worth being part of.