I've been told none of this falls on me (and I agree), but I'm still left with a bad taste in my mouth.I'm torn on Mongo.

As a developer, using Mongo makes life easier. It's literally spin up and go with a container. But after that point, after the development process, things get tricky. As a sysadmin, it causes nothing but pain and suffering. This isn't new, Mongo has had a history of stability issues, but I'm getting ahead of myself.

Part of what I do is help run and maintain microservices on-demand for researchers. The caveat for them is that what we're running is a research system and, not only can it go down at any time, we do not back up data. Now, that's not to say we don't do our best, and we absolutely try to go the extra mile to help someone out. However in this case, we're left with an unhappy customer.

A node that these on-demand services resides on had a hardware failure (faulty CPU) and everything on said node died hard. Recovery happened as VMs were evacuated, however the damage to the Mongo instance had been done. Three of WiredTiger's metadata files (WiredTiger.wt, WiredTiger.turtle, SizeStorer.wt) were corrupted. Mongo would try to start, WiredTiger detects an invalid file and dies. The end.



Research had shown that you can actually cut a ticket with Mongo and they'll attempt to restore your files. However in our case things weren't able to be recovered (supposedly due to how our instance crashed.) I've now been left with informing said researcher that

A) They should have run replica sets
B) They should have taken backups
C) There may be a way to retrieve the data, but it will take forever

I've been told none of this falls on me (and I agree), but I'm still left with a bad taste in my mouth. Look at other data stores (SQL and otherwise) and you don't see something that corrupts an entire instance with a hard-reboot. You don't see a community that thinks it's okay that the correct solution is "Run in a replica-set, take frequent backups." They can and should do better. Now that my sysadmin rant is done, I'm gonna go work on this side project I have that's using Mongo.



I'm a hypocrite.