Non-Volatile Memory Express (NVMe) is becoming the primary protocol for interconnecting modern storage technologies both within storage arrays and between storage arrays, storage networks, and...
So one of our field engineers reached out to me because they had a power outage of some sort and their vCenter appliance failed to boot with these errors about starting up the services.
So this is a common that has happened many times as seen from these KBs and community posts:
Simple blog post, and this is really more for my own future reference. But could help you if you don’t have the common mount problems in the forums etc.
Unlike a lot of these scenarios though. /dev/sda3, or sda2, or /dev/mapper/log_vg-log had no issues. They were all mounted and online.
I made them read-only by either editing the /etc/fstab from defaults to remount,ro and rebooted then tried fsck, but they came back clean.
So I went to the log and ran journalctl -xb
Then I saw the error:
Looks like the mount that is corrupt is not the typical one.
Logged back in and ran fsck -y /dev/mapper/db_vg-db
Yup lots of errors. Rebooted and good to go.
So the moral of this story–don’t just try fsck in whatever the mount communities say and hope that is the right one. That might not be the one you are having issues with. Look at the journal logs–will save you some time. This will allow you to fix the problem quickly without guess work and trying to mess with fstab or remounting etc. Which is what I tried first.
It is a fairly long log, but only takes a minute or so to page through–keep going until you see a fsck error towards the end–this is likely your real issue. Errors before that are often innocuous.