One of our field engineers reached out to me because they had a power outage of some sort and their vCenter appliance failed to boot with these errors about starting up the services.
This is common, as seen from these KBs and community posts:
- https://communities.vmware.com:443/t5/VMware-vCenter-Discussions/VCSA-6-5-Failed-to-start-File-System-Check/m-p/950475
- https://cormachogan.com/2015/06/03/fsck-of-vcenter-server-appliance-6-0-partitions/
- https://communities.vmware.com:443/t5/VMware-vCenter-Discussions/vCenter-Server-Appliance-failed-start-up/m-p/1393907
- https://kb.vmware.com/s/article/2147154
Unlike a lot of these scenarios, /dev/sda3, or sda2, or /dev/mapper/log_vg-log had no issues. They were all mounted and online.
I made them read-only by either editing the /etc/fstab from defaults to remount,ro and rebooted then tried fsck, but they came back clean. So I went to the log and ran journalctl -xb
Then I saw the error:
Looks like the mount that is corrupt is not the typical one.
Logged back in and ran fsck -y /dev/mapper/db_vg-db
Yup, lots of errors. Rebooted and good to go.
The moral of this story: Don’t just try fsck in whatever the mount communities say and hope that is the right one. That might not be the one you are having issues with. Look at the journal logs–will save you some time. This will allow you to fix the problem quickly without guess work and trying to mess with fstab or remounting, which is what I tried first.
It is a fairly long log, but only takes a minute or so to page through–keep going until you see a fsck error towards the end–this is likely your real issue. Errors before that are often innocuous.
