OCFS2 Cluster Node Crashes with an Inode Mismatch

This document (7008779) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10
Oracle Cluster File System 2 (OCFS2)

Situation

The third node in an OCFS2 file system cluster crashes when the file system is mounted or brought online. The following errors were observed in the /var/log/messages system log.

kernel: ocfs2: Mounting device (65,81) on (node 2, slot 2) with ordered data mode.
kernel: ocfs2_dlm: Nodes in domain ("6EF4AC640EDB490B8D357B19F3D2C46D"): 0 1 2
kernel: (23027,2):ocfs2_populate_inode:281 ERROR: ip_blkno 82 != i_blkno 98!
kernel: Kernel BUG at fs/ocfs2/inode.c:488
kernel: invalid opcode: 0000 [1] SMP
kernel: last sysfs file: /o2cb/interface_revision

Resolution

Run a file system check on the OCFS2 file system. Do the following to check the file system:

1. Back up all your file system data if you can.
2. Unmount the OCFS2 file system from all nodes. If the file system is a heartbeat cluster resource, stop the resource to unmount the file system. WARNING: You CANNOT run a fsck.ocfs2 on a mounted OCFS2 file system. It will damage it.
3. Run a file system check.

# fsck.ocfs2 -f /dev/ocfs2_device

4. If the fsck.ocfs2 command reports errors, repeat the file system check until there are no errors.
5. Mount the OCFS2 file system, and retest. If the file system is a heartbeat cluster resource, bring the resource online to mount the file system.

Additional Information

In this particular case, the //extent_alloc:0002 and //journal:0002 both have Inode: 98. They should have unique inode values. The same inode value indicates a corrupted file system.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.