The Ghost in the Fabric: Debugging an Intermittent Oracle RAC Eviction

Earlier in my infrastructure career, our database team was battling an intermittent cluster issue that was driving everyone crazy.

We were running a multi-node Oracle RAC cluster on Red Hat for high-throughput OLTP. The backend was an all-flash Tier-1 array pushing 200,000 8k read IOPS at under 1ms latency across a Brocade Fibre Channel SAN. Performance-wise, it was a beast—until one specific node started dropping LUN connections at random intervals and getting ruthlessly evicted from the cluster.

If you’ve ever worked with Oracle RAC, you know it doesn’t mess around. The moment a node misses its voting disk heartbeat due to an I/O stall, Clusterware fences that machine immediately to prevent split-brain corruption.

The DBAs were stuck, management was breathing down our necks, and the Director of Ops was ready to pull the trigger on replacing the entire multi-socket HP server. He was convinced it was a bad motherboard.

Swapping a massive production server means hours of downtime, re-provisioning, and cluster reconfiguration. I knew we couldn’t just shotgun-troubleshoot enterprise hardware without real proof, so I dug in.

Chasing the Drops: Linux to Brocade

I started by looking at the cluster logs and tracing down to the HBA layer on the failing node. Using QLogic’s scli alongside Linux’s native systool, I pulled the physical port statistics:

systool -c fc_host -v -d host0

The error counters told the real story. This single node had significantly higher invalid_crc_count and loss_of_sync_count metrics than any other machine in the 7-node rack.

Next, I jumped onto the Brocade switch and ran porterrshow. Sure enough, the switch port mapped to that server was logging identical CRC errors, specifically accumulating crc_g_eof (CRC error with a Good End-of-Frame). That was the smoking gun: it proved the frame corruption was happening right there on the physical link between the server and the switch, not upstream in the fabric.

Don’t Trust the Green Checkmark

I had our datacenter tech run an optical test on the patch cable connecting the server to the switch. A few minutes later, the tech got back to me: “Cable tested great, it passed.”

Early in my career, I spent plenty of time terminating both single-mode and multimode fiber. One thing that sticks with you is that a basic field tester’s “PASS” doesn’t mean a cable is actually healthy under high-stress production loads.

I asked the tech to send over the raw Fluke test results so I could see the actual numbers.

Looking at the optical loss budget, the attenuation technically fell within passing specs—but just barely. It was sitting right on the razor’s edge. When you’re pushing 200k IOPS, you don’t have room for a marginal dB loss. Add in normal rack vibration and subtle thermal shifts, and that tiny variance was enough to intermittently corrupt light pulses, starve buffer-to-buffer credits, drop frames, and trigger an I/O timeout long enough for RAC to pull the plug on the node.

The Fix

I told the tech to trash the cable, terminate a brand new one, and send me the test results. I wanted to see a much wider passing variance on the attenuation readout.

We plugged the new cable in, cleared the port counters on the Brocade, and checked the optics with sfpshow.

The optical power levels stabilized, the CRC errors completely stopped, and the node stayed rock-solid in the cluster without a single eviction. Total fix: a $15 patch cable and a few minutes of termination work.

What to Take Away

• Avoid shotgun troubleshooting: If we had listened to the Director and replaced the entire server, we would have burned hours of maintenance downtime just to plug the exact same bad patch cable into a brand-new box.
• Look past automated test results: A static Fluke pass only verifies basic continuity within broad factory limits. In high-density environments, marginal link budgets kill performance.
• Layer 1 knowledge pays off: You can spend all day analyzing high-level software logs, but at the end of the day, enterprise infrastructure runs on light and copper. When the physical layer is sick, everything above it falls apart.