So I woke up the other day and the server was making a LOT of noise. My first thought was that it was a fan going out [still think it is actually], so I tried 'fmadm faulty'. I was hoping to see something telling me which of the 8 fans were dying. Instead, I see that I was in the midst of a zpool failure.
Checking 'zpool status' I was able to verify that one of the disks had died. I'm running raid-z2, so it isn't like it was a hectic problem to solve... but I learned long ago it is better to fix it now than to wait until a couple more die.
Normally, you would have to go buy a replacement disk. Luckily for me, this Asus box never did recognize the last 3 drive bays. Counting across (there are 10 hot-swap SATAII drives -- want to make sure to pop the right one) I took a guess which one was failing and which ones were not currently in use. Luckily I was right.
Now, the "new" drive already had data on it because it used to be part of the root mirror before I upgraded them to larger drives. That being said, I was a little confused when the new drive wasn't being recognized.
I'm thinking I should hook up one of my woot-off lights to flash whenever fmadm faulty shows a failure...