サーバーがおかしくなったので、シリアルで繋いでみたら
[6879670.724188] ata2.00: failed to read SCR 1 (Emask=0x40)
[6879670.729572] ata2.00: failed to read SCR 0 (Emask=0x40)
[6879670.734932] ata2.01: failed to read SCR 1 (Emask=0x40)
[6879670.740297] ata2.15: exception Emask 0x4 SAct 0x0 SErr 0x0 action 0x6 frozen
[6879670.747575] ata2.00: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
[6879670.755037] ata2.00: failed command: READ DMA EXT
[6879670.759975] ata2.00: cmd 25/00:08:20:bc:c9/00:00:e8:00:00/e0 tag 2 dma 4096 in
[6879670.759975] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[6879670.775019] ata2.00: status: { DRDY }
[6879670.778896] ata2.01: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
[6879670.786353] ata2.15: hard resetting link
[6879671.272764] ata2.15: SATA link up 1.5 Gbps (SStatus 113 SControl F310)
[6879671.280206] ata2.00: hard resetting link
[6879672.380925] ata2.01: hard resetting link
[6879673.424658] ata2.01: failed to resume link (SControl 0)
[6879673.435971] ata2.00: configured for UDMA/33
[6879673.441275] ata2: EH complete
なんてのが延々出ていた。
そして強引に電源ブチで再起動してみると、fsckが走り出し、異常終了。
/dev/sda2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
fsck exited with status code 4
done.
Failure: File system check of the root filesystem failed
The root filesystem on /dev/disk/by-uuid/a655bf7a-3a32-4f2a-b0f1-fa0086bc9c39 requires a manual fsck
マニュアルで fsck をやれというのでやっているがとてつもなく時間が掛かりそう。