John, when I arrived at work last monday, it was my intension to ask somebody to replace the disk. I never needed to ask it, because the machine was active again. 

This morning, I did a

1) diagnostics on the disks (no errors)
2) certify on hdisk0, output:

Certify media task
device: hdisk0 in location U0.1-P1-I2/Z1-A0

The certify operation is in progress

please standby

... % completed
Disk drive capacity.....................18200 MB
Data errors recovered.......................1
Data errors not recovered..................0
Equipment check errors recovered.......0
equipment check errors not recovered..0

3) fsck on the filesystems /dev/hd1, /dev/hd2, /dev/hd3, /dev/hd4 and /dev/hd9var (no errors).

I guess this means everything is ok right now.

The only thing I'll have to look into is the quorum checking. And yes, I will keep an eye on the errpt....

Oh sorry but you said:

This has reminded me of another thread where we were talking about running diagnostics to certify (and maybe format) a suspect disk.

As Michael noted, a system can die without being able to write an error to the log on the disk - such dying throws can be extracted from a dump etc..  However. it is entirely possible that a disk can fail, perhaps only in one tiny part of it, and even kill the system without leaving an error log entry anywhere. 

You can test this sort of thing with for example adding a disk to the rootvg to put some paging space on, then just pull out that part of the paging space to simulate failure, and as soon as the system decides to try and use that area, bang.  Like having a piece of your brain removed I suppose ;-)

Mirroring is always the answer.

You've done all this now apparently, but, when mirroring from a potentially suspect disk be sure to check that suspect disk thoroughly first with the diags and also run fsck checks - requiring maintenance mode etc.   While mirroring a disk, look especially for errors at that point of accessing all of the data. 

In future at the first sign of any disk errors with that suspect disk, I would replace it.  When adding any new disk, it's a good idea (although boringly slow) to do the diag format and certify first.

As an aside, just imagine how troublesome such things can be when you have VIO and maybe dozens of different operating system instances sharing a disk - eggs, basket,... 

Going back to the beginning of this thread, and the original problem...

It appears that the original cause was likely to have been a disk bad block/sector, probably under the hd8 JFSLOG LV, in an unmirrored rootvg.

Sadly the effect of a disk error can be fatal to any system, that's the main reason why it is recommended (here anyway) that you mirror rootvg at least
- and turn quorum checking off on a two disk mirroring set.

You said that you were replacing the duff disk and adding another to mirror it, so there's no reason why that should occur again.  Obviously you will monitor your error log and act on disk errors when and if they are logged.  When you appear to have a failing disk, simply un-mirror it out of the volume group, replace and re-mirror - all of which can be done on the fly these days usually.

Thanks for the updates. Amazing what a power down can resolve!

New development on this case: 

Last weekend, there were some power-maintenance actions which needed all the machines to power down for the whole weekend. Last friday, I did a shutdown -Fh on all my machines (all but this machine, which ofcourse was already down due to the problems) and then I went home.

This morning, I did a ping on all the machine to check if they were active again and automatically, I tried to ping this machine also.

I was very suprised when the ping returned a normal reactie..... It seems that the powerdown was the key to the recovery of the system.

Naturally, the first thing I am trying to do is to mirror the rootvg, but that also causes some problems:

<hostname>:/root # extendvg rootvg hdisk1
0516-1398 extendvg: The physical volume hdisk1, appears to belong to
another volume group. Use the force option to add this physical volume
to a volume group.
0516-792 extendvg: Unable to extend volume group.

when I look at the disks, it seems hdisk1 is not assigned to any vg:

<hostname>:/root # lspv
hdisk0          005b835aa01e3d28                    rootvg          active
hdisk2          005b835af7e00482                    vgSQLTST_MOS_20 active
hdisk3          005b835a1195b4dc                    vg_bak          active
hdisk4          005b835aa01e3d68                    vg_esf          active
hdisk6          005b835af7e190cf                    vgSQLTST_MOS_20 active
hdisk1          005b835a1affe1d3                    None
hdisk5          00c8140e11571b1d                    None
hdisk7          none                                None
hdisk8          none                                None
hdisk9          none                                None
hdisk10         none                                None
dlmfdrv7        005b835a2d2517bf                    vg_p44          active
dlmfdrv8        005b835a2d2518e6                    vg_p44          active
dlmfdrv9        005b835abef765e6                    vg_esf          active
dlmfdrv10       005b835ac3bc2959                    vg_p44          active
dlmfdrv         none                                None

I am not sure if I should use the extendvg -f option, because I am not sure what's on hdisk1. Is there another way to check what is on the disk?

Sounds then like it may be the hd8 LV which is over an area of bad disk.
A similarly terminal experience may be had when some disk goes in your paging space area, if you haven't mirrored.

From the maintenance shell, try removing hd8 (rmlv)
and then recreate it in a different place on the disk
- so before that check where it was, doubtless "midway",
although it should be "center" but there probably isn't any space there anyway,
so try "edge" for the worst performance (move it later maybe)
- the key thing to note is that you have to say type is "jfslog"
not the default jfs, and there is of course no help or tab list on that smit field.
- then on the new hd8 LV do the logform before the fsck again.

Incidentally, the log doesn't have to be called hd8, you could change
the log associated with each rootvg filesystem to some other new name
and leave hd8 alone and corrupted (like Britney ?-)

# logform /dev/hd8