ROOTVG

Hello, I'm running AIX 4.3.3 ML8 and we had a scsi bus problem yesterday which caused 2 pdisks to go "stale". hdisk2 is part of rootvg and hdisk3 is part of homevg. Of course IBM will not help me since I'm still at 4.3.3 (we plan on upgrading to 5.2 after the first of the year). IBM CE ran diags on the scsi bus and drives and all are showing as OK.

I'm assuming that it's fairly easy to get these 2 drives back and mirrorred I just don't know the correct commands.

Of course this is a production box.

# lsvg -p homevg
homevg:
PV_NAME           PV STATE    TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk3            missing     542         77          00..25..27..25..00
hdisk1            active      542         77          00..13..07..25..32
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE    TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active      542         9           00..00..00..00..09
hdisk2            missing     542         9           00..00..00..00..09
# lspv hdisk2
PHYSICAL VOLUME:    hdisk2                   VOLUME GROUP:     rootvg
PV IDENTIFIER:      0008f5af182b889f VG IDENTIFIER     0008f5afec749b98
PV STATE:           missing
STALE PARTITIONS:   239                      ALLOCATABLE:      yes
PP SIZE:            16 megabyte(s)           LOGICAL VOLUMES:  8
TOTAL PPs:          542 (8672 megabytes)     VG DESCRIPTORS:   1
FREE PPs:           9 (144 megabytes)
USED PPs:           533 (8528 megabytes)
FREE DISTRIBUTION:  00..00..00..00..09
USED DISTRIBUTION:  109..108..108..108..100
lspv hdisk3
PHYSICAL VOLUME:    hdisk3                   VOLUME GROUP:     homevg
PV IDENTIFIER:      0008f5af0071eb82 VG IDENTIFIER     0008f5af372a18c1
PV STATE:           missing
STALE PARTITIONS:   350                      ALLOCATABLE:      yes
PP SIZE:            16 megabyte(s)           LOGICAL VOLUMES:  11
TOTAL PPs:          542 (8672 megabytes)     VG DESCRIPTORS:   1
FREE PPs:           77 (1232 megabytes)
USED PPs:           465 (7440 megabytes)
FREE DISTRIBUTION:  00..25..27..25..00
USED DISTRIBUTION:  109..83..81..83..109

I have to think a bit on this. 4.3.3 is a while back, and I need to focus on what *MISSING* actually means again.

In SMS can you see all 4 disks? verify SCSI termination, and if they are two seperate vg, try focusing on rootvg first, with each disk on a seperate adapter (SCSI chain) so that you can be very certain that there is no SCSI address error (as my memory comes back to me, double SCSI address was the most frequent "cause" of missing, if the disk controller wasnt burned itself).

Not using SMS with this server. Using smit. But in talking with AIX s/w support yesterday he walked me thru removing the mirrors on the 2 affected drives. Only problem now is that I can't "find" the drives after removing them from the system (via rmdev -dl "hdiskX") so we are now looking at the scsi bus (both drives are on the same bus). The IBM CE is "hoping" that our weekly reboot will "clear up" the scsi bus since it now seems to be stuck or locked out/busy. If that doesn't clear up the hold on the bus, he'll have to replace the scsi bus/adapter, which will be a long and lengthy process...

pete

The command you will need is ldeletepv. (on 4.3.3. 5.X made it 'easier'.)

http://www.faqs.org/faqs/aix-faq/part1/section-41.html for some hints. Partial quote below.

There is another command I have to look for which will list all the pvid of all the disks in a vg. Maybe someone else finds that command first.

Ok. Basic steps.
1. Do a complete backup.

2. List what AIX knows:

3. Perform the removal of a 'dead' disk from a working volumn group.

Let's assume that my hdisk1 was missing.
Then I would use:

# ldeletepv -g 0040dd9a00004c000000010d5ae1df16 -p 0040dd8acc23029a
# rmdev -dl hdisk1

This is what IBM AIX S/W support had me do to remove hdisk3 from homevg:

lsvg -p homevg
lsvg -l homevg
lquerypv -h /dev/hdisk3 80 10        <-- this will give the same info as reported by the lspv command
-- no info was reported, so that shows that the ODM is "out of sync"
lquerypv -h /dev/hdisk3              <-- no output
lquerypv -h /dev/hdisk1 80 10        <-- ran this to verify output was correct for hdisk1 (which is OK)
lquerypv -h /dev/hdisk1              <-- gives complete dump. look down at address 0080 and you'll see same as above "80 10" output
lsvg -M homevg                       <-- shows all hdisks associated with homevg
lsvg -M homevg|grep hdisk3           <-- look to see what is referring hdisk3
unmirrorvg -c 1 homevg hdisk1        <-- this failed, so have to individually break mirror on each lv independently.
lsvg -l homevg                       <-- to show ratios again
rmlvcopy lvbdslib 1 hdisk3           <-- picked smallest and least volatile
lsvg -l homevg|grep lvbdslib        <-- look for ratio between LP's and PP's. If not mirrored PV should show 1 and LV STATE s/b open/syncd
rmlvcopy lvatimmback 1 hdisk3       <-- run this command for all lv's associated with homevg (based on the "lsvg -l homevg" output from above
rmlvcopy lvlogfiles 1 hdisk3
rmlvcopy lvcashlogs 1 hdisk3
rmlvcopy lvhauler 1 hdisk3
rmlvcopy lvlocal 1 hdisk3
rmlvcopy lvmim 1 hdisk3
rmlvcopy lvatimm 1 hdisk3
rmlvcopy loglv00 1 hdisk3
rmlvcopy lvinformix 1 hdisk3
lsvg -l homevg                      <-- should only see 1 to 1 ratio and PVs of 1 with PV STATE at "open/syncd"
lsvg -M homevg                      <-- should only show hdisk1 refs
lsvg -M homevg|grep -v hdisk1       <-- except is 1 line stating "hdisk3:1-542" which just shows it being assigned to homevg
reducevg homevg hdisk3              <-- to remove hdisk3 from homevg
lsvg -p homevg                      <-- should show only 1 hdisk
rmdev -dl hdisk3                    <-- totally remove hdisk3 from system
lsdev -Cc disk                      <-- verify no hdisk3
synclvodm -Pv homevg          <-- this is to "sync up" the odm refs for homevg
lspv                                <-- validate no ref to hdisk3
cfgmgr -v                           <-- this "should" refind the hdisk.
lsdev -Cc disk                      <-- NO HDISK3!!! This means a hardware error at this point.

At this point I need to call IBM hardware since the system can not find either disk in scsi0.

The procedure I repeated was one which was used when a disk died (especially for mirrors) and therefore could no longer be updated.

Are there any disks active on scsi0?

To reset a scsi controller you need to bring the device into 'defined' state.

To be sure you have a device completely 'quiet' you need to get it into defined state down to it's slot, or pciX definition.

Starting with a hdisk we need to find it's parent.

$ lsdev -Cl hdisk1 -F parent

To set this controller, and all the devices it is supporting in defined state use the command:

# rmdev -l scsi0 -R

If all devices are in a state that they can be 'powered off' you will see a list of devices going into defined state. Chances are you will see something like this.

Some devices are still available:


Conclusion: on this system I wont be able to attempt a reset of scsi0 without rebooting the system.