Basically, the main thing I try to focus on is separating the application files (data) and applications (binarys) from rootvg as much as possible into a separate volume group.
This has the added advantage of making your application nearly HACMP ready.
Once you have this ready the next "best" step is to prepare an extra (empty) disk for a new version of rootvg - the method is called a "alt disk installation" and we choose the clone variation.

Basically, what this entails:

• make a mksysb backup of your current system to tape, or a nim server.
  install the alt_disk_install filesets for your current filesystem from the NEW AIX installation media (e.g. alt_disk_install for AIX 5.2)
  
• use the smit alt_disk_install dialogs to make a clone of the current AIX rootvg, and then run the migration install on the clone. This has two basic advantages: downtime is minimal - the migration is done while the old system is up and running; and you can return to the old system if it is not functioning properly with the new system.
  
• Thus - the next step is reboot - using SMS menu to choose new disk as boot device - and test the new situation. Testing should probably be considered as "downtime".
  
• Last: if you are satisfied (or QA is satisfied) then you can delete the old_rootvg.
  

My suggestions:

• Update the application first - assuming it is supported from AIX 5.2 thru AIX 6.1 and be sure the application is functioning properly with application updates applied (and later confirmed).
  
• Practice a few times on a AIX 5.2 system with AIX 5.2 to AIX 6.1 alt install - and compare that to just importing the application volume group on a fresh install of AIX 6.1`(or AIX 5.3, etc.).