AIX Alternate Disk Installation
Jeff Marsh
In this article, I will describe some tools within AIX (some new, some old) that can help you reduce the off-hours time spent by your administration staff during maintenance upgrades. I will also show you some uses for these same toolsets that can help you reduce recovery times due to rootvg corruption.
Alternate Disk Installation
What is it? According to the IBM AIX Installation Guide:
"Alternate disk installation, available in AIX Version 4.3, allows installing the system while it is up and running, allowing installation or upgrade down time to be decreased considerably."
Thus, with another set of bootable drives within a server, you can install maintenance (e.g., upgrade your system from AIX 4.3.3.04 to AIX 4.3.3.06) during the day without interruption or any effects to the running applications. However, you will still need a reboot to make it active.
The support model prior to Alternate Disk Installation required all work to be done off-hours during an application maintenance window that generally took two to four hours. Now you can reduce that off-hour time from two to four hours per server to just the time to reboot. I'll also show you how you can complete multiple upgrades in that same reboot window using Network Installation Manager (NIM).
Requirements
To enable Alternate Disk Installation, you need to install the following base-level filesets and upgrade to at least these corresponding fileset levels. These filesets do not require a reboot to install:
Base level filesets: Fileset levels:
bos.alt_disk_install.rte 26
bos.alt_disk_install.boot_images 27
You will also need another free, bootable drive within your server. In this case, you are configuring new servers with four internal drives for systems administration purposes: two drives for the primary rootvg mirrored, and two for alt_disk_install implementations. You could get by with just one additional drive, but we prefer to have two.
How It Works
Alternate Disk Installation works by cloning your primary rootvg running on hdisk0 and hdisk1, for example, to a second set of drives, hdisk2 and hdisk3. After the system completes those copies using basic find, backup, and restfile utilities, it will install the latest maintenance level you designate.
This process is shown in Figure 1. First, you clone hdisk0/1 to hdisk2/3, and then you apply maintenance to the newly cloned hdisk2/3 while the applications continue to run against hdisk0/1.
To complete this task from SMIT, issue the following fast path. You should expect to see the following panels:
smitty alt_clone
Clone the rootvg to an Alternate Disk:
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
* Target Disk(s) to install [hdisk2 hdisk3]
Phase to execute all
+
image.data file []
/
Exclude list []
/
Bundle to install [update_all]
+
-OR-
Fileset(s) to install []
Fix bundle to install []
-OR-
Fixes to install []
Directory or Device with images [/mnt]
(required if filesets, bundles or fixes used)
installp Flags
COMMIT software updates? yes
+
SAVE replaced files? no
+
AUTOMATICALLY install requisite software? yes
+
EXTEND file systems if space needed? yes
+
OVERWRITE same or newer versions? no
+
VERIFY install and check file sizes? no
+
Customization script []
/
Set bootlist to boot from this disk
on next reboot? yes
Reboot when complete? no
+
Verbose output? no
+
Debug output? no
+
[BOTTOM]
F1=Help F2=Refresh F3=Cancel
F4=List
F5=Reset F6=Command F7=Edit
F8=Image
F9=Shell F10=Exit Enter=Do
From the example above, note the following:
• You are cloning to hdisk2 and hdisk3.
• You are running an update_all operation of maintenance mounted in the /mnt mount point. (In this case, this is a CD-ROM with the AIX 4.3.3.06 maintenance filesets.)
• You are specifying that this operation should change our bootlist to hdisk2 and hdisk3 after completion.
• You are not asking the process to complete an immediate reboot upon completion of the upgrade because this is something you want to schedule in an appropriate maintenance window.
At the completion of the operation, you will notice from the bootlist -m normal -o command that the bootlist will be set to hdisk2 hdisk3, and issuing an lspv command will show the following:
root@aknimp1:/> lspv
hdisk0 000f261d90bf6ea0 rootvg
hdisk1 000f261dae86d104 rootvg
hdisk2 000f261db52d4d95 altinst_rootvg
hdisk3 000f261db52d4ca6 altinst_rootvg
hdisk4 000f018d07d4f412 None
hdisk5 000f261dbde71c66 None
hdisk6 000f261dbd8eea89 nimresvg
At this point, you have cloned and installed the latest AIX maintenance level during the day. You are now ready to activate that latest maintenance with a reboot operation at whatever time is appropriate for the outage to your application users. You can save significant off-hours time for maintenance upgrades; our off-hours time has been reduced to the time needed for a simple reboot.
Alternate Disk Installation -- After the Reboot
After the reboot, issue the oslevel command or complete the appropriate verifications to ensure your maintenance upgrade occurred as expected. If you issue the lspv command, you will notice the following:
root@aknimp1:/> lspv
hdisk0 000f261d90bf6ea0 old_rootvg
hdisk1 000f261dae86d104 old_rootvg
hdisk2 000f261db52d4d95 rootvg
hdisk3 000f261db52d4ca6 rootvg
hdisk4 000f018d07d4f412 None
hdisk5 000f261dbde71c66 None
hdisk6 000f261dbd8eea89 nimresvg
Both hdisk2 and hdisk3, from which you have booted, now show a volume group identifier of rootvg. Hdisks 0 and 1 now show a volume group of old_rootvg and are varied off.
Now, you have several options. My preference is to leave hdisk0 and hdisk1 alone with the old maintenance levels in case you need to fall back on them.
Let's assume that after the reboot your applications aren't working well with the latest maintenance. The previous support model suggests that you need to get the mksysb backup taken prior to your upgrade and begin a restore process. This could take two hours or more, with the hope that the tape image was good. The new support model with Alternate Disk Installation says to change your bootlist back to hdisk0 and hdisk1 and to reboot the server. At some future point, when you decide the maintenance is good and you don't need to fall back, you can clone the latest maintenance residing on hdisk2/3 back to hdisk0/1.
Cloning Back to hdisk0/1
To complete the cloning of hdisk2/3 back to hdisk0/1, you must issue the following commands:
• alt_disk_install -W hdisk0 hdisk1 -- Wakes up the old_rootvg
• alt_disk_install -S -- Puts the old_rootvg back to sleep
• alt_disk_install -X old_rootvg -- Removes the old_rootvg volume group name associated with hdisk0/1 from the ODM and assigns them a value of "none", which will allow the cloning to recur cleanly.
• smitty alt_clone -- Reclone back to hdisk0/1 using the previous example.
I will discuss using the above commands further in the next section; however, in order to reclone drives that have been previously used to boot from, you must follow the commands verbatim to remove the knowledge of the old_rootvg volume group name from the ODM.
Other Uses for alt_disk_install
Some other items that alt_disk_install may be helpful with are:
• Nightly backup of your system -- Using alt_disk_install, you can backup your system nightly (or at whatever frequency is appropriate) without having to manage mksysb tapes. If you suffer some type of rootvg corruption, either major or minor, you can restore using the data on the cloned drives.
• mksysb Images -- The alt_disk_install command can be used to install images (AIX 4.3 or later) onto AIX 4.1 and later versions.
• You can also use alt_disk_install for recovery of corrupted files in rootvg and to reduce the size of logical volumes in rootvg, as described in the following sections.
Recovery of Corrupted Files in rootvg
If you suffer major corruption (hdisk failure), and the server crashes, and if you have cloned that data to another bootable drive, you could interface with SMS, for example, to change your bootlist to your other cloned drives and quickly recover the server.
If you suffer minor corruption within the rootvg where a file or a few files are corrupted or inadvertently deleted, you can wake up the cloned copy of the rootvg and copy those deleted or corrupted files back to the primary rootvg while the server is up and running.
In this example, you are booted against hdisk0/1 and have recently cloned the system to hdisk2/3. To access the cloned copy of the rootvg while the server is up and running, complete the following:
1. alt_disk_install -W hdisk2 hdisk3 -- Wakes up the cloned copy:
root@aknimp1:/> alt_disk_install -W hdisk2 hdisk3
Waking up altinst_rootvg volume group ...
Replaying log for /dev/alt_hd4.
2. From a df -k command, you will notice that the wake up command has mounted the alternate rootvg logical volumes, which are prefaced with /alt_inst prefix:
root@aknimp1:/> df -k
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 49152 5608 89% 1226 5% /
/dev/hd2 753664 5056 100% 19966 11% /usr
/dev/hd9var 16384 14340 13% 222 6% /var
/dev/hd3 32768 30376 8% 98 2% /tmp
/dev/lvexport 131072 126772 4% 41 1% /export
/dev/lv01 4980736 94468 99% 4546 1% /export/lpp_source
/dev/lv02 917504 448868 52% 29468 13% /export/spot
/dev/lvmksysb 15204352 3381328 78% 31 1% /export/mksysb
/dev/lvadmin 131072 126868 4% 25 1% /admin
/dev/hd1 16384 15820 4% 20 1% /home
/dev/lvadsm 16384 56 100% 21 1% /var/adsm
/dev/alt_hd4 49152 5704 89% 1192 5% /alt_inst
/dev/alt_lvadmin 131072 126868 4% 25 1% /alt_inst/admin
/dev/alt_hd1 16384 15820 4% 20 1% /alt_inst/home
/dev/alt_hd3 32768 30376 8% 98 2% /alt_inst/tmp
/dev/alt_hd2 753664 5056 100% 19966 11% /alt_inst/usr
/dev/alt_hd9var 16384 14380 13% 219 6% /alt_inst/var
/dev/alt_lvadsm 16384 1848 89% 20 1% /alt_inst/var/adsm
3. Copy the corrupted files from the appropriate alt_inst logical volume/filesystem. In this case, I corrupted my /etc/hosts file, so I will issue the following command to restore it from my latest cloned backup:
cp /alt_inst/etc/hosts /etc/hosts
4. When you have restored the required files, put the altinst_rootvg back to sleep, which will unmount the /alt_inst logical volumes/filesystems by issuing:
alt_disk_install -S
Reducing Logical Volumes Size Within the rootvg
Remember the pain associated with the need to reduce the size of a logical volume within the rootvg? It took a tape restore of the system to complete. Now, you can complete that reduction within a simple cloning process. The steps to complete that process are as follows:
1. Issue a mkszfile command to create the /image.data file.
2. Edit the /image.data file and specify SHRINK=yes in the logical_volume_policy stanza:
image_data:
IMAGE_TYPE= bff
DATE_TIME= Tue Oct 3 10:29:55 CDT 2000
UNAME_INFO= AIX aknimp1 3 4 000F261D4C00
PRODUCT_TAPE= no
USERVG_LIST= nimresvg
OSLEVEL= 4.3.3.10
logical_volume_policy:
SHRINK= yes
EXACT_FIT= no
ils_data:
LANG= en_US
3. Clone the rootvg to hdisk2 and hdisk3, specifying your customized /image.data file by issuing one of the following commands:
sm itty alt_clone (remember to specify the location of your image.data file on the image.data file prompt)
or
al t_disk_install -i/image.data -B -C hdisk2 hdisk3 (from the command line)
4. After the completion of the cloning operation, wake up the altinst_rootvg by issuing:
alt_disk_install -W hdisk2 hdisk3
5. Review your df -k output and compare the primary logical volume sizing to their /alt_inst counterparts.
6. If you are satisfied with the sizing reduction, change your bootlist (bootlist -m normal hdisk2 hdisk3) and reboot.
Network Installation Managment (NIM)
I want to briefly discuss NIM and show how well it interfaces with alternate disk installation. It can easily help you to manage upgrades on a group of servers, thus saving you even more time.
What Is NIM?
Paraphrasing from the AIX Network Installation Management Guide and Reference, "NIM is a base component of AIX and permits and aids in the installation and maintenance of AIX, it's basic operating system, and additional software and fixes that may be supplied over the network. NIM provides for the customization of machines both during and after installation. As a result, NIM has eliminated the reliance of the systems administration staff on tapes and CD-ROMs for software installation and maintenance."
In this case, you are using NIM to centrally manage a group of standalone machines (NIM clients) from a centrally located network attached to a NIM master. From the NIM master, you can manage operating system installations, maintenance upgrades, mksysb images for backup and recovery, installation of new servers (cloning), and the re-installation of existing servers in case of a disaster.
There's a great deal of functionality provided by NIM. I recommend reviewing the usage guide to see what NIM features could benefit your environment. I also recommend a good Redbook from IBM, NIM: From A to Z in AIX 4.3 (SG24-5524-00), which was published in February 2000.
I won't cover the specifics of setting up the NIM master and the corresponding NIM client configurations; it is not an overly complicated process. However, it will require someone with NIM-specific knowledge to lay out the functional NIM environment. If you support SP complexes, you have already had a fair amount of exposure to NIM even though it is buried one layer below PSSP.
One key feature of NIM that will help manage a group of servers concurrently is the Machine Group definition. Within NIM, you can operate as easily on a single machine as you can a group of machines. For instance, we have defined several machine groups within our NIM master environment. These definitions allow us to operate on a group of like servers concurrently.
How Does It Integrate with Alternate Disk Installation?
NIM knows how to fully exploit Alternate Disk Installation. For example, look at the initial clone and update_all operation. Let's say you want to use NIM to extend the model (instead of upgrading the maintenance level on a single server) and you want to complete this operation on ten Lotus Notes servers that are similarly configured and are defined in a Notes machine group within NIM. From SMIT on the NIM master, issue the following fast path and you will see this panel:
smitty nim_alt_clone
Clone the rootvg to an Alternate Disk
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Target Machine / Group to Install [NOTES] +
* Target Disk(s) to install [hdisk2 hdisk3]
Phase to execute all +
IMAGE_DATA resource [] +/
EXCLUDE_FILES resource [] +/
(leave blank to include all files in backup)
BUNDLE to install [] +
-OR-
Fileset(s) to install []
FIX_BUNDLE to install [] +
-OR-
FIXES to install [update_all]
LPP_SOURCE [aix433_lppsource] +
(required if filesets, bundles or fixes used)
installp Flags
COMMIT software updates? yes +
SAVE replaced files? no +
AUTOMATICALLY install requisite software? yes +
EXTEND filesystems if space needed? yes +
OVERWRITE same or newer versions? no +
VERIFY install and check file sizes? no +
Customization SCRIPT resource [] +/
Set bootlist to boot from this disk
on next reboot yes +
Reboot when complete? no +
Verbose output? no +
Debug output? no +
Group controls (only valid for group targets):
Number of concurrent operations [] #
Time limit (hours) [] #
F1=Help F2=Refresh F3=Cancel F4=List
F5=Reset F6=Command F7=Edit F8=Image
F9=Shell F10=Exit Enter=Do
In this example, you would cause every server defined in the Notes Machine group to begin a process to clone itself from hdisk0/1 to hdisk2/3. At the completion of the cloning operation, NIM would then NFS-mount the aix433_lppsource resource (in this case, it's the AIX 4.3.3 lppsource filesystem, which includes the 4.3.3.06 maintenance) and apply it to the newly cloned hdisk2/3 on each of these servers. This also instructs NIM to change the bootlist on each of these servers as a part of the operation but does not cause an immediate reboot. I recommend, however, using NIM to schedule a reboot of all these servers during the maintenance window.
All of this work, including the cloning and upgrading of the maintenance level, can be completed during the day without affecting the running application (e.g., Notes). For the previous support model, this same upgrade would have taken about 2 hours per server plus reboot time to complete during an application maintenance window, generally in the middle of the night. If a single person worked to complete this process, this could have taken about 25 hours spread across multiple weekends to complete. With NIM and Alternate Disk Installation, this upgrade outage can be reduced to the time to reboot these 10 servers concurrently (or about 30 minutes, in our case). Note that your time may vary depending on speed of network, number of filesets being updated, time to reboot, and problems encountered.
Figure 2 shows the process using NIM/Machine Groups and Alternate Disk Installation. First, you instruct the NIM master to have each of the servers in the defined machine group clone hdisk0/1 to hdisk2/3 (depicted in red). Then, NIM will NFS-mount the appropriate LPPSOURCE filesystem containing the AIX 4.3.3.06 maintenance level and apply that maintenance to the newly cloned drives (operation in green). Again, this process happens concurrently on all servers in the defined NIM machine group without affecting the running applications.
Conclusion
My team is in the process of rolling out this methodology change. I think we can significantly reduce the amount of time spent in support of our current AIX standalone infrastructure. I also think Alternate Disk Installation and NIM, can help you better manage your infrastructure and provide some consistency to your installation, upgrade, maintenance, and build procedures. In conclusion, I hope the above discussion will help you significantly reduce the amount of off-hours time associated with maintenance or fileset upgrades within AIX.
Jeff Marsh is the Systems Advisor to the UNIX Server Team working at American Century Investments, a premier investment manager serving nearly two million individual and institutional investors. Jeff can be contacted at: [email protected].
