Print-friendly Version

Automating Patch Installation for the Solaris OS

Mike Myers, July, 2004

Like other computing sites, we've seen security become more and more important in our environment. One aspect of security that we had done a particularly poor job on was regular (proactive) system patching. Mostly, this was because the system admins, after working a long hard day, had little desire to work late at night or on weekends to execute system patching. Also, there was nothing to "trigger" a system to get patched. We only patched when we did major work on systems or upgraded them, or when a system crashed and support identified a patch to fix the issue -- generally, that was all. Sadly, we had systems that went for over three years without patching, and we had dozens of different patch sets on our servers. Consistent it was not.

Because we carry a Platinum contract from Sun and run Sun Explorer Software on all our systems (about 115 of them), we receive a monthly status report. This showed us just how badly we were doing -- lots of red. (As an aside, I'd highly recommended these reports to anyone who's not getting them -- talk to your account rep.) We would try to fix the worst of these, but without an actual policy for this, the fixes tended to get preempted by "real" work.

This obviously wasn't acceptable.

To solve this, we first created a "security" policy saying that systems must be patched every 90 days (security is in quotes because this also has a major effect on availability -- far fewer system crashes). This gave us a requirement to meet and allowed us to build a process to do just that.

The process we developed included the following points:

• Go out to Sun every 90 days and pull the "Recommended and Security" patch cluster for each OS revision that we're running. These are stored on a central NFS server. We also pull a few extra patches, for example, CDE, Java technology, and VERITAS patches.
• Write a report (run weekly) that shows how many days it has been since each system was patched. This report is delivered to the coordinator who schedules the patching.
• Create a new "rc" script that applies the patch set from the central NFS server. This was one of the critical bits, so this script was designed to be easy enough to use that virtually anyone could run it -- the only input required is a "should we patch (Y/N)?". If it runs into errors, it has instructions to page an admin.
• Write a script to be run via sudo (or RBAC) to insert the new rc script into the startup scripts at /etc/rc2.d/S73autopatching -- just after S72inetsvc runs, so most network services are available but no applications should have started yet. Patching in "true" single-user mode would have been much more difficult since we would have had to set up the network routing and such. This seemed like a good compromise. This script removes itself after running.

Fortunately for our site, we have a 7x24 operations staff. These folks are not admins (even junior level) but are very good at following rigidly designed procedures. The boot time script has just such a well-documented, rigid flow.

Once all that was in place and tested in our lab, we unleashed it on our little world.

Things were a bit rocky to start -- system owners were not used to taking outages so often and often complained long and loud about how we were driving down availability. We stuck to our guns, mostly playing the security card, and eventually people got used to this new procedure. Some folks even gave us a schedule of outages for the next 12 months.

Some of the admins were a little disgruntled, too. They believed that patching was a critical aspect of their jobs and were uncomfortable with others taking on this role. This didn't last long once it became clear that the process was reliable and gave them a few weekends back.

Things have improved in our environment dramatically since then. Our monthly reports from Sun are now mostly green. We can now let the regular patching cycle address certain less critical security announcements. We still patch the more critical announcements outside of this schedule; fortunately, many of those don't require a reboot. Unscheduled system outages have dropped dramatically. I don't think we've had a single service call to Sun that ended with, "That problem is addressed by a patch," since our first complete pass through the systems.

We've had to use a few tricks to make things run smoothly. The rc script has to be smart enough to know if it's patching the central NFS server itself, in which case it seeks out the files locally instead of trying NFS. Obviously we have to make sure to schedule the NFS server patching at a time when no one else is patching.

We use a heavily modified version of the install_cluster script that Sun ships with the patch clusters. It scans the patch_order file and eliminates patches that are already on the system. This decreases the time to apply a patch significantly.

Since all patches must be tested in the lab before we release them to the system owners, we built a facility into the rc script to pull the patches from a "staging" area (and turn on a bunch of debugging) when appropriate responses are provided (for example, answering "debug" to the "shall we patch" question).

To keep our processes clean and consistent, the same script can be run from the command line (as root) during system build to patch the new system to the same cluster that is being deployed in our environment.

The rc script uses /etc/nologin to block logins while the system is being patched. It also uses a local alerting to a console in our operations room to let the operators know that the process is stopped waiting for some type of response (either the initial "shall we patch" questions or an error later on). The full transcript of the patching session is captured and emailed to the list of system admins for post-patching review (either immediately if there's a problem or the next business day otherwise).

One trick: If you're running from an rc script, you cannot just say read foobar in your script -- you must explicitly specify /dev/console as your tty:

	read foobar < $mytty

The structure on the central patch server is that all patches are in a share called /patches. Under these are directories for each OS release that is supported, which are called Sun<release>_DATE where <release> is whatever is returned by uname -r, and DATE is a date stamp of the form DDMMYYYY. These directories in turn hold a _Recommended and _Addon directory that hold the Recommended and security bundle and a local bundle of your own choosing. The install_cluster script in each directory is what gets executed to drive the actual patching.

There are also directories called VxVM_DATE/vx_release/OS_RELEASE and VxFS_DATE/vx_release/OS_RELEASE, which have patches for VERITAS Volume Manager and File System, respectively (these also are driven by install_cluster scripts so they can house multiple patches).

Here is a link to the autopatching rc script, which will require some customization for each environment.

The information and links on this page have been provided by a BigAdmin user. The submitter is solely responsible for such information and links. Sun is not responsible for the availability of external sites or resources, and does not endorse and is not responsible or liable for any content, advertising, products, or other materials on or available from such sites or resources. Sun will not be responsible or liable, directly or indirectly, for any actual or alleged damage or loss caused by or in connection with use of or reliance on the information posted here, or goods or services available on or through any external site or resource.