Q: Would you recommend applying Sun recommended patches regularly to keep current or rather just applying the Security Alert bundle and any other patches that you require to fix specific problems (to "minimize introducing more changes to a system than necessary")?

A: We always recommend the latter, that is, just apply the "Security Alert" bundle and any other patches that are specific to the problem.

ibummer - November 26, 2007 Back to top

---

Q: Can you outline the Sun internal process of what happens before a patch is released to the public.

A: Please read A Bug's Lifecycle @ Sun on the Patches hub on BigAdmin. This pdf describes all the steps in detail.

Andrew Watkins - November 26, 2007 Back to top

---

Q: When does Sun recommend you patch your servers (not including security patches): quarterly, bi-yearly? I am asking because Sun seems to have changed how it handles this. It used to be when you called in a problem Sun always wanted to have the latest and recommended patches installed. Also, what does Sun recommend as the best way to patch your servers: the latest recommended and security patches, or using Sun Update Manager?

A: To be honest, there is no one answer to this question. First of all, patches also deliver new functionality, i.e. driver changes for new hardware, new core functionality, i.e. ZFS, and so on. This is all alongside the bug fixes that are also delivered.

So customers must evaluate their own business needs (uptime SLAs and so on) and decide what the best strategy is. There are two common strategies: patch when a defect occurs (reactive), or patch on a regular schedule to avoid known issues.

Both of these have inherent risks. The reactive patching strategy will usually be initiated under a stressful situation (i.e., a server has crashed), so time to root cause the event may be limited. Often the latest patch cluster will be applied in the hope that it rectifies the problem, and if it does not (often the case) then the situation is much worse, as the original problem has been obscured by adding a lot of extra change.

The second approach is where a regular maintenance window is used to apply patches. This is to prevent running into the known issues that the patch fixes. But of course applying any change to a system introduces risk in itself.

In my opinion the best approach is to limit the change to the smallest known set of needed changes; often this can be the latest security cluster.

Before applying any patches, the user needs to evaluate all patches being applied, inspect all SPECIAL INSTALL INSTRUCTIONS from the patch READMEs, and identify from that any possible issues, or special steps that need to be taken.

It is also reasonable to deduce that different users have different needs. A mission-critical system should never be patched unless the patches being applied have been thoroughly examined in a dedicated test system that exactly matches the target system in every spec (hardware and software). Also such a system should only be patched with patches that have been stable in the field for a certain period of time, i.e. up to one year in critical cases.

A desktop user, on the other hand, will be happy to take newly released patches in a lot of cases in order to evaluate new functionality. Their testing requirements are going to be less than those of a mission-critical sysadmin. But they still should read all README notes at a minimum.

So to my mind, a good patching strategy needs a properly planned change management approach, to determine what the proper approach is.

For patching important servers I would recommend using patch clusters; for desktop users I would recommend Update Manager.

Sorry for not providing a definitive answer, but I believe that until you examine your own business needs and SLAs, a patching strategy cannot be determined.

Earl - November 27, 2007 Back to top

---

Q: I've often applied masses of patches to systems at once using the command patchadd -M <directory name> where directory name is the directory where I have all of my patches stored (old, applied, and new). This has always worked very well for me. The patchadd command seems to correctly determine by itself in which order the patches should be installed, even when some patches can only be applied after a patch requiring a reboot. I have, however, heard that this is not so good. Can you please tell me if using patchadd -M <directory with all patches> in this manner is actually good, and how it works when it is used this way.

A: There can be problems with this approach, one being that some patches, such as certain Kernel patches, cannot be applied via -M.

It is best to run patchadd -a -M <directory name> where -a forces a dry run (no changes are made to system). From this you can determine if there are any unexpected issues, such as an unexplained patch dependency missing and so on.

Also if your system has zones, you need to be extra careful. If the last patch that is specified in -a -M fails in the global zone, then none of the zones will get patched. So examine the last patch carefully, and make sure it has no patch-level scripts (such as prepatch, postpatch).

Only then would I drop the -a, and actually apply the cluster.

Ian Ballantyne - November 27, 2007 Back to top

---

Q: If I have multiple patches that require a reconfiguration boot, can I install all the patches and then perform one reconfiguration boot? (Please pardon my ignorance as I am new to Solaris.)

A: When installing patches in this way, it is important for the system to be in quiet mode and the installation to be performed in single-user mode (Run Level S). This is how the recommended patch clusters advise installation to take place.

But the preferable approach is to patch your system using Solaris Live Upgrade. For more information, please see the BigAdmin article How To Use Solaris Live Upgrade to Install Patches.

Vikram Luke - November 27, 2007 Back to top

---

Q: I have sol10u3, but the patches are for sol10. I have applied all patches with Update Manager (patchadd where manual investigation needed; 120012-14 etc => Generic_127112-02). Do I now de facto have a sol10u4, only missing pkgs introduced in sol10u4, and a somewhat misleading /etc/release, or what? If I also add these packages, what is different from a [live]upgrade?

A: If you are just looking to maintain fixes in line with an update release, then yes, this can be done by using patchadd in this way. You are now simply missing the new packages that were delivered with u4. So the new features that required the new packages will not work, but all features delivered in update 4 which did not require a new package will be present along with all the fixes for all the packages that you have installed on your system.

Were you to install the new packages also, then it would be installed with the same software as if you had performed an upgrade.

Note that the way in which updates are generated, by a process called fresh-biting, ensures this is true. The process takes the old packages and the current patches. The binaries from each patch are applied to the old packages (swapping new binaries for old binaries).

November 27, 2007 Back to top

---

Q: Since there are all kinds of things that can happen to cause zones to get out of step with the global zone as far as patches go, what's the best way to keep a global zone and non-global zones up-to-date with regards to patches?

A: The best way is to be proactive in installing patches. Firstly, always run patchadd with -a (this is an undocumented interface at present). Basically, it forces a dry run, so no changes are made to the system. You can examine the output of this (it is very verbose to the point of being hard to parse). It can flag certain types of issues like dependencies not being met in zones, or zones not bootable for one reason or another.

If using patch clusters, run patchadd -a -M . and examine the output.

Pay close attention to any error messages in -a output, and when actually applying patches, make sure that you examine the output from patchadd carefully in both global and non-global zones.

Stan Dutler - November 27, 2007 Back to top

---

Q: Can old packages be removed from the /var/sadm/pkg/ directory tree to free up space, and if so how?

A: When a Solaris patch is installed, it saves the old information in case it has to be backed out. This old information can take up a considerable amount of space as it accumulates in the /var/sadm/pkg//save directories.

As a patch is applied, it creates a compressed archive of all the packages which are being updated by the patches. The compressed files reside in the /var/sadm/pkg//save/ directories. Some patches only update a single package, while others update several packages.

Information on which packages are updated by a patch can be gathered from the showrev -p or patchadd -p commands.

For example:

To find which packages are updated by Patch 103093, run the following command:

  # showrev -p | grep  "Patch: 103093"
  Patch: 103093-03  Obsoletes:  ...  Packages: SUNWcsr, SUNWcar

The "Packages:" part of the output lists the specific packages. In this case, you would find the backout information files for Patch 103093-03 in the following directories:

  /var/sadm/pkg/SUNWcsr/save/103093-03
  /var/sadm/pkg/SUNWcar/save/103093-03

The actual file names can be either undo.Z or obsolete.Z. The most current version of the saved information will be called undo.Z. The backout file name from a previous revision of the patch changes from undo.Z to obsolete.Z. The renaming takes place as the newer revison of the patch is installed.

So, if Patch 103093-01 is the only revision installed, you would find the following files:

  /var/sadm/pkg/SUNWcsr/save/103093-01/undo.Z
  /var/sadm/pkg/SUNWcar/save/103093-01/undo.Z

Then, if Patch 103093-03 were applied, you would find the following files:

  /var/sadm/pkg/SUNWcsr/save/103093-01/obsolete.Z
  /var/sadm/pkg/SUNWcar/save/103093-01/obsolete.Z
  /var/sadm/pkg/SUNWcsr/save/103093-03/undo.Z
  /var/sadm/pkg/SUNWcar/save/103093-03/undo.Z

There are two ways of deleting the backout information.

1. Remove all undo.Z and obsolete.Z files from the /var/sadm/pkg directory structure. This frees most of the space but also prevents any patch from being backed out. This can be catastrophic if a patch is applied that causes problems and needs to be backed out. If you have removed the backout files following the patch install, it's impossible to remove the patch.

2. Remove only the obsolete.Z files from the /var/sadm/pkg directory structure. This still frees a lot of space but it also preserves the ability to remove any of the most recent versions of any patch. This can be particularly useful if a problem is found with a freshly applied patch. If removal of backout information is unavoidable, this would be the better method of the two.

Here is an example of a command which deletes all obsolete.Z files:

# find /var/sadm/pkg -name obsolete.Z -exec rm {} \;
	

NOTE 1: In the Solaris 10 OS, there are additional entries located in the /var/sadm/pkg//save directory structure: specifically, the pspool directory and everything under it. These files and directories are created and utilized by the updated Solaris 10 patching utilities. For example, there will be more undo.Z files found under the pspool directory. Such as:

/var/sadm/pkg/SUNWcsr/save/pspool/SUNWcsr/save/118833-36/undo.Z

Removing those undo.Z files will free up some space but it is highly discouraged.

(Editor's note: Information based on Infodoc 14295.)

November 28, 2007 Back to top

---

Q: I recently patched a system running Solaris 10 6/06 OS with eight containers. The system was originally installed with typically 'all' in Solaris so I tried to minimize downtime by cleaning out packages (like GNOME) we don't use, in advance. Despite this it took too long, more than 6 hours, to install the recommended patch cluster (Oct/09/07): for example, kernel patch 118833-36 took 67 minutes and 120011-14 took 90 minutes. I ran patching in single-user mode. Do you have suggestions on reducing downtime for patching other systems on the same level or future patch rounds?

A: Kernel Patches are normally quite large in size, and hence they require a good amount of time for installation. 118833-36 and 120011-14 are both Kernel Update patches. The second depends on the first in order to get installed. You cannot get rid of the time to install a Kernel patch. However, with the advent of Rejuvenated patches, the new changes would be delivered in chunks of patches rather than combining them together in one single Kernel patch. To know more about Rejuvenated patches, go to the BigAdmin Patch hub.

As for your question on reducing downtime for patching: One such method is to use Solaris Live Upgrade. The time taken when Live Upgrade is used to patch a boot environment (with a Kernel patch) would be the same, however, there is no downtime involved with the current machine, and it is up and running during this time.

Mat - November 28, 2007 Back to top

---

Q: On AIX you can install the patches before your downtime window and reboot to commit the patch. Is there something like that in Solaris? This reduces your downtime window.

A: Yes, you can do this with the Solaris OS using Live Upgrade. Go to the new BigAdmin Patch hub and see the article How to Use Solaris Live Upgrade to Install Patches. This article provides the basic steps for using Solaris Live Upgrade to avoid patching a running system and to avoid using single-user mode.

goodsun - December 04, 2007 Back to top

---