Print-friendly Version

Managing I/O Topology Changes Due to CPU Chip Failures in Multi-Node sun4v Platforms

Sree Vemuri, February 2009

The Sun SPARC Enterprise T5240 Server is a dual-socket server with up to two UltraSPARC T2 Plus processors. The Sun SPARC Enterprise T5440 Server is a quad-socket server with up to four UltraSPARC T2 Plus processors.

Both servers have an integrated Peripheral Component Interconnect (PCI) topology that causes the I/O device paths to change when a Chip Multiprocessing (CMP) node fails. The system can be booted to OpenBoot PROM (OBP) with the remaining CMP nodes, but the operating system might fail to boot, because the PCI Express (PCIe) device paths have changed. When the same device is found at a different I/O path after the power-cycle it is treated as a new device. A change to the boot path requires that the OS be reinstalled.

The device_remap Perl script, which is included in OpenSolaris build 106, addresses this longstanding issue of having to reinstall the OS after I/O topology changes.

In a CPU node failover condition, the script combines the current Solaris device paths and the ioalias information from the firmware to remap the device paths in the /etc/path_to_inst file and the symlinks under /dev so they match the hardware.

Usage

/usr/platform/sun4v/sbin/device_remap
   [-v | -R <dir>]

The following options are supported:

• -v prints the <path_to_inst> and /dev/symlink changes.
• -R <dir> operates on /etc/path_to_inst and /dev on the root image at <dir>.

Procedure

After adding CPU nodes or removing CPU nodes, boot the system to the OBP prompt, and use the following steps:

1. Boot either the failsafe miniroot using boot -F failsafe or an install miniroot using boot net -s or a similar command.

2. Mount the root disk as /mnt.

3. Change to the mounted root disk directory:

cd /mnt

4. Run the device_remap script:

 /mnt/usr/platform/sun4v/sbin/device_remap

5. Boot the system from disk.

All the error messages are self-explanatory except for the error message missing ioaliases node, which means the firmware on the system doesn't support device remapping.

Examples

Example 1: When CMP node 1 fails, the device_remap script replaces the I/O path /pci@500 with /pci@700/pci@0/pci@1:

# /a/usr/platform/sun4v/sbin/device_remap -v
replacing /pci@500 with /pci@700/pci@0/pci@1 in /etc/path_to_inst
path_to_inst changes:
45,52c45,52
< "/pci@500/pci@0" 5 "pxb_plx"
< "/pci@500/pci@0/pci@9" 6 "pxb_plx"
< "/pci@500/pci@0/pci@c" 7 "pxb_plx"
< "/pci@500/pci@0/pci@c/network@0" 0 "nxge"
< "/pci@500/pci@0/pci@c/network@0,1" 1 "nxge"
< "/pci@500/pci@0/pci@c/network@0,2" 2 "nxge"
< "/pci@500/pci@0/pci@c/network@0,3" 3 "nxge"
< "/pci@500/pci@0/pci@d" 8 "pxb_plx"
---
> "/pci@700/pci@0/pci@1/pci@0" 5 "pxb_plx"
> "/pci@700/pci@0/pci@1/pci@0/pci@9" 6 "pxb_plx"
> "/pci@700/pci@0/pci@1/pci@0/pci@c" 7 "pxb_plx"
> "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0" 0 "nxge"
> "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0,1" 1 "nxge"
> "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0,2" 2 "nxge"
> "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0,3" 3 "nxge"
> "/pci@700/pci@0/pci@1/pci@0/pci@d" 8 "pxb_plx"

updating /dev symlinks

#

Example 2: When CMP node 1 is added, the device_remap script replaces the I/O path /pci@700/pci@0/pci@1 with /pci@500:

# /a/usr/platform/sun4v/sbin/device_remap -v
replacing /pci@700/pci@0/pci@1 with /pci@500 in /etc/path_to_inst
path_to_inst changes:
45,52c45,52
< "/pci@700/pci@0/pci@1/pci@0" 5 "pxb_plx"
< "/pci@700/pci@0/pci@1/pci@0/pci@9" 6 "pxb_plx"
< "/pci@700/pci@0/pci@1/pci@0/pci@c" 7 "pxb_plx"
< "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0" 0 "nxge"
< "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0,1" 1 "nxge"
< "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0,2" 2 "nxge"
< "/pci@700/pci@0/pci@1/pci@0/pci@c/network@0,3" 3 "nxge"
< "/pci@700/pci@0/pci@1/pci@0/pci@d" 8 "pxb_plx"
---
> "/pci@500/pci@0" 5 "pxb_plx"
> "/pci@500/pci@0/pci@9" 6 "pxb_plx"
> "/pci@500/pci@0/pci@c" 7 "pxb_plx"
> "/pci@500/pci@0/pci@c/network@0" 0 "nxge"
> "/pci@500/pci@0/pci@c/network@0,1" 1 "nxge"
> "/pci@500/pci@0/pci@c/network@0,2" 2 "nxge"
> "/pci@500/pci@0/pci@c/network@0,3" 3 "nxge"
> "/pci@500/pci@0/pci@d" 8 "pxb_plx"

updating /dev symlinks

#

For More Information

Here are some additional resources:

• Product documentation at http://docs.sun.com and the Documentation Center, for example:
  • manpages section 1M: System Administration Commands Beta (pdf) (Notes: See page 341 of 3038. PDF download is 9.7 MB. URL subject to change since Beta doc.)
  • Sun SPARC Enterprise T5240 Server documentation
  • Sun SPARC Enterprise T5440 Server documentation
• Sun SPARC Enterprise T5240 Server web page
• Sun SPARC Enterprise T5440 Server web page
• Sun downloads page for SPARC servers and workstations
• Hardware 60-day Free Trials web page
• Sun training courses at http://www.sun.com/training/, for example:
  • Sun SPARC Enterprise T5140/T5240 Server Installation, Administration, and Troubleshooting (WET-6142)
  • Sun SPARC Enterprise T5440 Server Installation, Administration, and Troubleshooting (WET-6229)
• Discussions, such as Sun forums and the BigAdmin Discussions collection
• Sun wikis, for example:
  • Main Sun BluePrints wiki page
  • CMT BluePrints wiki page, which contains Sun BluePrints and links to resources, such as white papers, for the Sun SPARC Enterprise T5440 Server
  • BigAdmin wiki, especially the Servers wiki page
• Related resources on BigAdmin:
  • System Administrator Resources for Sun's CoolThreads Servers
  • Patching Center, especially Patching Center: Firmware and Patching Center: Systems Management
  • Resource Collection: Hardware
• Support:
  • Sun resources:
    • Register your Sun gear
    • Services
    • SunSolve Online
  • Community system administration experts
• Events of interest to users of Sun products:
  • Worldwide Developer Events and Sun Tech Days
  • Current Events

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.