Print-friendly Version

Automatically Configuring ce Cards With ce_settings.ksh

Tom Kranz, April 2005

Download the ce_settings.ksh script.

We've all been there. The server's been jumpstarted, the applications installed, and the machine's been handed over to the users. Suddenly, the calls start coming in. Network access is slow. File transfers take too long. A brief check of the obvious things like cables and switch settings leads on to fun with ndd, revealing that auto-negotiation has resulted in a disagreement between the Sun box and your switch. Never mind -- a few additions to /etc/system and a reboot later, and everyone's happy.

Annoying, but such is life. We've come to accept that auto-negotiation between servers running the Solaris OS and Cisco switches is one of those things that would be nice to have, but can be tricky to achieve.

As always, though, things are changing. Sun is moving away from editing /etc/system, and instead is implementing driver-specific config files -- possibly less straightforward to manage, given there's no longer just one central file to edit, but also possibly more flexible, allowing you to individually configure card instances. Sun's ce (Cassini Ethernet) cards are currently the most mature implementation of this new model. They cannot be configured from /etc/system, and rely on a ce.conf file somewhere in the kernel module path.

While the ce cards are reliable and fast, they cause some unique issues. This is because there are two types of card, using the same core chipset and driver. In the blue corner, we have X2222A -- the 100 Mbit/sec dual UTP/SCSI PCI Combo Card. In the red corner, we have X1150A -- the Sun GigaSwift Ethernet PCI card.

Our problem here is that we have two different card speeds. We know we have to force the 100 Mbit card to 100 Mbit/sec full duplex. The trick is to not force the Gigabit Ethernet card -- it needs to auto-negotiate, so it can implement useful GigE features like flow-control. Normally, this means we would need to manually edit the ce.conf file, creating scope for all sorts of fun with misconfigured card instances or typos causing the module load to fail. Consider the complications in a well-stocked domain of Sun Fire 15K servers, where you could have a dozen different Ethernet cards. The prospect of manually building ce.conf doesn't fill me with enthusiasm. After all, no one likes a GigE card running at 100 Mbit/sec half-duplex, do they?

An issue exists where a ce GigE instance, which is forced to any speed/duplex setting but is not plumbed, will constantly "flap" (going up and coming down) even though it is not configured and may not even be plugged in. This results in a huge storm of "Link Up" "Link Down" messages on the console and in /var/adm/messages, which is a nuisance at the best of times.

This article covers the ce_settings.ksh script. It configures ce.conf by looking at /etc/path_to_inst to work out what cards you have, and then assumes any 100 Mbit ce cards need to be 100 Mbit/sec full duplex, and that the GigE cards (if any) can be left alone to auto-negotiate their own settings. From this the script will build you a ce.conf file, automatically configuring all your ce instances correctly.

A couple of things need to be covered before we start looking at the script. I know many people reading this will be thinking "string manipulation," and then thinking "Why isn't he using perl?" The simple answer is, when deploying this on several hundred servers, I know they will all have ksh and that it will be in the same place on every machine and behave in the same way. Unfortunately, the same cannot be said about perl -- certainly not on the estates I've dealt with. Also a shell script is an easier thing for more junior sys admins to handle; regular expressions in perl can be a source of constant bemusement and frustration for many.

Following on from that, we need to look at how we can split strings in korn. Here we're calling on a shell built-in called IFS -- the Internal Field Separator. By dropping a string into an array, we can change the value of IFS and thus split out the array into individual strings, assigning them to variables. This gives us a very flexible way of chopping and changing things, which we can apply to other system shell scripts.

Taking the above into account, it can be seen that by writing the script in korn we get one major advantage -- it's not too much work to take this and morph it into a Bourne script, which can be used as part of a custom JumpStart install -- something fun for you to try once you're happy with how this script does its stuff. The only hint I'll give you here is that Bourne doesn't have arrays.

Right, let's look at /etc/path_to_inst. This file contains the physical name of each hardware component known to the Solaris OS. The exact format is:

"physical-name" instance-number "driver-binding-name"

Here's a sample ce entry from /etc/path_to_inst on a Sun Fire 15K server:

"/pci@fc,700000/pci@1/network@0" 0 "ce"

Next, we'll examine the format of the ce.conf file:

name="pci108e,abba" parent="<device path up to /network@part>"
unit-address="<instance-number>" <configuration parameters>

Let's look at a sample line from ce.conf, showing all the configuration parameters:

name="pci108e,abba" parent="/ssm@0,0/pci@1e,600000" unit-address="1"
adv_autoneg_cap=0 adv_1000fdx_cap=0 adv_1000hdx_cap=0 adv_100fdx_cap=1
adv_100hdx_cap=0 adv_10fdx_cap=0 adv_10hdx_cap=0;

We can see instantly that path_to_inst contains all the information we need to build ce.conf -- we just need to get it out and edit it a bit. ce.conf will have a line per driver instance. Normally you can just specify the configuration parameters you need to set, but ce_settings.ksh will put them all in. Not only does this make it easier to read and understand, but it also makes it easy to manually edit for special cases, like a GigE card that must run at 100 Mbit/sec until the switch is upgraded.

Now, let's look at ce_settings.ksh and cover the specifics of what it needs to do. Our basic tasks are:

• Get the device path from /etc/path_to_inst
• Work out what settings we need to apply
• Build ce.conf

In the first section of the script, we define where our path_to_inst and ce.conf files should be -- this is useful for testing, allowing us to slot in dummy files. Then, we back up the value of IFS, and then we back up the existing ce.conf, if there is one.

Next, we define our basic parameters. The only one that will ever change is the auto-negotiation setting -- it will be enabled for GigE and disabled for 100 Mbit cards.

Next, we start to build our output. First of all, we need to scan and pull apart path_to_inst so we can grab the physical name. Then, we need to pull that apart to get the relevant device path for our driver instance. The first major problem here is that the device path is not the same across all machines. Midframes (3x00, 4x00 and 6x00) address devices differently compared to their bigger brothers (Sun Fire 12K, 15K, 25K servers).

Since the 100 Mbit card has dual Ethernet and SCSI ports (compared to the single Ethernet port on the GigE cards), it will always have an extra segment to its device path. We can use this to work out what sort of card we have when splitting up the physical name from path_to_inst.

At this stage, we've gathered all the information we need so we can construct our ce.conf file. The script will loop through all the ce entries in path_to_inst, looking at each one in turn to decide if it's a 100 Mbit or Gig Ethernet card, and adding a line to ce.conf accordingly.

All that's needed is a quick manual check of ce.conf to make sure it's doing what we want. Then, reboot and away we go!

The information and links on this page have been provided by a BigAdmin user. The submitter is solely responsible for such information and links. Sun is not responsible for the availability of external sites or resources, and does not endorse and is not responsible or liable for any content, advertising, products, or other materials on or available from such sites or resources. Sun will not be responsible or liable, directly or indirectly, for any actual or alleged damage or loss caused by or in connection with use of or reliance on the information posted here, or goods or services available on or through any external site or resource.

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.