OpenSolaris Project Brussels: A Uniform Interface for Driver Administration Through the dladm Command
May 2008
The Driver Configuration Framework component of the Brussels project became available in SXCE
build 83 and is included in the OpenSolaris release. The Framework component provides
users the following benefits when configuring drivers that use the GLDv3 framework:
A single user interface, dladm, is used to configure network driver properties.
A uniform syntax is used regardless of the properties: dladm subcommandpropertiesdata-link.
The dladm interface is applied to both public and private properties of the driver.
Using the tool on a specific driver does not disrupt other network connections.
This article elaborates on the administrative features provided by the Brussels project.
Introduction and Problem Description
Prior to the Brussels implementation, configuring data-link parameters on a machine running the
Solaris Operating System (Solaris OS) was needlessly difficult. Commands and interfaces were cryptic
and inconsistent across the drivers, and the process often required service interruption
to all connected networks, regardless of whether the connecting link was actually being impacted
by the change.
An example of the deficiencies in these administrative methods is the refining of
interface maximum transmission unit (MTU) that is typically done to enable support for
the Jumbo Frame MTU. The complications are evident as described in an inquiry in the Sun Managers Mailing List.
Overview of NIC Driver Properties
NIC driver properties that configurable by using the dladm command fall into one
of two categories:
Public properties that can be applied to any driver of the given media type such as link speed, autonegotiation for Ethernet, or the MTU size that can be applied to all data-link drivers.
Private properties that are particular to a certain subset of drivers for a given media type. These properties can be specific to that subset because they are closely related either to the hardware that is associated with the driver or to the details of the driver implementation itself, such as debugging-related tunables.
Properties of a NIC driver are typically set with default values. However, certain
networking scenarios might require you to change specific property settings of a NIC.
These property settings can be either public or private properties. For example, a
NIC might be communicating with an old switch that does not properly perform
autonegotiation. Or, a switch might have been configured to support Jumbo frames. Or,
driver specific properties that regulate packet transmission or packet receiving might need to
be modified for the given driver. With the implementation of the Brussels project,
these settings can now be reset by a single administrative tool, dladm.
dladm Subcommands to Administer NIC Properties
For NIC drivers that have been converted to the GLDv3 framework, properties are
configured by using the dladm command. This command enables you to configure the
properties dynamically without causing any network disruption on other NICs of similar types.
The values that you set are stored in a dladm repository and persist
even after you reboot the system or unplumb the interface. Therefore, use dladm
as the preferred command to configure NICs, instead of the ndd command.
To administer NIC drivers, you use the following dladm subcommands:
dladm show-linkprop displays the properties that are associated with the data link.
dladm set-linkprop sets values for specified data-link properties.
dladm reset-linkprop restores property settings to the default values.
dladm show-ether displays Ethernet parameter settings of a data link.
For more information about these commands, see the dladm(1M) man page.
Note - Customizing NIC properties by using the dladm command is supported only in network
drivers that have been converted to the GLDv3 framework, such as the following:
bge - supported in SXCE build 83
nge - supported in SXCE build 86
e1000g - supported in SXCE build 88
nxge - supported in SXCE build 88
Work continues to make other drivers become supported in the Brussels implementation. To
confirm whether your specific driver supports this feature, refer to the driver's man
page.
The following section provides procedures to set certain NIC driver properties. The selected
properties are public and common to all NIC drivers. A separate section describes
driver specific properties and also provides procedures to configure selected private properties of
the e1000g driver..
Procedures and Examples
The following section provides procedures with examples that show how to configure public
drivers by using the dladm command.
How to Enable Support for Jumbo Frames
Enabling support for Jumbo frames in a network setup is a common
task for most network scenarios. Support for Jumbo frames requires increasing the size of
a data link's maximum transmission unit (MTU). The following procedure includes the use
of customized names to identify data links, a feature that is introduced by
the Clearview project. This Clearview overview document introduces the concept of customized link names.
On the system that has the link whose MTU you want to
modify, assume the System Administrator role.
The System Administrator role includes the Network Management profile. To create the role
and assign the role to a user, see “Chapter 9, Using Role-Based Access
Control (Tasks) in System Administration Guide: Security Services in the docs.sun.com site.
Identify the specific Ethernet device whose MTU size you need to reset, display
the links in the system.
# dladm show-phys
You might need to perform this step if your network configuration uses customized
names for data links. With customized names, data links are no longer necessarily
identified by their hardware-based names. For example, the Ethernet device is bge0. However,
the data link over the device is renamed net0. Therefore, you would need
to configure the MTU size of net0. Refer to the documentation in the Clearview project page in OpenSolaris
for examples of configuration tasks on data links that use customized names.
(Optional) Display the data link's current MTU size and other properties.
To display a specific property of a data link, use the following syntax:
dladm show-linkprop -p propertydata-link
This command displays the settings of the property that you specify.
To display several selected properties of the data link, use the following syntax:
# dladm show-link data-link
This command displays data-link information, including MTU size.
Note - See How to Display Data Link in the Project Clearview documentation page for additional examples
of the use of the dladm show-link syntax to display data-link information.
Unplumb the interface that is configured over the data link.
# ifconfig interface unplumb
Change the value of the link's MTU size to 9000, the value
for Jumbo frames.
# dladm set-linkprop -p mtu=9000 data-link
Plumb the IP interface over the link.
# ifconfig interface plumb IP-address up
For additional options that you can use with the ifconfig command, see the
ifconfig(1M) man page.
(Optional) Verify that the interface uses the new MTU size by using one
of the command syntaxes in Step 3.
# dladm show-linkprop -p mtu data-link
(Optional) Display the link's current Ethernet settings.
# dladm show-ether data-link
Example 1 Enabling Support for Jumbo Frames
The following example of enabling support for Jumbo frames builds on the following
scenario:
The system has two bge NICS: bge0 and bge1.
The device bge0 is used as a primary interface, while the device bge1 is used for test purposes.
You want to enable support for Jumbo frames on bge1, while you retain the default MTU size of the primary interface.
The network configuration uses customized names for data links. The link name of bge0 is net0. The link name of bge1 is web1.
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 ether up 100Mb full bge0
itops1 ether up 100Mb full qfe3
web1 ether up 100Mb full bge1
# dladm show-linkprop -p mtu web1
LINK PROPERTY VALUE DEFAULT POSSIBLE
web1 mtu 1500 1500 --
# ifconfig web1 unplumb
# dladm set-linkprop -p mtu=9000 web1
# ifconfig web1 plumb 10.10.1.2/24 up
# dladm show-link web1
LINK CLASS MTU STATE OVER
web1 phys 9000 up --
Notice that the MTU value is now 9000. In this example, the
dladm command enabled you to change web1's MTU size directly. The previous method would
have required you to unplumb net0 as well, which would have unnecessarily disrupted
the primary interface's operations.
How to Change Link Speed Parameters
Most network setups consist of a combination of systems with varying speed capabilities.
For example, the advertised speed between an older system and a newer system
might need to be changed to a lower setting to allow communication. By
default, all the speed and duplex capabilities of a NIC card are
advertised. This procedure shows how to turn off the gigabit capabilities and advertise only
the megabit capabilities.
On the system that has the NIC whose properties you want to
modify, assume the System Administrator role.
(Optional) Display the current status of the property you want to modify.
# dladm show-linkprop -p propertydata-link
To advertise lower speed capabilities, turn off the higher speed capabilities to prevent
them from being advertised.
# dladm set-linkprop -p property=value1data-link
Example 2 Disabling Advertisement of a NIC's Gigabit Capabilities
This example shows how you can prevent the link web1 from advertising gigabit
capabilities.
# dladm show-linkprop -p adv_1000fdx_cap web1
LINK PROPERTY VALUE DEFAULT POSSIBLE
web1 adv_1000fdx_cap 1 -- 1,0
# dladm show-linkprop -p adv_1000hdx_cap web1
LINK PROPERTY VALUE DEFAULT POSSIBLE
web1 adv_1000hdx_cap 1 -- 1,0
The properties that advertise the link's gigabit capabilities are adv_1000fdx_cap and adv_1000hdx_cap.
To disable these properties from being advertised, you would type the following commands:
Listing the Ethernet parameter settings would display the following output:
# dladm show-ether web1
LINK PTYPE STATE AUTO SPEED-DUPLEX PAUSE
web1 current up yes 1G-f both
How to Obtain Status Information About NIC Properties
You can obtain information about the NIC driver's properties by displaying either the
Ethernet parameter settings or the link properties.
On the system that has the NIC whose properties you want to
modify, assume the System Administrator role.
To obtain information about the Ethernet parameter settings, use the following command:
# dladm show-ether [-x] data-link
where the -x option includes additional parameter information about the link. Without the
-x option, only the current parameter settings are displayed.
To obtain information about all the properties of the link, use the following
command:
# dladm show-linkprop data-link
Example 3 Displaying Ethernet Parameter Settings
This example display an extended list of parameter information about a specified link.
# dladm show-ether -x web1
LINK PTYPE STATE AUTO SPEED-DUPLEX PAUSE
web1 current up yes 1G-f both
-- capable -- yes 1G-fh,100M-fh,10M-fh both
-- adv -- yes 100M-fh,10M-fh both
-- peeradv -- yes 100M-f,10M-f both
With the -x option, the command also displays the built-in capabilities of the
specified link, as well as the capabilities that are currently advertised between the
host and the link partner. The following information is displayed:
For the Ethernet device's current state, the link is up and functioning at 1 gigabits per second at full duplex. Its autonegotiation capability is enabled and has bidirectional flow control, in which both host and link partner can send and receive pause frames.
Regardless of the current setting, the capabilities of the Ethernet device are listed. The negotiation type can be set to automatic, the device can support speeds of 1 gigabits per second, 100 megabits per second, and 10 megabits per second, at both full and half duplex. Likewise, pause frames can be received or sent in both directions between host and link partner.
The following capabilities of web1 are advertised: autonegotiation, speed-duplex, and flow control of pause frames.
Similarly, the link partner advertises the following capabilities: autonegotiation, speed-duplex, and flow control of pause frames.
Example 4 Displaying Link Properties
This example shows how to list all the properties of a link.
If you want to display only a specific property, you use the -p
option with the specific property that you want to monitor.
The settings for the speed and duplex capabilities of the link are
manually configured on the enabled-speed properties which are labeled en_*_cap. For example, en_1000fdx_cap is
the property for the gigabit full-duplex capability, and en_100hdx_cap is the property for the
100 megabits half-duplex capability. The settings of these enabled speed properties are advertised
between the host and its link partner by corresponding advertised speed properties, which
are labeled adv_*_cap such as adv_1000fdx_cap and adv_100hdx_cap. Normally, the settings of a
given enabled speed property and the corresponding advertised property are identical. However, if
a NIC supports some advanced features such as Power Management, those features might
set limits on the bits that are actually advertised between the host and
its link partner. For example, with Power Management, the values of the adv_*_cap
properties might only be a subset of the values of the en_*_cap properties.
For more details about the enabled and advertised speed properties, see the
dladm(1M) man page.
Configuring Properties Specific to the e1000g Driver
Brussels support for the e1000g driver is available in SXCE build 88.
The driver is widely used on many Sun platforms to support the Intel
PRO/1000 Gigabit NICs, such as the Sun Fire X4200 server, the Sun Netra X4450 server,
and the Sun Fire T2000 server.
Previously, the e1000g driver, like many other NIC drivers, was configured by using
the ndd command and by defining property settings in the driver's e1000g.conf configuration file.
This driver is now converted to the GLDv3 framework. Thus, the administration of
its configuration can be uniformly performed just as other similarly converted drivers with
the dladm command.
The same command can be used to adjust parameters that are specific to
the e1000g driver. For example, you might need to configure the driver to
use Direct Memory Access (DMA) binding for transmission instead of the bcopy mode.
Likewise, you might need to reset certain interrupt parameters to improve the performance
of the driver. See the procedures that show how to configure these parameters.
Note - The nxge driver has become Brussels-supported in SXCE build 88. The driver, which
is widely used on many platforms, also has nxge-specific properties. The following procedures about
configuring private properties apply to the e1000g driver. However, the general procedures can
also apply to the nxge driver.
Procedures and Example
This section provides procedures and an example that show how to configure selected
private properties of the e1000g driver.
How to Set the e1000g Driver to Use Direct Memory Access Binding
Bulk traffic, such as file transfers, normally involves negotiation of large packets across
the network. In such cases, you can obtain better performance from the e1000g
driver by configuring it to automatically use DMA binding.
In the following procedure, a threshold is defined for packet fragment sizes. If
a fragment size surpasses the threshold, then DMA binding is used for transmitting.
If a fragment size is within the threshold, then bcopy mode is
used, where the fragment data is copied to the preallocated transmit buffer. For
more details about this threshold and other tunables, see the e1000 tunables list in OpenSolaris.
To set the threshold, perform the following steps:
On the system that has the NIC whose properties you want to
modify, assume the System Administrator role.
Set the appropriate value for the _tx_bcopy_threshold property.
Parameters that regulate the rate at which interrupts are delivered by the e1000g
driver also affect network and system performance. Typically network packets are delivered to
the upper layer of the stack by generating an interrupt for every packet.
In turn the interrupt rate, by default, is automatically adjusted by the GLD
layer in the kernel. However, this mode might not be desirable in all
network traffic conditions. For a discussion of this issue, refer to this document
(http://www.stanford.edu/class/cs240/readings/mogul.pdf) that was presented at the USENIX technical conference in 1996. Thus, in
certain circumstances, setting the interrupt rate manually becomes necessary to obtain better performance.
To define the interrupt rate, you set the following parameters:
_intr_throttling_rate determines the delay between interrupt assertions regardless of network traffic conditions.
_intr_adaptive determines whether automatic tuning of the interrupt throttling rate is enabled. By default, this parameter is enabled.
On the system that has the NIC whose driver properties you want to
modify, assume the System Administrator role.
If necessary, identify the device whose driver property you want to modify.
# dladm show-phys
Disable automatic tuning of the interrupt throttling rate.
Note - When automatic tuning of the interrupt throttling rate is enabled, then any value
that is set for the parameter _intr_throttling_rate is ignored.
Unplumb the network interface.
Set the value for the minimum inter interrupt level.
Note - The default value of the _intr_throttling_rate parameter is 550 on SPARC based systems
and 260 on x86 based systems. Setting the minimum inter-interrupt level to 0
disables the interrupt throttling logic.
Plumb the interface and configure an IP address for the interface.
(Optional) Display the threshold's new settings.
Example 5 Configuring for DMA Binding and Setting the Interrupt Throttling Rate
This example uses an x86 based system with an e1000g NIC. The driver
is configured with a threshold setting toggle between using DMA binding or the
bcopy mode for transmitting packets. The setting for the interrupt throttling rate is
also modified. Further, the e1000g data link has been renamed with a
customized name. Therefore, the configuration is performed on the data link by referring
to the customized name, public0.
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
public0 ether up 100Mb full e1000g0
# dladm show-linkprop -p _tx_bcopy_threshold public0
LINK PROPERTY VALUE DEFAULT POSSIBLE
public0 _tx_bcopy_threshold 512 512 --
# dladm show-linkprop -p _intr-throttling_rate
LINK PROPERTY VALUE DEFAULT POSSIBLE
public0 _intr-throttling_rate 260 260 --
# ifconfig public0 unplumb
# dladm set-linkprop -p _tx_bcopy_threshold=1024 public0
# dladm set-linkprop -p _intr_adaptive=0 public0
# dladm set-linkprop -p _intr-throttling_rate=1024 public0
# ifconfig public0 plumb 10.10.1.2/24 up
# dladm show-linkprop -p _tx_bocopy_threshold=1024 public0
LINK PROPERTY VALUE DEFAULT POSSIBLE
public0 _tx_bcopy_threshold 1024 512 --
# dladm show-linkprop -p _intr_adaptive public0
LINK PROPERTY VALUE DEFAULT POSSIBLE
public0 _intr-adaptive 0 1 --
# dladm show-linkprop -p _intr-throttling_rate
LINK PROPERTY VALUE DEFAULT POSSIBLE
public0 _intr-throttling_rate 1024 260 --
Ongoing Work
The Brussels ndd compat component will provide additional functionality in the Framework for legacy
support of the ndd(1m) commands so that drivers will not have to include
complex and undocumented interfaces to support the ndd command.
Also, the Brussels Persistence component will add additional functionality to allow the property
settings to be automatically applied across reboot and driver restart.
Future articles will describe the details of the Framework, as well as the
methods used to provide ndd compatibility and persistent property settings.
Work also continues to provide a GUI to administer data links. A
prototype of the GUI is available at the OpenSolaris project page. The prototype
is an interactive GUI for demonstration purposes. Please send feedback to the Brussels discussion list.
Access to the Brussels-supported command-line interfaces is now available in the OpenSolaris 2008.05
release.
Discuss and comment on this resource in the BigAdmin Wiki
Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.
Print-friendly Version
