So You Wanna Write Solaris Device Drivers?James Liu, June 2008 Contents:
Abstract: This article focuses on providing device drivers for the Solaris OS, beginning with foundational information about the Solaris kernel architecture in general and device drivers in particular. This article then develops a minimal skeleton driver. Getting Acquainted with Solaris Device DriversWhenever a bunch of tech-savvy folks get together and compare operating system kernels, there is a good chance the issue of device support will come up. While the kernel is the core piece of software that bridges the gap between hardware and software, it's the small chunks of code known as device drivers that enable a platform's peripherals to interface with users. Access by device drivers to and from the kernel relies on a well-defined set of interfaces [1]. The Solaris OS was first released in 1992, so it has a mature set of interfaces for a wide variety of devices. In fact, some developers might have started out writing graphics, network, and storage controller drivers using the Solaris OS years ago. However, until recently, the number of experienced Solaris device driver writers has been a relatively small elite group. Some critics cite the Solaris OS being closed-source as the main reason. But since the release of core kernel code into the open source community in June 2005 [2], the number of programmers engaged in Solaris driver development has grown and attracted a more casual user and developer interest in drivers. In this article, we get acquainted, or perhaps for some readers, reacquainted with Solaris device drivers. We take a pragmatic approach that looks at device drivers, first from the point of view of a system installer or administrator who wants to configure a binary driver to make it work with a particular device, and second, from the point of view of a developer who wants to port a driver to the Solaris OS or write a driver from scratch for the Solaris OS. We hope to demystify Solaris drivers and make driver development as straightforward as application development. A First Look at Kernel ModulesFor the uninitiated, the first approach to writing a device driver can be very intimidating. You do need to overcome a steep learning curve before you can successfully write device drivers. Part of the obstacle is that device drivers do interact with both the kernel and hardware. The standards for quality are necessarily higher since poor programming of a driver module can kill the whole kernel and panic the entire OS. Applications, however, either in the Solaris OS or in other well architected operating systems, are segmented from the kernel. A poorly written program can consume undue system resources, dump core, or mysteriously exit, but rarely will it bring down the kernel or other independently running programs. Aside from the fact that a device driver is code that runs typically in the kernel, there are many similarities between writing a device driver and writing a standard application. Like applications, device drivers are binary code, usually written in C, and compiled and linked to libraries. The Solaris Device TreeThe Solaris OS organizes and categorizes drivers into a device hierarchy known as
a device tree. We can think of this just like a file directory structure.
Just as a file hierarchy has directories, subdirectories, and files, each point in
this tree represents a node that might be a branch point, sub-branch point,
or endpoint. The device file system, or
Figure 1 Sample Device Tree Types of Device DriversNexus drivers typically control bridge, bus, hub, and switch chips on the motherboard and are usually supplied by the Solaris OS. Nexus drivers are not usually implemented by third party driver writers. The Solaris OS includes nexus drivers for various bus and controller subsystems such as:
Because the Solaris OS supplies the nexus drivers, most developers focus on leaf node drivers, which come in two basic types: character and block. Character device drivers are used to access serial ports and custom I/O boards. Character device drivers can be implemented as non-STREAMS drivers or as STREAMS drivers. (This introduction won't go deeply into STREAMS drivers. Instead, that will be presented in a later article.) Block device drivers are often used to access filesystem disks and storage. What's important to understand is that device drivers enable the Solaris kernel to access lower level hardware. At a high level, the kernel provides applications with a standard set of APIs that perform networking, device control, process and memory management, and access to storage. However, in order for the kernel to use a particular vendor's device, it needs to have code that provides that low-level functionality. That code is what maps the standard kernel library function calls into actions that will take place with a hardware device. This is the job of the device driver. The following figure shows an overview of the Solaris OS architecture. Applications are at the user level. Device drivers are in the kernel. The hardware level includes CPU, disks, NICs.
Figure 2 Overview of the Solaris OS Architecture The Solaris DDI/DKIPerhaps the main difference between application programming and driver programming is in freedom.
The implementation of application programs usually is constrained only by a minimal set
of rules that govern how the application interacts with the user's shell or
the user's graphical interface. How the application actually does its job can be
very flexible and virtually independent of the OS from the developer's perspective. For
example, we could write a standalone program that has its own subroutines and
we only implement the standard In driver programming, we have much less freedom and less of a safety net. We are no longer writing standalone code. Instead, we're writing software extensions that will enable the Solaris kernel to access a device. So when the kernel needs to use a device, we need to provide a driver module that implements an agreed-upon set of subroutines to enable the kernel to predictably control that device. Those subroutines follow a strict set of naming and coding constraints. Without those strict rules, the kernel would need to implement custom subroutines for every driver it would possibly load. Implementing custom subroutines in each driver would make the kernel very sensitive to changes in the device, and make it very impractical for an OS to keep up with a growing market for third-party peripheral devices. Today, all operating systems define relatively standardized device driver interfaces (DDIs) and device-kernel interfaces (DKIs). The Solaris DDI/DKI has two parts. One part provides a stable set of callable function subroutines that simplify development. These functions are also portable, so using a particular DDI/DKI function in a driver allows a Solaris device driver to be compiled and run on a 32-bit x86 platform, on a 64-bit x86 platform, or on a SPARC platform with little or no change to source code. The DDI/DKI will do the right thing when running on the particular platform. The standard C libraries are still available to device driver writers, but when devices need to access I/O and manipulate bits within data, not using the DDI/DKI function calls can lead to big problems with sizes, endianness, and portability between platforms. The following figure shows a block diagram of a typical driver module with basic Solaris DDI/DKI.
Figure 3 Driver Module Block Diagram In addition to these callable functions, the other half of the DDI/DKI defines interfaces for callback routines that developers must implement in their drivers. The kernel expects each driver to implement these interfaces and will invoke or “call back” those functions during known phases of the driver module's lifecycle. These callbacks include interfaces needed to handle loading and unloading operations, initializing and finishing, attaching and detaching, setup of interrupt handling, and standard I/O control to the physical device. Finally, developers need to implement additional interfaces in order for the driver to attach to a particular type of nexus controller such as SCSI, SATA, USB, LAN. A natural question that arises from discussions of driver functions and interfaces is,
How and where are driver states managed in the driver? Three primary data
structures track driver state in the Solaris OS. Every driver references these three
data structures. At the top level is the module operations or Driver LifecycleLet's examine the driver lifecycle in a bit more detail. When the kernel
loads a driver, it expects the driver to implement certain subroutines and pass
in a reference to the module's _info(9E) _init(9E) _fini(9E) After the system loads a module (and also when the system is going to unload a module), the kernel looks for one of two device-specific callbacks to do the actual hardware initialization or finalization. These callbacks are similar to those required for the system to manage modules, but these are designed to manage the hardware. We can summarize the next two mandatory callbacks all drivers must implement as: attach(9E) detach(9E) At a minimum, all device drivers must implement the above five functions as defined in the DDI/DKI. The DDI/DKI also specify some standard functions for communicating with the device and performing I/O to and from the kernel to the device. These functions are: open(9E) close(9E) read(9E) write(9E) All of the above callbacks are function names with specific arguments that the Solaris OS requires of drivers so that the kernel knows how to notify and perform I/O with the driver. You can run the following command to see the signature of the function and arguments, read a synopsis, and possibly see some sample code showing how the function is used within a driver: % man -s 9E function_name More information can be found in the system documentation that accompanies the Solaris OS. You can also obtain more detailed references online [1,4,5]. System Administration - How the Solaris OS Manages DriversThis section discusses the device tree, driver management files, and driver management commands. An Overview of 32-bit and 64-bit Drivers in the KernelOn x86 platforms, the Solaris OS can run in both 32-bit and 64-bit mode depending on the processor architecture. By default, if we have a 64-bit capable platform, the Solaris OS boots into 64-bit mode and thus requires all its drivers to be 64-bit modules. This is because kernel modules share the same memory address space as the rest of the kernel. If the platform is only 32-bit capable, then the Solaris OS boots into 32-bit mode and loads only 32-bit modules. The general rule is that driver binaries must match the kernel bit-wise in pointer size. However, because applications do not run inside the kernel, a 64-bit kernel can run both 32-bit and 64-bit applications. A 32-bit kernel can only run 32-bit applications. It is possible to force the boot of the Solaris OS into
32-bit mode on a 64-bit system by editing the grub boot menu during
boot time and editing the grub command line. Either edit the file Where We Deploy Driver ModulesWhen you finish compiling a driver and want to deploy the module, you need to know where the Solaris OS keeps the modules. For x86 and x64 platforms, the Solaris OS expects most driver modules to be in the following directories:
There are other directories we could copy modules to. The above directories are the recommended places for x86 driver binaries. Typically, when we build a driver module, we generate both 32-bit and 64-bit binaries and deploy them into the directories shown above, even if we only intend to run in one mode or the other. This makes the driver available for both types of platforms if we should ever copy the binaries over to another platform, or boot into a different bit-mode as explained above. System Driver Interaction and the Device TreeWe mentioned above in The Solaris Device Tree that the Solaris OS actually implements a representation
of the device tree as a file system known as While the The reason to have two separate directories for device files is that the
same logical device name might be expected by the system or by
applications. But on a different platform, that logical device might be serviced by
a completely different device from a different manufacturer using a different driver module. Consider
the following audio example. If audio is working on a system, we will
find Note that while the contents in Driver Management FilesUp to this point, some readers might still be scratching their heads about how the Solaris OS knows which module to load. Like any other OS, there is a master database file that maps driver modules to the actual device as defined by the Vendor and Device ID extracted either through the PCI bus or by another device hub such as the USB nexus. This text file is located in: /etc/driver_aliases Another text file maps the actual physical device name (actually a path in
/etc/path_to_inst The system maps driver names to major numbers in the following file: /etc/name_to_major The system tracks a database of default permissions or access control lists for each instance of a driver in the following file: /etc/minor_perms While the Commands to Manage, Add, Remove, Update and Check DriversBelow is a list of basic commands we use to maintain devices. These commands are the preferred way to manipulate the driver management files discussed above, rather than directly editing those files.
Basics of Building a DriverTwo steps are required when building a driver binary. The first step is to generate a driver binary object. The second step is to link the binary object. Step 1: Compiling the Binary ObjectTo generate a driver binary object we can conveniently choose either the Sun
Studio compiler [7] or the GNU C compiler ( While this is not supported or guaranteed, driver binaries compiled on the Solaris OS usually are forward compatible along the same version of the OS. In other words, a Solaris 10 1/06 driver binary should work on the Solaris 10 5/08 OS with no changes, as long as it uses only committed DDI/DKI interfaces. The same applies to Solaris Express or OpenSolaris drivers compiled for earlier versions. These should run with no changes or recompile on forward versions of the OS. Note that the reverse is not true: A driver compiled against a newer OS might not run on an older OS. We summarize the compiler commands with a few examples below. For a complete summary, see the end of the first chapter of [5]. If you are compiling for a 64-bit x86 architecture using Sun Studio 10
or Sun Studio 11, use both the # cc -D_KERNEL -xarch=amd64 -xmodel=kernel -c foobar.c We must use the If you are compiling for a 64-bit x86 architecture using the GNU C compiler, use the following compile command: # gcc -D_KERNEL -ffreestanding -m64 -c foobar.c If you are compiling for a 32-bit architecture using the Sun Studio C compiler, use the following compile command: # cc -D_KERNEL -c foobar.c If you are compiling for a 32-bit architecture using the GNU C compiler, use the following compile command: # gcc -D_KERNEL -ffreestanding -c foobar.c Step 2: Linking the Binary ObjectAll driver binaries must then be linked to resolve symbols in the final binary object. For basic modules that don't depend on any other kernel driver frameworks or other driver modules, the command is simply: # ld -r -o foobar foobar.o In many instances, a driver module also depends on other driver modules or must be dynamically linked to other kernel libraries. We present two examples below. The following example shows how to link an audio driver binary object that
depends on the Solaris OS audio support, mixer, and mixer source modules to
generate a binary. The # ld -r -dy -N misc/audiosup -N misc/mixer \
-N misc/amsrc -o audiodrv audiodrv.o
The following example shows how to link a typical network driver with the Solaris OS generic LAN driver (GLD) layer. # ld -r -dy -N misc/gld -o nicdrv nicdrv.o You can easily check whether a given module is 32-bit or 64-bit
simply by using the Putting It All Together - A Sample DriverThis section shows a skeleton driver that implements the minimum number of interfaces required. This section then shows the header file for that skeleton driver and the configuration file for that driver. A Skeleton DriverTypically, a driver is composed of one or more C files that
implement the functions used by the kernel to access the driver. It is
usual practice to find the module, device, and driver operations structures declared explicitly in
each C file where the operations structure is used and not in
the header file. It is common practice to declare the name of the
driver as a prefix to functions and global variables that are specific to
that driver. For example, the /*
* foobar.c - example skeleton driver
*/
#include <sys/errno.h>
#include <sys/conf.h>
#include <sys/cmn_err.h>
#include <sys/modctl.h>
#include <sys/sunddi.h>
#include <sys/stat.h>
#include "foobar.h"
static int foobar_attach(dev_info_t *dip, ddi_attach_cmd_t cmd);
static int foobar_detach(dev_info_t *dip, ddi_detach_cmd_t cmd);
static struct cb_ops foobar_cb_ops = {
nodev, /* no open */
nodev, /* no close */
nodev, /* no strategy (only for block drivers) */
nodev, /* no print */
nodev, /* no dump (only for block drivers) */
nodev, /* no read */
nodev, /* no write */
nodev, /* no ioctl */
nodev, /* no devmap */
nodev, /* no mmap */
nodev, /* no segmap */
nochpoll, /* no chpoll entry point */
ddi_prop_op, /* Use system-supplied prop_op entry point */
NULL,
D_NEW | D_MP
};
static struct dev_ops foobar_ops = {
DEVO_REV,
0,/* reference count: always 0 initially */
nulldev, /* No getinfo entry point */
nulldev, /* DEPRECATED: identify entry point */
nulldev, /* no probe entry point */
foobar_attach,
foobar_detach,
nodev, /* no reset entry point */
&foobar_cb_ops, /* Reference the cb_ops defined above */
(struct bus_ops *)NULL /* Not a nexus driver, so no bus_ops */
};
extern struct mod_ops mod_driverops;
static struct modldrv Modldrv = {
&mod_driverops, /* Use system-supplied mod_driverops */
"foobar driver v" FOOBAR_VERSION, /* Module Name/Version */
&foobar_ops,
};
static struct modlinkage Modlinkage = {
MODREV_1,
&Modldrv,
NULL
};
/*
* This bit of static data is used by the DDI to
* keep track of the per-instance driver "soft state"
*/
static void *soft_statep;
int
_init(void)
{
/*
* Initialize the soft state APIs so we can
* allocate soft state in foobar_attach()
*/
if (ddi_soft_state_init(&soft_statep,
sizeof (struct foobar_state), 1)
!= DDI_SUCCESS)
return (DDI_FAILURE);
if (mod_install(&Modlinkage) != 0) {
ddi_soft_state_fini(&soft_statep);
return (-1);
}
return (0);
}
int
_info(struct modinfo *modinfop)
{
return (mod_info(&Modlinkage, modinfop));
}
int
_fini(void)
{
ddi_soft_state_fini(&soft_statep);
return (mod_remove(&Modlinkage));
}
static int
foobar_attach(dev_info_t *dip, ddi_attach_cmd_t cmd)
{
/* Use the instance number as the minor number */
instance = ddi_get_instance(dip);
if (ddi_soft_state_zalloc(soft_statep, instance)
== DDI_FAILURE)
return (DDI_FAILURE);
softp = ddi_get_soft_state(soft_statep, instance);
ASSERT(softp != NULL);
if (ddi_create_minor_node(dip, FOOBAR_MINOR_NAME,
S_IFCHR, instance, DDI_PSEUDO, 0)
!= DDI_SUCCESS) {
cmn_err(CE_WARN, "Minor creation failed!");
return (DDI_FAILURE);
}
softp->init_state |= FOOBAR_INIT_MINOR;
softp->dip = dip;
mutex_init(&softp->mutex, NULL, MUTEX_DRIVER, 0);
softp->buffer = (char *)kmem_alloc(FOOBAR_BUFLEN,
KM_SLEEP);
ddi_report_dev(dip); /* Announce we've attached! */
return (DDI_SUCCESS);
}
static int
foobar_detach(dev_info_t *dip, ddi_detach_cmd_t cmd)
{
int instance;
struct foobar_state *softp;
if (cmd != DDI_DETACH)
return (DDI_FAILURE);
/* Use the instance number as the minor number */
instance = ddi_get_instance(dip);
softp = ddi_get_soft_state(soft_statep, instance);
ASSERT(softp != NULL);
if (softp->init_state & FOOBAR_INIT_MINOR) {
/* Remove minor nodes associated with dip */
ddi_remove_minor_node(dip, NULL);
}
ASSERT(softp->buffer != NULL);
kmem_free(softp->buffer, FOOBAR_BUFLEN);
ddi_soft_state_free(soft_statep, instance);
return (DDI_SUCCESS);
}
The Skeleton Driver Header FileThe header file contains remaining declarations such as those defined below. /*
* foobar.h - example skeleton driver header
*/
#ifndef _FOOBAR_H
#define _FOOBAR_H
#define FOOBAR_INIT_MINOR 0x00000001
#define FOOBAR_VERSION "1.0"
#define FOOBAR_BUFLEN 1024
#define FOOBAR_MINOR_NAME "xyzzy"
struct foobar_state {
dev_info_t *dip; /* Opaque dev_info pointer */
int init_state; /* See FOOBAR_INIT_* */
kmutex_t mutex; /* driver lock */
char *buffer; /* message buffer */
};
#endif /* #ifdef _FOOBAR_H */
The Skeleton Driver Configuration FileThe last file is the driver configuration file. In this example, the driver
configuration file is # # foobar.conf - example skeleton driver conf file # name="foobar" parent="pseudo" instance=0; Recall that this What's Next?In the next article in this series, we'll cover some more frameworks for specific types of drivers people write and provide some insights into when to use which framework for a driver. Meanwhile, developers should have a peak at the OpenSolaris.org web site and consider joining the device driver community [8]. AcknowledgmentThis paper would not be possible without the help and contributions of kernel expert Seth Goldberg, who works at Sun Microsystems. This document is based on work that Seth coauthored with James and delivered at the Intel Developer Forum in 2007 on Solaris Device Drivers. References
Comments (latest comments first)Discuss and comment on this resource in the BigAdmin Wiki
Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License. |
BigAdmin SubscriptionsBigAdmin Areas
BigAdmin Sun Center
BigAdmin Topics | ||||