Print-friendly Version

Consolidation Demo Using Solaris Containers

Angel Camacho, September, 2004

Equipment (Minimum)

• System based on UltraSPARC or x86 processor, with one CPU (two or more preferred)
• 256 Mbyte of RAM (1 GB preferred)
• 20-Gbyte hard disks (two or more hard disks preferred)
• Solaris 10 Operating System build 63 (Solaris Express 8/04)
• MySQL 3.23 (other RDMBS may be substituted)
• Apache Web Server 1.3 (or Sun Java System Web Server)

Objective

The goal is to demonstrate the capabilities of the Solaris 10 Operating System, using the Solaris Zones feature and Solaris Containers technology in an everyday situation, to facilitate and encourage customer adoption.

Prerequisites

Knowledge of system administration using the Solaris OS or another UNIX-like OS is required to set up the environment (the steps will be provided). Also, an understanding of RDBMS and web server technologies is recommended.

Description

Two zones are configured to be used as containers for an RDBMS and a web server. Some parameters are modified for each zone, to control the CPU and other resources in use, according to each application.

The demo is carried out simulating the load from the web server that is accessing information contained in the database. These two applications were selected because they solve a real business problem, but their different natures make them unsuited to share resources inside a traditional server or partition.

Steps

Preparations: Before installing the OS, prepare three IP addresses and CDs with Solaris 10 build 63, MySQL, and the Apache web server.

Note: You can get the Solaris 10 OS from the web site of the Sun Software Express program, and the other software from SunFreeware. If you need to simulate the load on the applications to check the boundaries of the containers, use the "CPU spinner" as described in Step 8 of the following Cookbook section.

1. Install the Solaris OS.
2. Configure the system after installation.
3. Create the structure to hold the zones.
4. Create and install two new zones.
5. Update the resources for each zone including the global zone.
6. Install and configure the RDBMS.
7. Install and configure the web server.
8. Create workloads.
9. Monitor performance and manage resources.

Cookbook

Preparations: You will need at least three different IP addresses, not DHCP. For this demo you will not need network equipment. Everything will be contained in the same system, unless you need network access to get the software from an NFS server or a server running Solaris JumpStart software.

In this case, my machine is a Sun Blade 100 workstation, with 2 Gbyte RAM and two 20-Gbyte hard disks.

IP addresses must be from the same class and netmask, and within the same subnetwork:

• IP for server: xx.xx.xx.10
• IP for zone #1: xx.xx.xx.11
• IP for zone #2: xx.xx.xx.12

1. Install the Solaris 10 OS. If you have one hard disk, make three partitions: the first for swap, the second for Solaris, and the third to be used by the zones. Allocate between 8 and 10 Gbyte for each zone as a separate partition, as partitions will hold the local files for each new zone, and some extra software. In this example, because we are not loading extra applications, the partition size might be as small as 3 Gbyte for each zone, or 9 Gbyte of disk space.

If you have two hard disks, use one to hold the Solaris OS and swap, and leave the other one for the zones. For this example, one disk will hold all the zone file systems. Just be aware that in doing so, if we lose one partition, we lose all the zones. A better approach would be to create a partition for each zone to be used. This partition should be large enough to hold the root file system, plus the applications to run. You can find more information in the documentation for Solaris Containers.

• Server name: servidor
• Primary IP address: xx.xx.xx.10
• Naming service: None (files under /etc to be used)
• Root file system: /dev/dsk/c0t0d0s0

/etc/hosts
     127.0.0.1		localhost
     xx.xx.xx.10	servidor
     xx.xx.xx.11	myweb
     xx.xx.xx.12	mydb

3. Assign the third partition to /zones as the mount point, update the vfstab file and create two subdirectories called 1 (for myweb) and 2 (for mydb). This will keep things under a simple directory structure, but any mountpoint or name will work; just avoid using long directory structures.

Add the following line (as needed) to /etc/vfstab:

/dev/dsk/c0t2d0s2	/dev/rdsk/c0t2d0s2	/zones	ufs	2	yes	-

Mount the /zones file system and create the two subdirectories, and for security reasons change access permissions to 700:

servidor# mkdir /zones
servidor# mount /zones
servidor# mkdir /zones/1	/zones/2
servidor# chmod 700 /zones/1	/zones/2

4. Create and install the zones called myweb and mydb, and assign them the root file system /zones/1 or /zones/2 accordingly, using the zonecfg, zoneadm, and zlogin commands.

servidor# zonecfg -z myweb
myweb: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:myweb> create
zonecfg:myweb> set zonepath=/zones/1
zonecfg:myweb> add net          
zonecfg:myweb:net> set address=xx.xx.xx.11
zonecfg:myweb:net> set physical=eri0
zonecfg:myweb:net> end
zonecfg:myweb> set autoboot=true
zonecfg:myweb> info
zonepath: /zones/1
autoboot: true
pool: 
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
net:
        address: xx.xx.xx.11
        physical: eri0
zonecfg:myweb> commit
zonecfg:myweb> exit
servidor# zoneadm -z myweb install
Preparing to install zone <myweb>.
Creating list of files to copy from the global zone.
Copying <2369> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <782> packages on the zone.
Initialized <782> packages on zone.
Successfully initialized zone <myweb>.
servidor# zoneadm list -cv
  ID NAME             STATUS         PATH                          
   0 global           running        /                             
   - myweb            installed      /zones/1                     
servidor#

Do the same for the zone called mydb, but use xx.xx.xx.12 instead, and verify.

servidor# zoneadm list -cv
  ID NAME             STATUS         PATH                          
   0 global           running        /                             
   - myweb            installed      /zones/1                     
   - mydb             installed      /zones/2                     
servidor#

To finalize the server configuration, boot the zone and log in using the console.

servidor# zoneadm -z myweb boot
servidor# zoneadm -z mydb boot
servidor# zoneadm list -cv
  ID NAME             STATUS         PATH                          
   0 global           running        /                             
   1 myweb            running        /zones/1                     
   2 mydb             running        /zones/2                     
servidor# zlogin -C myweb

(Note: similar messages to those presented here should appear at the terminal.)

[Connected to zone 'myweb' console] (hit enter once)
...
... answer configuration questions to fit your environment
...
>System identification is completed.

rebooting system due to change(s) in /etc/default/init

[NOTICE: zone rebooting]


SunOS Release 5.10 Version s10_54 64-bit
Copyright 1983-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: myweb
The system is coming up.  Please wait.
starting rpc services: rpcbind done.
syslog service starting.
...
...
...
The system is ready.

myweb console login: root
Password: ********
Apr 21 10:25:08 myweb login: ROOT LOGIN /dev/console
Sun Microsystems Inc.   SunOS 5.10      s10_54  May 2004
# hostname
myweb
# exit

myweb console login: ~. (~. to exit console session)
[Connection to zone 'myweb' console closed]
servidor# 

Same for mydb.

servidor# zlogin -C mydb
...
...
...
servidor# zoneadm list -cv
  ID NAME             STATUS         PATH                          
   0 global           running        /                             
   3 myweb            running        /zones/1                     
   4 mydb             running        /zones/2                     
servidor# 

Done!

The system now has three zones: one global zone, and two extra zones, myweb and mydb. The ID number of myweb and mydb changed because of the reboot of the zones.

Log in to the zones (using zlogin), to customize variables and defaults for each zone so that you can log in as root from any terminal, and update the prompt, and so on.

Note that each zone has its own resources; look at the end of the output.

servidor# prctl -i process $$
1659:   -sh
...
. more output
...

zone.cpu-shares               [ no-basic no-local-action ]
                            1 privileged none
                   65535 system     none           [ max ]
servidor#

This means that the process $$, which happens to be in the global zone (0), has one share (the minimum for a zone). Use zlogin to log in to the other zones (myweb or mydb), and run the same command, and they bring the same result. Up to this moment, we have three containers that share all the same resources at the same time, with no limits on each other, in particular on CPU time allocation.

5. Use prctl to allocate cpu-shares to each of the three containers (zones). In this case, for simplicity, use a number that adds up to 100: Assign 45 to mydb, 30 to myweb, and 25 to the global zone. Then in the same way, a new "subdirectory" structure will be added as /usr/local, where all the local installations will be made.

First create the directory structure that will hold the /usr/local directory and update the number of shares for myweb:

servidor# mkdir /zones/1/local
servidor# chmod 700 /zones/1/local
servidor# zonecfg -z myweb
zonecfg:myweb> info
zonepath: /zones/1
autoboot: true
pool: 
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
net:
        address: xx.xx.xx.11
        physical: eri0
zonecfg:myweb> add rctal
zonecfg:myweb:rctl> set name=zone.cpu-shares
zonecfg:myweb:rctl> set value=(priv=privileged,limit=30,action=none)
zonecfg:myweb:rctl> end
zonecfg:myweb> add fs
zonecfg:myweb:fs> set dir=/usr/local
zonecfg:myweb:fs> set special=/zones/1/local
zonecfg:myweb:fs> set type=lofs
zonecfg:myweb:fs> add options [rw,nodevices]
zonecfg:myweb:fs> end
zonecfg:myweb> info
zonepath: /zones/1
autoboot: true
pool: 
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
fs:
        dir: /usr/local
        special: /zones/1/local
        type: lofs
        options: [rw,nodevices]
net:
        address: xx.xx.xx.11
        physical: eri0
rctl:
        name: zone.cpu-shares
        value: (priv=privileged,limit=30,action=none)
zonecfg:myweb> verify
zonecfg:myweb> commit
zonecfg:myweb> exit
servidor# zoneadm -z myweb reboot
servidor# prctl -i zone (myweb new zone ID)
3064:   /usr/lib....
...
zone.cpu-shares               [ no-basic no-local-action ]
                           30 privileged deny           
                   65535 system     deny           [ max ]
servidor#

Here is an alternative method to obtain information regarding the number of shares the zone can use:

servidor# zlogin myweb
[Connected to zone 'myweb' pts/7]
Last login: Tue Apr 27 16:40:31 on pts/7
Sun Microsystems Inc.   SunOS 5.10      s10_54  May 2004
myweb# prctl -i process $$
3080:   -sh
...
zone.cpu-shares               [ no-basic no-local-action ]
                           30 privileged deny           
                   65535 system     deny           [ max ]
myweb# df -k
Filesystem   kbytes    used   avail capacity  Mounted on
/            19239673  227300 18819977     2%    /
/dev         19239673  227300 18819977     2%    /dev
/lib         12641267 4660252 7854603    38%    /lib
/platform    12641267 4660252 7854603    38%    /platform
/sbin        12641267 4660252 7854603    38%    /sbin
/usr         12641267 4660252 7854603    38%    /usr
/usr/local   19239673  227300 18819977     2%    /usr/local
proc                 0       0       0     0%    /proc
mnttab               0       0       0     0%    /etc/mnttab
fd                   0       0       0     0%    /dev/fd
swap           3655608      40 3655568     1%    /var/run
swap           3655568       0 3655568     0%    /tmp
myweb#

Note that 30 zone.cpu-shares are allocated for this zone, which only privileged users can change. In case the zone requires more shares, they will be denied. On the other output, verify /usr/local and play around creating some files and deleting them. Whatever is created inside the zone's /usr/local is not available to other zones nor the global zone, but if you check /zones/1/local, the files are there. Can you delete files from /zones/1/local while in the global zone? Yes, so be careful.

Do the same for the zone called mydb, but allocate 45 zone.cpu-shares instead, and provide /usr/local for it as well.

servidor# mkdir /zones/2/local
servidor# chmod 700 /zones/2/local
servidor# zonecfg -z mydb
zonecfg:mydb> info
zonepath: /zones/2
...
mydb# prctl -i process $$
3275:   -sh
...
zone.cpu-shares               [ no-basic no-local-action ]
                           45 privileged deny           
                   65535 system     deny           [ max ]
mydb#

After these changes, the global zone still has only one share, myweb has 30 shares, and mydb has 45. Each one has its own /usr/local with total isolation for local installs. Remember that at this moment all the shares account for 76 shares.

Changing resources in the global zone involves a different process than the one used for the non-global zones, but it is simpler.

servidor# prctl -i process $$
3363:   -sh
...
zone.cpu-shares               [ no-basic no-local-action ]
                            1 privileged none           
                   65535 system     deny           [ max ]
servidor# prctl -r -n zone.cpu-shares -v 25 -i process $$
servidor# prctl -i process $$
3367:   -sh
...
zone.cpu-shares               [ no-basic no-local-action ]
                           25 privileged none           
                   65535 system     deny           [ max ]
servidor# 

The only problem with this modification is that every time that the system boots, we lose the global zone configuration. We need to add this change into the boot scripts, in order to get the shares we need in the global zone, even after reboots.

servidor# cd /etc/init.d
servidor# vi zonesshares
#!/sbin/sh
#
# Global zone share update
# Angel Camacho
#
/usr/bin/prctl -r -n zone.cpu-shares -v 25 -i process $$
exit 0

servidor# ln /etc/rc3.d/S99zonesshares zonesshares
servidor# ls -i  zonesshares
  176404 zonesshares
servidor# ls -i /etc/rc3.d/S99zonesshares
  176404 /etc/rc3.d/S99zonesshares

Now the system has three containers, well defined in terms of CPU usage, and a zone-owned /usr/local file system where you can make local installs. I encourage you to start using other parameters to manipulate resources like memory, buffers, and lwps, to adjust your containers to be as airtight as you wish.

Here is a list of values that might be interesting to use, but be aware that the wrong configuration may bring down the server performance.

process.max-sem-ops
process.max-address-space
process.max-file-descriptor
process.max-file-size
process.max-cpu-time
task.max-lwps
project.max-shm-memory

6. Install and configure MySQL: You can carry out this installation in two different ways. One is to log in as root into mydb and perform an exclusive installation, into the local /usr/local subdirectory, to achieve total isolation of the application. The other one is to perform a "global" installation, so we can access the application from other zones. The decision is yours.

While in the global zone, download MySQL from Sunfreeware.com. Get all the software MySQL depends on, which may not be present in the Solaris 10 build you are using. MySQL 3.23.53 depends on zlib 1.2, ncurses 5.3, and libgcc 3.3.

Other versions should work, but these are part of the tested configuration. The software might be in the global zone, under a directory called /sw, already uncompressed and ready to add as a package with pkgadd. The process will include mounting /sw into the local zone and extracting the software, then proceeding with the configuration.

servidor# vi /etc/dfs/dfstab
...
share -F nfs -o rw,root=myweb,root=mydb /sw
servidor# chmod 777 /sw
servidor# chmod 666 /sw/*
servidor# /etc/init.d/nfs.server start
servidor# dfshares
RESOURCE                  SERVER ACCESS     TRANSPORT
servidor:/sw        servidor -     -
servidor# zlogin -C mydb
[Connected to zone 'mydb' console]

mydb console login: root
Password:
May 6 17:46:54 mydb login: ROOT LOGIN /dev/console
Last login: Thu May 6 16:13:19 on console
Sun Microsystems Inc.   SunOS 5.10   s10_54  May 2004
mydb# mkdir /sw
mydb# mount -F nfs -o rw servidor:/sw /sw
mydb# ls /sw
libgcc-3.3-sol9-sparc-local
mysql-3.23.53-sol9-sparc-local
ncurses-5.3-sol9-sparc-local
zlib-1.2.1-sol9-sparc-local
mydb# pkgadd -d zlib-1.2.1-sol9-sparc-local
mydb# pkgadd -d ncurses-5.4-sol9-sparc-local
mydb# pkgadd -d libgcc-3.3-sol9-sparc-local
mydb# pkgadd -d mysql-3.23.53-sol9-sparc-local
mydb# cd /usr/local/mysql/bin
mydb# ./safe_mysqld &
mydb# ./mysqladmin -u root password 'new password'

7. Install and configure Apache HTTP Server: The same issues are present with Apache, but in this case, Apache web server is already present inside the Solaris software, so you only need to configure it.

servidor# zlogin -C myweb
servidor# cd /etc/apache
servidor# cp httpd.conf-example httpd.conf
servidor# vi httpd.conf

8. At the global zone, use prctl and prstat to show workloads, open a web browser, and connect to myweb and use the web server.

Note: If you want to stress the CPU/memory of your system, you can get spinners from BigAdmin. The applications shown so far wouldn't be enough to stress the system, as the load that can be generated is minimum even for a single CPU machine with limited resources.

To get the spinners, download the tar file into the global zone, in a working directory, /sw. Uncompress the file and copy and rename the file to its corresponding zone, into /usr/local/bin.

The tar file includes the spinners for both the UltraSPARC and x86 platforms. For both platforms, you will also need MySQL, zlib, ncurses, and libgcc.

The following example is for the x86 architecture. (The equivalent for a system based on the UltraSPARC CPU uses nspins.gz.)

servidor# cd /sw
servidor# gunzip nspinx.gz
servidor# chmod +x nspinx
servidor# cp nspinx /usr/local/bin/clientspinx
servidor# cp nspinx /zones/1/local/webspinx
servidor# cp nspinx /zones/2/local/dbspinx

9. Play around with the zones, change cpu-shares, and shut down zones, to demonstrate the different capabilities of the system, but first change the scheduler from Time Share (TS) to Fair Share Scheduler (FSS).

servidor# dispadmin -d FSS
servidor# reboot

After reboot you can check that the FSS is in use.

servidor# ps -cafe|more

UID   PID PPID CLS PRI    STIME TTY         TIME CMD
root  0   0  SYS  96 09:20:54 ?    0:03 sched
root  1   0  FSS  59 09:20:55 ?    0:00 /etc/init -r
root  2   0  SYS  98 09:20:55 ?    0:00 pageout
root  3   0  SYS  60 09:20:55 ?    1:00 fsflush
root  42  1  FSS  59 09:21:40 ?    0:00 /usr/lib/saf/sac -t 
root  25  1  FSS  59 09:21:29 ?    0:00 /usr/lib/utmpd
daemon 15 1  FSS  10 09:21:26 ?    0:00 /usr/sbin/rpcbind
root  60  1  FSS  29 09:21:13 ?    0:00 /usr/lib/syseventd
daemon 68 1  FSS  29 09:21:14 ?    0:00 /usr/lib/crypto/kcfd
root  71  1  FSS  29 09:21:14 ?    0:02 /usr/lib/picl/picld
root  129 12 FSS   1 11:17:33 pts/2       0:00 /
-More-

For the following demonstration, it's preferable to log in from a different system. The reason is that the Xsun server might interfere with the overall results as it is running directly from the global zone. However, if the demonstration does not require high accuracy on the performance metrics, you can do it all from the global zone. For simplicity the assumption is that the demo will be carried out logging in from a remote system.

Open four terminals, and telnet into the three zones: global, myweb, and mydb. On the fourth one, telnet into the global zone as well and monitor the processes with prstat.

mydb# dbspinx &
myweb# webspinx &
servidor# clientspinx &
servidor# prstat -J
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1720 root 1096K  672K run  2   0 0:00:23  43% dbspinx/1
1721 root 1096K  672K run  1   0 0:00:11  26% webspinx/1
1722 root 1096K  672K run  4   0 0:00:05  14% clientspinx/1
1536 root  71M  19M sleep  47  0 0:00:50 2.8% gnome-term/1
1156 root  45M  41M sleep  25  0 0:04:34 2.0% Xsun/1
1294 root  64M  13M sleep  28  0 0:02:59 0.5% gnome-panel/1
1290 root 58M 8352K sleep  38  0 0:00:36 0.5% metacity/1
1593 root 6184K 5112K cpu0 27  0 0:00:01 0.2% prstat/1
1698 root 4160K 2464K sleep 44 0 0:00:00 0.0% zlogin/1
1274 root 4928K 3072K sleep 59 0 0:00:01 0.0% xscreensaver/1
1690 root 1240K 968K sleep  59 0 0:00:00 0.0% sh/1
1689 root 4160K 2464K sleep 59 0 0:00:00 0.0% zlogin/1
1699 root 1240K  968K sleep 59 0 0:00:00 0.0% sh/1
527 daemon 2400K 984K sleep 5  0 0:00:00 0.0% rpcbind/1
213 root  2560K 752K sleep  25 0 0:00:00 0.0% cron/1

PROJID    NPROC  SIZE   RSS MEMORY      TIME  CPU PROJECT                     
     1       39  610M  199M   9.9%   0:10:03  89% user.root                   
     0       86  258M  130M   6.4%   0:00:05 0.0% system                      

Total: 125 processes, 295 lwps, load averages: 2.28, 0.75, 0.34

And finally, use prctl, dispadmin, and zonecfg to change the variables and modify resource usage. Try to find out what would happen if we start a second dbspinx, or try using some other kind of application.

About the Author

Angel Camacho is Technical Product Manager for Systems Software Product Marketing at Sun Microsystems, Inc.

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.