Print-friendly Version

Impact of Swap Space on System Performance for the Solaris 9 and 10 OS

Victor Feng, February 2008

This article explains how virtual swap space, physical (disk) swap space, and physical memory affect system performance on systems running the Solaris 9 or 10 Operating System. This document demonstrates the negative impact of not having enough physical swap space when the system is low on physical memory.

This article covers the following topics:

• How Swapping Works
• Script to Demonstrate Swapping
• Demonstration
• Conclusion

How Swapping Works

Swapping in the Solaris OS is accomplished through the swapfs pseudo file system. When a system first creates a segment, it reserves virtual swap space. When the system first touches and allocates a page, it allocates virtual swap space for that page. In the Solaris OS, the system reserves enough virtual swap space up front so that swap space allocation assignment is done at the time of request, rather than at the time of need.

With the swapfs file system, the amount of virtual swap space available is the amount of available pageable physical memory (physical memory minus used memory minus reserved memory) plus the amount of physical (disk) swap space available.

In the following demonstration, the system has 16 GB physical memory, 4 GB physical swap space, and about 17.8 GB available virtual swap space.

root@host # vmstat 3
kthr      memory            page               disk        faults
cpu
r b w   swap     free   re  mf  pi po fr de sr s1 s2 s3 -- in   sy
cs    us  sy  id
0 0 0 17767520 15791376 46 130 210  1  1  0 98 15 -0 -0  0 523 2662
1183  0   0   99
0 0 0 17876888 15717136  0   4   0  0  0  0  0  0  0  0  0 336  140
184   0   0  100
0 0 0 17876696 15716944  0   0   0  0  0  0  0  0  0  0  0 334  108
179   0   0  100

If the number of free pages available in the system is below the value of kernel parameter lotsfree, the system will start to scan memory pages. The default value of lotsfree is 1/64 of physical memory.

root@host # echo "lotsfree/E" | mdb -k | sed '/:$/d'
lotsfree:       32064

32064 * 8192 = 250 MB, which is about 1/64 * 16 GB = 256 MB.

Script to Demonstrate Swapping

In this demonstration, we are going to consume memory by creating large files in the /tmp directory, whose files resides in memory instead of disk.

Here's the script we'll use to consume memory:

root@host # more myfilltmp.sh
#!/bin/ksh
# Counter variables
x=0
count=145

# Create a 14.5G load in 100m chunks in /tmp
#
echo "\a\nPress $(tput smso)Enter$(tput rmso) to add 14.5G of
  space to tmp:\c"
read USERINPUT
until
    [ $x -ge $count ]
  do
    mkfile 100m /tmp/file.$x
    let x=x+1
  done
echo "Created 14.5G of Tempfiles in /tmp"

# Create a 2G load in 100m chunks in /tmp
#
x=0
count=20
echo "\a\nPress $(tput smso)Enter$(tput rmso) to add 2G of
  space to tmp:\c"
read USERINPUT
until
    [ $x -ge $count ]
  do
    mkfile 100m /tmp/file_swap.$x
    let x=x+1
  done
echo "Created 2G of Tempfiles in /tmp"

echo ""
echo "\a\nPress $(tput smso)Enter$(tput rmso) to clean tmp:\c
"
read USERINPUT
rm /tmp/file*.*

You need to modify the count variable based on your own available physical memory and available virtual swap space.

Demonstration

In the following stages, we compare the time it takes to run the nm command.

Stage 1

In terminal 2, we start to consume memory:

root@host # ./myfilltmp.sh

Press Enter to add 14.5G of space to tmp:
Created 14.5G of Tempfiles in /tmp

root@host # vmstat 3
kthr      memory            page               disk        faults
cpu
r b w   swap     free   re  mf  pi po fr de sr s1 s2 s3 -- in   sy
cs    us  sy  id
0 0 0 3018824   906752   0  0   0  0  0  0  0  0  0  0  0  348  111
182   0    0 100

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m0.803s
user    0m0.732s
sys     0m0.166s

After filling up /tmp with 14.5 GB, the available physical memory is about 906 MB, and available virtual swap space is 3 GB. Command nm takes 0.803 seconds to finish.

Stage 2

In this stage, we manually fill up /tmp until free memory is close to the lotsfree value, which is 250 MB in this case.

root@host # mkfile 100m /tmp/file.149
root@host # mkfile 100m /tmp/file.150

root@host # vmstat 3
kthr      memory            page               disk        faults
cpu
r b w   swap     free   re  mf  pi po fr de sr s1 s2 s3 -- in   sy
cs    us  sy  id
0 0 0 2609144   497016   0  0   0  0  0  0  0  0  0  0  0  350  116
205    0   0 100
0 0 0 2506736   394608   2 25   0  0  0  0  0  0  0  0  0  369 4426
192    0   1  99
0 0 0 2404056   291816   0  0   0  0  0  0  0  0  0  0  0  433   93
451    0   0 100

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m0.799s
user    0m0.736s
sys     0m0.162s

Available physical memory (291 MB) is above lotsfree (250 MB), available virtual swap space is 2.4 GB, and the system performs well on the nm command.

Stage 3

root@host # mkfile 100m /tmp/file.151

root@host # vmstat 3
kthr      memory            page               disk
faults cpu
r b w   swap     free  re  mf  pi   po    fr de    sr  s1 s2 s3 --
in    sy  cs us  sy id
0 0 0 2404112  291880   0   0  0     0     0  0     0   0  0  0  0
426   95 453  0  0 100
0 0 0 2303056  208992   2  25  0     0     8  0 23924   0  0  0  0
457 4419 561  0  2  98
0 0 0 2303816  199112   0   0  0 16360 16360  0 38852   1  0  0  0
598   98 563  0  1  99
0 0 0 2302088  328320  16 104  0 43894 43894  0   571 363  0  0  0
983  567 852  1  1  99
0 0 0 2303768  369504   0   0  0   340   340  0     0   3  0  0  0
445  103 474  0  0 100
0 0 0 2303768  369840   0   0  0     0     0  0     0   0  0  0  0
425  109 445  0  0 100

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m0.775s
user    0m0.744s
sys     0m0.169s

Available physical memory (208 MB) is below lotsfree (250 MB), and the scan rate (sr) is 23924 pages/second. (The system starts to scan with 100 pages/second, and the scanning speed increases linearly when free memory continues to drop.) Available virtual swap space is 2.3 GB. The system performs well on the nm command, although the system is short on memory.

Stage 4

Let's go back to terminal 2 and press Enter to fill up /tmp so available swap space decreases.

Press Enter to add 2G of space to tmp:

root@host # vmstat 3
kthr      memory          page                  disk
faults cpu
r b w   swap    free  re  mf pi    po    fr de  sr   s1 s2 s3 --
in      sy  cs  us sy  id
0 0 0 2235200 328544   6  65  0 11519 11519 0 9629    0  0  0  0
584   10395 597 0   2  98
0 0 0 1963000 144392   2  21  0 58588 58591 0 69377 473  0  0  0
1301  6884 1331 0   3  97
0 0 0 1894720 245784   0  12  3 59460 59462 0 12282 478  0  0  0
 934    98  572 0   1  99
0 0 0 1894720 423280   0   0  0 60200 60200 0     0 475  0  0  0
 830   111  275 0   1  99
0 0 0 1894720 603168   0   0  0 59449 59449 0     0 469  0  0  0
 813   106  198 0   0 100
0 0 0 1894720 781776   0   0  0 59562 59562 0     0 473  0  0  0
 810    89  183 0   0 100
0 0 0 1894720 958856   0   0  0 57842 57842 0     0 456  0  0  0
 810    92  175 0   0 100
0 0 0 1894720 1134160  0   0  0 58941 58941 0     0 475  0  0  0
 836   111  173 0 0 100
0 0 0 1894720 1310576  0   0  0 58068 58068 0     0 462  0  0  0
 823    89  177 0 0 100
0 0 0 1894720 1486000  1   3  3 59108 59108 0     0 474  0  0  0
 843   109  188 0 0 100
0 0 0 1894616 1661640  0   6  5 48646 48646 0     0 464  0  0  0
 745  3664  169 0 1  99
0 0 0 1587368 1483880 10 103  0     0     0 0     0   0  0  0  0
 374 19104  186 0 3  97
0 0 0 1280288 1144736  5  52 11     0     0 0     0   1  0  0  0
 378  3210  206 0 1  99
0 0 0 1280360 1126824  0   0  0     0     0 0     0   0  0  0  0
 357   111  174 0 0 100
0 0 0 256208 318656    0   0  0     0     0 0     0   0  0  0  0
 419   107  433 0 0 100

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m0.837s
user    0m0.736s
sys     0m0.168s

Available virtual swap space is 256 MB, available physical memory is 318 MB, and the system performs well.

Stage 5

root@host # mkfile 100m /tmp/file_swap.20

root@host # vmstat 3
kthr      memory          page                disk          faults
cpu
r b w   swap     free  re mf pi   po   fr de sr s1 s2 s3 -- in   sy
cs   us  sy  id
0 0 0 153880   259240   0  0  0 3731 3731 0   0 29 0 0   0  454   92
468   0   0 100
0 0 0 153880 260472     0  0  0  298  298 0   0  3 0 0   0  442  141
485   0   0 100

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m0.763s
user    0m0.736s
sys     0m0.166s

Available virtual swap space is 153 MB, available physical memory is 260 MB, and the system performs well.

Stage 6

root@host # mkfile 100m /tmp/file_swap.21

root@host # vmstat 3
kthr      memory        page                  disk           faults
cpu
r b w   swap  free  re  mf pi    po    fr de  sr  s1 s2 s3 --  in  sy
cs  us sy  id
0 0 0 153808 270832  2  25  0 17927 17927 0 4080  50 0  0  0  587  134
591 0   1  99
0 0 0  85464 284696  0   0  0 13998 13998 0 3082 110 0  0  0  555 4378
503 0   1  99
0 0 0  51448 275352  0   0  0 25417 25417 0 7140 200 0  0  0  614  120
509 0   1  99

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m48.465s
user    0m0.738s
sys     0m0.176s

Available virtual swap space is 51 MB, available physical memory is 275 MB, and the system performs very poorly.

Stage 7

One minute later, the system reserves free memory up to 627 MB.

root@host # vmstat 3
kthr      memory        page              disk          faults
cpu
r b w   swap   free re mf pi po  fr de  sr s1 s2 s3 --  in   sy
cs  us sy  id
0 0 0  51568 627064  0  0  0  0   0  0  0   0  0  0  0  419  89
455 0   0  100
0 0 0  51568 627064  0  0  0  0   0  0  0   0  0  0  0  432 117
540 0   0  100

root@host # time nm /dev/ksyms | wc -l
   35039

real    0m47.474s
user    0m0.744s
sys     0m0.171s

With available swap space at 51 MB (free memory is 627 MB), the system performs very poorly, and the system does not scan any more.

root@host # vmstat 3
kthr      memory       page            disk           faults
cpu
r b w   swap  free re mf pi po fr de  sr s1 s2 s3 --   in   sy
cs  us sy  id
0 0 0  50968 626680 3 29  0  0  0  0  0  0  0  0   0  406  241
518  0  0  99
0 0 0  49424 625632 1 10  0  0  0  0  0  0  0   0  0  410   96
459  0  0 100

Conclusion

As demonstrated, when the system is short on physical memory, system performance is affected because of memory page scanning. But when the system is short on virtual swap space, this shortage affects system performance much more seriously than physical memory shortage. Usually, the size of physical swap space should be 0.5 to 1.5 times the size of physical memory.

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.