CoolThreads - CMT Tuning and Resources

Apache Web Server Configuration

Apache 2.0.58 is available as part of Solaris 10. We recommend using Apache 2.2.4 available as part of Cool Stack 1.2. Apache in Cool Stack is pre-configured for optimum performance on Solaris.

If using SSL, please add the following line to the apache conf/extra/httpd-ssl.conf file to use the hardware crypto acceleration in UltraSPARC T1 & T2 processors:

SSLCryptoDevice pkcs11

Lighttpd

Lighttpd Configuration Notes

A Solaris package based install of Lighttpd is available as part of Cool Stack version 1.2. Cool Stack is a collection of some of the most commonly used open source applications optimized for the Sun Solaris Operating Environment. You can download components of the Cool Stack including Lightpd from http://cooltools.sunsource.net/coolstack.

Building Lighttpd

For instructions on how to build Lighttpd on Solaris please see the build instructions in the CSKlighttpdSrc package of Cool Stack 1.2. You can also browse the Lighttpd sources and build files used for the Cool Stack 1.2 build by going to http://cvs.opensolaris.org/source/ and selecting project 'webstack'.

Configuring Lighttpd

Lighttpd is configured through a single configuration file which is specified with the -f switch when starting Lighttpd. After creating or modifying the configuration file it is recommended that you run.

lighttpd -t -f <config file>

This will tell you if the syntax of the file is correct but will not verify values within quotes.

The following settings are recommended:

server.max-fds = 10000
server.max-worker = 16
server.event-handler = "solaris-devpoll"
server.network-backend = "writev"
server.max-keep-alive-requests = 100000
server.max-keep-alive-idle = 30

16 workers should be adequate for most workloads although 32 workers can be used with minimal additional overhead even if the additional 16 workers are not used. The keep alive settings allow for 100000 open connections and each will time out after 30 seconds of idle time, for loads with very high Think Times or for workloads where request processing is in excess of 30 seconds this can be increased as required.

Nagle's Algorithm

Nagle's algorithm is used to control congestion on Ethernet networks by reducing the amount of small packets that are exchanged between a server and a client. It also has the potential side effect of increasing network latency for those small packets. Web Servers generally disable Nagle's Algorithm at the socket call level, the version of Lighttpd in CoolStack v1.2 has been patched to disable Nagle's Algorithm. Source distributions of Lighttp available from the Lighttpd Community Web Site will not have this patch and the only way to disable Nagle's Algorithm is at the OS level. To disable Nagle's Algorithm on a System running the Solaris Operating Environment, first look at the current value for the TCP parameter: tcp_naglim_def by doing the following:

ndd -get /dev/tcp tcp_naglim_def

If the value is anything other than 1 (defaults to 4095) issue the following command (requires the appropriate access rights):

ndd -set /dev/tcp tcp_naglim_def 1

Note that disabling Nagle's Algorithm is something that you should try if Lighttpd is sending small packets back to its clients, this could be because it's serving either small static files or serving small amounts of dynamic data (as you would see with AJAX applications). If disabling Nagle's Algorithm provides no obvious benefit for your applications re-enable it as it does have positive benefits for some workloads.

More help with Lighttpd

• Lighttpd Performance Tuning
• Lighttpd Configuration options
• Lighttpd as a Service on Solaris

Sun Java System Web Server

Sun Java System Web Server Configuration Notes

SJSW can be used in several ways, like a Servlet/JSP engine and or a Static File serving server and or running traditional NSAPI plug ins.

Unless a lot of caching or huge Java-Heap is needed, the 32bit web server is good for most generic cases.

Given below are some generic tunings applicable to a server capable of serving all of the above for about 8000 connections.

magnus.conf
---------------------
ListenQ: 8192
ConnQueueSize: 8192
RqThrottle: 128
ThreadIncrement: 128
UseNativePoll: 1
KeepAliveTimeout: 30
MaxKeepAliveConnections: 8192
KeepAliveThreads: 2
KeepAliveQueryMeanTime: 50

Init fn="cache-init" disable="true"
Init fn="pool-init" block-size="65536"

nsfc.conf
---------------------
FileCacheEnable=on
CacheFileContent=on
TransmitFile=off
MaxAge=3600
MediumFileSizeLimit=1000001
MediumFileSpace=1
SmallFileSizeLimit=500000
SmallFileSpace=1000000000
MaxFiles=16384
MaxOpenFiles=16384

server.xml
---------------------
Make sure to use the following JVM parameters

<JVMOPTIONS>-server</JVMOPTIONS>
<JVMOPTIONS>-Xbatch</JVMOPTIONS>
<JVMOPTIONS>-Xloggc:/tmp/gc.log</JVMOPTIONS>
<JVMOPTIONS>-Xmx1024m</JVMOPTIONS>
<JVMOPTIONS>-Xms1024m</JVMOPTIONS>
<JVMOPTIONS>-XX:ParallelGCThreads=4</JVMOPTIONS>
<JVMOPTIONS>-XX:+DisableExplicitGC</JVMOPTIONS>
<JVMOPTIONS>-XX:-BindGCTaskThreadsToCPUs</JVMOPTIONS>

Replace
LIBMTMALLOC=/usr/lib/libmtmalloc.so
with
LIBMTMALLOC=/usr/lib/libumem.so

Tomcat Web Server

Tomcat Configuration Notes

Edit the file catalina.sh and modify for optimal jvm tunable parameters.

Please refer to "General Java Tuning" for generic jvm tuning for UltraSPARC T1.

Java and Options that should make a difference

• Make sure that Tomcat uses JDK 1.5.0_06
• Make sure to turn on the gc-log -Xloggc:/tmp/gc.log
• Make sure these jvm parameters are present
  • XX:-BindGCTaskThreadsToCPUs
  • XX:ParallelGCThreads=4 (often 2 is good)
• For a reasonable load with about 300 connections, set the Connector parameters for the port from where the req. comes to something similar to this.

  <Connector port="8080" maxHttpHeaderSize="8192"
  maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
  enableLookups="false" redirectPort="8443" acceptCount="100"
  connectionTimeout="20000" disableUploadTimeout="true" />

The rest of the server.xml does not matter that much.

During runtime, watch the file /tmp/gc.log for "Full" GC activity. If there are frequent "Full" GC happening, you would need to tune the Java-Max-Heap-Size Parameter - -Xmx - A good starting value is 1024m

Make sure that the -Xms parameter is set to the same value as -Xmx

More help with Tomcat

• Deployment Docs
• Context Config
• Config Hints
• Optimization Hints