BigAdmin Feature Article: Deploying JBoss Application Server on Sun Fire T2000 Servers

This document serves as a starting point for the deployment of the JBoss Application Server (JBoss AS) on Sun Fire T2000 server systems. Since general documents about JBoss AS and the Sun Fire T2000 server already exist, only links to those documents will be provided. The rest of this document provides tuning recommendations specific to JBoss AS on the Sun Fire T2000 server.

JBoss AS Introduction

JBoss AS is a Java 2 Platform, Enterprise Edition (J2EE) certified platform for developing and deploying enterprise Java applications, web applications, and portals. JBoss AS provides the full range of J2EE 1.4 features as well as extended enterprise services, including clustering, caching, and persistence.

Sun Fire T2000 Server Introduction

Here is an excerpt of sun.com's description of the Sun Fire T2000 server:

Record-breaking performance, unequaled energy efficiency, and space-saving compute density
Award-winning 4 or 8 core UltraSPARC T1 processor, supporting 16 or 32 simultaneous threads (UltraSPARC T1 architecture is SPARC V9 compliant)
Support for Solaris 10 and Linux operating systems (software applications written to the Solaris Application Binary Interface can run on new UltraSPARC T1-based systems without modification)

For more information on the Sun Fire T2000 server, including benchmarks and the SwaP (Space, Watts and Performance) metric, see sun.com.

Deployment Tasks

JBoss AS Installation Instructions

One of the strengths offered by JBoss AS is its ease of installation. To install JBoss AS, do the following:

1. Download the latest revision at http://labs.jboss.com/portal/jbossas/download.

JBoss AS is distributed in the form of a .zip file to avoid incompatibilities between various versions of tar.

2. Unzip the distribution to the desired location. Recommended installation directories are /opt or /home/jboss for the Solaris OS.

root@host# unzip jboss-4.0.4.GA.zip

That's it! You should now have a directory called jboss-4.0.4.GA. JBoss AS is ready to run.

JBoss AS Configuration Instructions

JBoss AS stores its configuration under $JBOSS_HOME/server. After unzipping the distribution, you should see three subdirectories there. Each of these directories is a different JBoss AS configuration. Here is a short description of each of the default configurations.

all -- This configuration has all JBoss AS features enabled, including clustering.
default -- This configuration will be suitable for most developers and many single-instance production environments.
minimal -- This is a bare-bones configuration that includes the JBoss microkernel and a few core services. Use this configuration to add features as you need them.

After you have chosen the configuration you will be using, you should probably make a working copy of it where you can make your changes.

# cp -r default myconfig

Now you can use the command $JBOSS_HOME/bin/run.sh -c myconfig to start JBoss AS using your configuration.

Another of the strengths that JBoss AS can offer is the extensive number of possible configurations. Because every feature in JBoss AS is a pluggable module, you can remove or add features as needed. For advanced configuration, see the JBoss Application Server Guide at http://labs.jboss.com/portal/jbossas/docs.

Performance

OS Tuning (/etc/system)

Please make sure the following parameters are set in /etc/system to improve the Sun Fire T2000 Server performance:

set pcie:pcie_aer_ce_mask=0x1
set segmap_percent=60
set smallfile=1572864
set maxpgio=128
set rlim_fd_max=260000
set rlim_fd_cur=260000
set shmsys:shminfo_shmax=0xffffffff
set tune_t_fsflushr = 1
set autoup = 60
set sq_max_size = 10000
set ip:ip_squeue_bind = 0

Note: These tuning settings might not be optimal if the system is shared by other applications.

TCP Tuning

Please run the following commands to tune TCP on client systems:

ndd -set /dev/tcp tcp_smallest_anon_port 2048
ndd -set /dev/tcp tcp_max_buf 8388608
ndd -set /dev/tcp tcp_recv_hiwat 131072

Interrupt Coalescence Tuning on the Sun Fire T2000 Server

Please run the following command in a shell to force the server to use only virtual CPUs 29, 30, and 31 for interrupts. (Note: Type the command all on one line.)

psradm -i 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28

Afterwards, only virtual CPUs 29, 30, and 31 serve the interrupts.

Network Configuration

Please use the following script to tune the network port on the Sun Fire T2000 server:

 #!/bin/bash
 INSTS="0 1 2 3"
 NIC=ipge
 for port in $INSTS ; do
     ndd -set /dev/$NIC instance $port
     ndd -set /dev/$NIC rx_intr_pkts 600
     ndd -set /dev/$NIC rx_intr_time 600
 done

Tuning Considerations for JBoss AS

Because J2EE applications are so diverse, a tuning that works very well for one application might not work so well for another. There are, however, some basic changes you can make that will have benefits in almost any situation.

Java Virtual Machine (JVM) options:

Edit $JBOSS_HOME/bin/run.conf to configure the JVM settings used when launching JBoss AS. Here are some example settings for the Solaris 10 OS, which increase the available memory heap and utilize a higher throughput garbage collector.

JAVA_OPTS="-server -Xms2650m -Xmx2650m -Xss128k -verbosegc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-XX:+DisableExplicitGC -XX:+UseParallelGC
-XX:ParallelGCThreads=8 -XX:LargePageSizeInBytes=256m"

Tomcat options:

Edit $JBOSS_HOME/server/default/deploy/jbossweb-tomcat55.sar/server.xml to configure thread pool size and other Tomcat connector options. You should also disable some of the Tomcat development options, which are enabled in $JBOSS_HOME/server/default/deploy/jbossweb-tomcat55.sar/jboss-service.xml.

JBoss options:

On Sun Fire T2000 Server systems, it's recommended that you run JBoss AS in the FX class using the following command. You will need root privileges to execute priocntl.

/usr/bin/priocntl -e -c FX ./run.sh <jboss options>

Check out http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossASTuningSliming for more tuning and slimming options.

Advanced Deployments

JBoss AS has been tested and certified inside non-global zones in the Solaris OS. Because non-global zones behave almost identically to a normal Solaris system, no JBoss AS configuration changes need to be made to take advantage of Solaris zones. Simply copy the JBoss AS installation to the zone. You can then start JBoss AS from inside the zone as usual.

Scaling Up

As you increase the load on your server, you will likely run into some resource limits. JBoss AS has many resource pools that can be tuned according to your application needs. The Tomcat thread pool was discussed previously, but there are also many internal pools, for Message Driven Beans (MDB), for example. Many of these can be found in the configuration file $JBOSS_HOME/server/default/conf/standardjboss.xml.

If you have saturated the CPU on one server, you can continue to scale up using clustering. Simply use the all configuration and bind to a specific interface that has multicast enabled on each server. JBoss AS will automatically find the other instances on the network! There are some other considerations with clustering, such as load-balancing and session replication. Please see the JBoss High Availability and Clustering Topics web page at http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHA for some related examples.

Staying Up

JBoss AS offers several options for monitoring. Here are the most common.

Java Management Extensions (JMX) -- JBoss AS is built on top of a JMX micro-kernel. JMX provides a great way of monitoring or querying the value of any internal resources. Because JMX is a standard, any JMX tool can be used. Consider using the JBoss Operations Network, which is designed to make monitoring (as well as upgrades and patches) very easy.

Also included is a small tool called twiddle, which can be used to query JBoss AS. You can run this program using the script at $JBOSS_HOME/bin/twiddle.sh.

SNMP -- Another alternative is SNMP, which is already used in many organizations. Using the SNMP adapter, you can be notified of existing JMX events through your current SNMP infrastructure! For configuration of this module, please see the JBoss SNMP Adapter web page.

Network Architecture

Generally, JBoss AS doesn't require any special network configuration. By default, JBoss AS binds to all local interfaces. If you would like to bind to a specific interface, you should use the -b option to run.sh. This option takes a single argument: the IP address or DNS name of the interface to bind to. Additionally, clustering does require multicast to be enabled on the interface that JBoss AS is bound to. Multicast is configured correctly on the Solaris OS by default.