Print-friendly Version

Sun Blogs: A Sun Java System Web Server 7.0 Reference Deployment

Meena Vyas, Murthy Chintalapati, and Allen Gilliland, October 2008

Introduction

Sun Blogs, available at http://blogs.sun.com, is a blogging service dedicated to Sun employees worldwide, hosted and managed by sun.com engineering. Blogs.sun.com is powered by Sun Java System Web Server 7.0 Update 2 (hereafter, Web Server) running on a pair of Sun Fire T2000 servers with UltraSPARC T2 processors. The site also uses a custom version of the open source blogging server Apache Roller Weblogger Version 4.0.0.12, MySQL database server, and Memcached. The service is integrated with Sun's secure single sign-on identity system backed by a worldwide employee LDAP directory for authentication and authorization.

This article presents an overview of the Sun Blogs site's usage statistics, systems architecture, and Web Server configuration. It also provides information about the system administrator's experience managing one of the busiest externally facing Web 2.0 site deployments at Sun.

This article covers the following topics:

• Sun Blogs Statistics
• System Architecture
• Sun Blogs Configuration
• Sun Blogs Site Administrator's Perspective
• Conclusion
• About the Authors
• For More Information
• Appendix

Sun Blogs Statistics

Overall, as of September 2008, Sun Blogs hosted the following:

The graph in Figure 1 and the table that follows show monthly raw access log statistics, analyzed by The Webalizer. For example, in August 2008, the Sun Blogs site (blogs.sun.com) served 125 million hits, averaging roughly 4 million per day.

Usage Statistics - blogs.sun.com
Summary Period: Last 12 Months
Generated 25-Sep-2008 04:05 PDT

Figure 1: Graph

System Architecture

Figure 2 shows the system architecture behind the Sun Blogs service (a mature instantiation of the Apache Roller Weblogger code). It is deployed on a pair of Sun Fire T2000 servers with 32 Gbytes of RAM, negligible local disk, and a shared file system (NFS from Sun Cluster), running Web Server 7.0 Update 2. The service is behind a hardware load balancer splitting traffic evenly. A ton of network gear on the front, from Juniper Network Netscreen firewalls to Nortel Alteon load balancers, does a lot of the heavy network work; the gear is fully redundant all the way through from the provider to the actual machines. The Sun Blogs site plans to switch soon to Sun Fire X4450 servers to take advantage of higher single-thread performance, because the Sun Fire T2000 servers are fairly under-utilized.

The database that powers all this is MySQL on Sun Fire V40z servers with 16 Gbytes of RAM and lots of internal disk. Memcached is used for optimal performance. No external storage is used. The database servers are paired (there are two of them) running in hot/standby mode.

Figure 2: System Architecture

Sun Blogs Configuration

The server.xml configuration file contains information related to server runtime behavior, such as number of threads, number of concurrent connections, and Java Virtual Machine (JVM) tuning for the built-in web container to serve servlet and JSP applications. This section shows the configuration changes that the Sun Blogs team needed to make to get the desired site performance.

Here are relevant sections of the web server server.xml configuration file, with inline comments. For the sake of brevity, only selected attributes are shown.

<server>

  <log>
    <log-file>../logs/errors</log-file>
    <archive-suffix>.m%d</archive-suffix>
    <log-level>info</log-level>
  </log>

  <http>
    <output-buffer-size>40960</output-buffer-size>
  </http>

  <!-- Typical worker thread pool with higher stack size
  not uncommon for Java web applications with deeper
  invocation stack depths -->
  <thread-pool>
    <max-threads>250</max-threads>
    <stack-size>256000</stack-size>
  </thread-pool>

  <!-- Java VM settings  -->
  <jvm>
    <java-home>.../jdk</java-home>
    <server-class-path>...</server-class-path>
    <debug>false</debug>
    <debug-jvm-options>-Xdebug -Xrunjdwp:transport=dt_socket,
    server=y,suspend=n,address=7896</debug-jvm-options>
    <jvm-options>-Djava.security.auth.login.config=login.conf
    </jvm-options>

    <!-- Increased ms and mx; default was -Xms128m -Xmx256m -->
    <jvm-options>-Xms3g -Xmx3g -XX:NewSize=1g</jvm-options>
    <jvm-options>-server -Xsqnopause</jvm-options>
    <jvm-options>-XX:+UseParallelGC -XX:+UseParallelOldGC
    -XX:ParallelGCThreads=20</jvm-options>

    <!-- Added java keystore locations -->
    <jvm-options>-Dcom.sun.management.jmxremote
    -Djavax.net.ssl.keyStore=...keystore
    -Djavax.net.ssl.keyStorePassword=...</jvm-options>
  </jvm>

 <!-- MySQL data source configured as a JDBC resource -->
  <jdbc-resource>
    <jndi-name>jdbc/rollerdb</jndi-name>
    <datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlDataSource
    </datasource-class>
    <property>
      <name>Password</name>
      <value>...</value>
      <description/>
    </property>
    <property>
      <name>User</name>
      <value>...</value>
      <description/>
    </property>
    <property>
      <name>URL</name>
      <value>jdbc:mysql://somedburl:12345/database?relaxAutoCommit=
      true&useUnicode=true&characterEncoding=utf-8</value>
      <description/>
    </property>
    <description>roller datasource</description>
    <min-connections>15</min-connections>
    <max-connections>250</max-connections>
    <idle-timeout>180</idle-timeout>
    <connection-validation>meta-data</connection-validation>
  </jdbc-resource>

  <mail-resource>
    <jndi-name>mail/Session</jndi-name>
    <description>roller mail session</description>
    <property>
      <name>mail.smtp.host</name>
      <value>mailhost</value>
      <description>Mail server hostname</description>
    </property>
  </mail-resource>

<!-- blogs web app with document root '/' runs virtual server -->
  <virtual-server>
    <name>blogs</name>
    <host>blogs</host>
    <http-listener-name>http-listener-1</http-listener-name>
    <web-app>
      <uri>/</uri>
      <path>../web-app/blogs/_default</path>
      <description/>
    </web-app>
  </virtual-server>

<!-- Logs are rotated once a day. -->
  <event>
    <time>
      <time-of-day>00:00</time-of-day>
    </time>
    <rotate-access-log>true</rotate-access-log>
  </event>

  <event>
    <time>
      <time-of-day>00:00</time-of-day>
    </time>
    <rotate-log>true</rotate-log>
  </event>
</server>
	

Sun Blogs Site Administrator's Perspective

Here's some information from the site administrator.

"As far as my experience with Sun Java System Web Server 7, I love it. We used Tomcat for the first couple years of the site when only Sun Java System Web Server 6.1 was available, and quite frankly, Sun Java System Web Server 6.1 lacked some features and ease of use. At that time, we preferred Tomcat, even though it lacked a lot of cool features.

Shortly after Sun Java System Web Server 7.0 came out, we switched and since then I've been a big fan. Web Server 7 offers way more features than Tomcat does, and has been very stable and nice to work with. Also, we are using Web Server because we like some of the benefits the native Web Server 7 code gives us.

One of the interesting problems we solved using Web Server 7 was an issue caused by slow clients that were tying up Web Server threads in j2ee-module. Basically, if a client was slow and the thread got stuck in j2ee-module, it would continue to tie up limited resources, such as database connections, and at one point, this became a real problem.

To resolve the problem, we utilized the ability of the Web Server to set a larger output-buffer-size for each thread so that the response data for a slow client could be effectively buffered by the Web Server and the thread wouldn't be stuck in j2ee-module.

In my experience, this ability is something you don't get from other web servers. I have found that Web Server 7 provides better integration between a native web server and a Java 2 Platform, Enterprise Edition (J2EE) container than other solutions such as Apache Tomcat.

I've also been a big fan of the Web Server perfdump and stats.xml data, which can be retrieved from the Web Server at runtime, even when the site appears to be hanging from the Web. That data has been immensely valuable in debugging Web Server issues, such as the one described previously.

Apart from that, our use of Web Server 7 has been pretty much clear sailing. We've actually been running our current installations for about a year with almost no tweaks necessary. The application just sits there and runs."

Conclusion

Sun Blogs (blogs.sun.com) offers a secure blogging service to thousands of Sun employees worldwide. The site runs the Apache Roller Weblogger application and is powered by Sun Java System Web Server 7.0, MySQL database, and Memcached, running respectively on pairs of Sun Fire T2000 and V40Z servers. The Sun Blogs site serves 4 million hits a day on average, representing one of the busiest Web 2.0 sun.com sites.

About the Authors

Meena Vyas is a software engineer with the Sun Java System Web Server development engineering team. Murthy Chintalapati was an architect and is engineering manager for Sun's Web Tier products. Allen Gilliland is a systems specialist with the sun.com engineering team that manages Sun Blogs.

For More Information

Here are additional resources:

• Sun Forums, such as Sun Java System Web Server forum and the BigAdmin Discussions collection
• Web Tier section of Sun Developer Network (SDN)
• Sun wikis, such as:
  • Sun Java System Web Server wiki, including the Sun Java System Web Server FAQ
  • Sun BluePrints wiki
  • BigAdmin wiki
• Documentation at http://docs.sun.com, such as:
  • Sun Java System Web Server 7.0 Update 3 Documentation Center
  • Sun Java System Web Server 7.0 Update 3 Administrator's Guide
  • Sun Java System Web Server 7.0 Update 3 Performance Tuning, Sizing, and Scaling Guide
• Sun Web and Proxy Servers download page
• Sun training courses at http://www.sun.com/training/, such as
  • Sun Java System Web Infrastructure Suite Overview (WMT-ECR-1508)
  • Sun Java System Web Server 6.1: Administration and Maintenance (ECR-2307)
• Sun Java System Web Server web site
• Related resources on BigAdmin:
  • Using Sun Java System Web Server With Sun Identity Manager and MySQL Software
  • Sun Forums: A Sun Java System Web Server 7.0 Reference Deployment
  • Installing Sun Java System Web Server 7.0 on OpenSolaris Operating System
• Support:
  • Sun resources:
    • Register your Sun gear
    • Services
    • SunSolve Online
  • Community system administration experts
• Events of interest to users of Sun products:
  • Worldwide Developer Events and Sun Tech Days
  • Current Events

Appendix

Here is a sample of output from perfdump:

Sun Java System Web Server 7.0 B12/04/2006 10:15 (SunOS DOMESTIC)

Server started Fri Sep 12 09:09:07 2008
Process ... started Fri Sep 12 09:09:07 2008

ConnectionQueue:
-----------------------------------------
Current/Peak/Limit Queue Length            0/96/1474
Total Connections Queued                   28960884
Average Queue Length (1, 5, 15 minutes)    0.00, 0.00, 0.01
Average Queueing Delay                     0.05 milliseconds

ListenSocket http-listener-1:
------------------------
Address                   http://0.0.0.0:8080
Acceptor Threads          1
Default Virtual Server    blogs

KeepAliveInfo:
--------------------
KeepAliveCount        165/200
KeepAliveHits         13743024
KeepAliveFlushes      0
KeepAliveRefusals     1740047
KeepAliveTimeouts     2509109
KeepAliveTimeout      30 seconds

SessionCreationInfo:
------------------------
Active Sessions           55
Keep-Alive Sessions       0
Total Sessions Created    250/250

CacheInfo:
------------------
enabled             yes
CacheEntries        275/1024
Hit Ratio           8662111/25918743 ( 33.42%)
Maximum Age         30

Native pools:
----------------------------
NativePool:
Idle/Peak/Limit               1/1/128
Work Queue Length/Peak/Limit  0/0/0

DNSCacheInfo:
------------------
enabled             yes
CacheEntries        0/1024
HitRatio            0/0 (  0.00%)

Async DNS disabled

Performance Counters:
------------------------------------------------
                           Average         Total      Percent

Total number of requests:               26058151
Request processing time:    0.7873  20515742.0000

default-bucket (Default bucket)
Number of Requests:                     26058151    (100.00%)
Number of Invocations:                 298097638    (100.00%)
Latency:                    0.0021    55890.8477    (  0.27%)
Function Processing Time:   0.7852  20459852.0000    ( 99.73%)
Total Response Time:        0.7873  20515742.0000    (100.00%)

Sessions:
----------------------------------------------------------------
Process Status Client      Age     VS  Method  URI      Function

28793 response xxx.xx.xxx.xxx 392 blogs GET
/mramcha/resource/P1010825s.JPG
28793 response xxx.xx.xxx.xxx 58  blogs GET
/barton808/resource/MarksWelcome.JPG
28793 response xxx.xx.xxx.xxx 31  blogs GET
/katakai/entry/php_framework_support_in_netbeans
28793 response xxx.xx.xxx.xxx 28  blogs GET
/wizidm/feed/entries/atom
28793 response xxx.xxx.xx.xxx 6   blogs GET /roger/ service-j2ee
28793 request  xxx.xxx.xx.xxx 4
28793 response xxx.xxx.xx.xxx 4   blogs GET /jonathan/
28793 response xx.xxx.xxx.xxx 2   blogs GET
/jimgris/feed/entries/atom
28793 response xx.xxx.xxx.xxx 1   blogs GET
/PotstickerGuru/feed/entries/rss service-j2ee
....
	

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.