Sun Fire CoolThreads Try and Buy Resource

Sun Fire CoolThreads technology servers, specifically Sun Fire T1000 and Sun Fire T2000 servers, system contains a single Ultrasparc T1 chip or processor. The T1 processor is a unique design of 8 individual processing units called cores sharing one on-chip interconnect. It is somewhat like an 8-way system on a single chip.

Each core supports 4 hardware (HW) threads of execution. These HW threads are scheduled on the core's processing unit in round-robin order. A different Software thread can run on each one of these HW threads. 32 Software threads can run in parallel on a single T1 processor.

        # psrinfo
        0       on-line   since 01/31/2006 21:33:18
        1       on-line   since 01/31/2006 21:33:19
        2       on-line   since 01/31/2006 21:33:19
        3       on-line   since 01/31/2006 21:33:19
        4       on-line   since 01/31/2006 21:33:19
        ...
        29      on-line   since 01/31/2006 21:33:19
        30      on-line   since 01/31/2006 21:33:19
        31      on-line   since 01/31/2006 21:33:19
	

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
    0. c1t0d0 [SUN72G cyl 14087 alt 2 hd 24 sec 424>
       /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0
    1. c1t1d0 [SUN72G cyl 14087 alt 2 hd 24 sec 424>
       /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@1,0
Specify disk (enter its number): 
	

The Sun Fire T1000 server and Sun Fire T2000 server integrated 1Gig network devices are labeled net0, net1, net2, net3 on the back of a physical system.

On a Sun Fire T1000 server specifically, the ports are called bge0, bge1, bge2, bge3. Use ifconfig(1M) to display them

# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 129.148.94.85 netmask ffffff00 broadcast 129.148.94.255
        ether 0:14:4f:2:e1:8a 
	

On the Sun Fire T2000 server, the ports are called ipge0, ipge1, ipge2, ipge3. Use ifconfig(1M) to display them

# ifconfig -a
...
ipge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500
index 2 inet 10.7.195.225 netmask ffffff00 broadcast 10.7.195.255
   ether 0:3:ba:d8:ea:62 
	

The Sun Fire T1000 server has 1 PCI-E slot, supporting 1x, 4x, and 8x width.

The Sun Fire T2000 server has 3 PCI-E slots which are the shorter black slots and 2 PCI-X slots which are the longer white slots. The PCI-X slots support PCI 33MHz, 66MHz and PCI-X cards

The internal disk controller currently occupies one of the PCI-X slots but in future releases this will be integrated on the motherboard.

The PCI-E slot on the T1000 is low profile.

On the Sun Fire T2000 server, generally all cards must be low profile, specifically PCIe cards which are 6.93 x 5.15 cm and PCI/PCIX cards MD2 form factor.

The official version of Solaris shipped with Sun Fire T2000 server is Solaris 10 1/06. This requires some mandatory patches listed in the Sun Fire T2000 Product Notes available here:
http://www.sun.com/products-n-solutions/hardware/docs/Servers/coolthreads/t2000/

Solaris 10 6/06 is also supported. This is currently available for download at http://www.sun.com/software/solaris/ Solaris Next which is available as part of the Solaris Express program is also available: http://www.sun.com/software/solaris/solaris-express/

cat /etc/release
             Solaris 10 3/05 HW2 s10s_hw2wos_05 SPARC
   Copyright 2005 Sun Microsystems, Inc.  All Rights Reserved.
                Use is subject to license terms.
                   Assembled 26 September 2005
	

# uname -a
SunOS ontario-mc70 5.10 Generic_118833-02 sun4v sparc SUNW,Sun-Fire-T200

        Virtual cpus 0,1,2,3  are in core 0
        Virtual cpus 4,5,6,7  are in core 1
        Virtual cpus 8,9,10,11  are in core 2
        Virtual cpus 12,13,14,15  are in core 3
        Virtual cpus 16,17,18,19  are in core 4
        Virtual cpus 20,21,22,23  are in core 5
        Virtual cpus 24,25,26,27  are in core 6
        Virtual cpus 28,29,30,31  are in core 7

 netstat -I  ipge1  -i 1
 input   ipge1       output           input  (Total)    output
 packets errs  packets    errs    packets      errs  packets errs  
 10      0     175         0          390       0        550     0       
 30      0     96          0          421       0        218     0       
  5      0     64          0          331       0        147     0      
 18      0     32          0          309       0        326     0    
	

        set ip:ip_squeue_bind = 0
        set ip:ip_squeue_fanout = 1
        set ipge:ipge_tx_syncq=1
        set ipge:ipge_taskq_disable = 0
        set ipge:ipge_tx_ring_size = 2048
        set ipge:ipge_srv_fifo_depth = 2048
        set ipge:ipge_bcopy_thresh = 512
        set ipge:ipge_dvma_thresh = 1

        # iostat -xcn 5

        <<<< DISCARD >>>>
     cpu
 us sy wt id
  0  0  0 100
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    1.0    0.2   12.9    1.2  0.0  0.0    0.0    7.4   0   0 c1t0d0
    0.8    0.0  105.6    0.0  0.0  0.0    0.0    1.9   0   0 c1t1d0
0.0	0.0    0.0    0.0  0.0  0.0    0.0    0.5   0   0 c0t0d0
0.1	
        <<<< VALID >>>>
     cpu
 us sy wt id
  0  0  0 100
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t0d0
  488.7    0.0 62552.5    0.0  0.0  0.9    0.0    1.9   1  93 c1t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0

# newfs /dev/dsk/c1t0d0s6
newfs: construct a new file system /dev/rdsk/c1t0d0s6: (y/n)? y
Warning: 2496 sector(s) in last cylinder unallocated
/dev/rdsk/c1t0d0s6:     57056832 sectors in 9287 cylinders of 48 tracks, 
128 sectors
   27859.8MB in 581 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
Initializing cylinder groups:
...........
super-block backups for last 10 cylinder groups at:
56139168, 56237600, 56336032, 56434464, 56532896, 56623136, 56721568,
56820000, 56918432, 57016864
	

The Sun Fire T1000 server is already configured with the write cache enabled.

On the Sun Fire T2000 server, one option which is used on Linux systems today by default is turning on the write caches of the internal disks

# format -e
Searching for disks...done


AVAILABLE DISK SELECTIONS:
        0. c1t0d0 [SUN72G cyl 14087 alt 2 hd 24 sec 424>
           /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0

        1. c1t1d0 [SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@1,0
Specify disk (enter its number): 1
selecting c1t1d0
[disk formatted]

        FORMAT MENU:
                disk       - select a disk
                type       - select (define) a disk type
                partition  - select (define) a partition table
                current    - describe the current disk
                format     - format and analyze the disk
                repair     - repair a defective sector
                label      - write label to the disk
                analyze    - surface analysis
                defect     - defect list management
                backup     - search for backup labels
                verify     - read and display labels
                save       - save new disk/partition definitions
                inquiry    - show vendor, product and revision
                scsi       - independent SCSI mode selects
                cache      - enable, disable or query SCSI disk cache
                volname    - set 8-character volume name
                !     - execute , then return
                quit
        format> cache


        CACHE MENU:
                write_cache - display or modify write cache settings
                read_cache  - display or modify read cache settings
                !      - execute , then return
                quit
        cache> write_cache


        WRITE_CACHE MENU:
                display     - display current setting of write cache
                enable      - enable write cache
                disable     - disable write cache
                !      - execute , then return
                quit
        write_cache> display
        Write Cache is disabled
        write_cache> enable
        write_cache> display
        Write Cache is enabled
        write_cache> q  
        cache> q
        format> q

Yes, Jumbo frames are supported on the Sun Fire T2000 server but this is not enabled by default. For instructions to enable Jumbo Frames on the Sun Fire T2000 server, see:

Jumbo frames can be implemented on both the internal Gigabit Ethernet interfaces of the Sun Fire T2000 Server, and those provided by optional PCI-E and PCI-X based cards.

Note that the maximum MTU stated in that URL is 8192 but this has been since reduced to 8170.

On the Sun Fire T1000 server Jumbo frames are not currently supported.

The key to good performance on a Sun Fire T2000 server is throughput. Maximum throughput is achieved when the application keeps many hardware threads busy in parallel.

If an application scales on a Sun Fire v880 server, it should also do well on a Sun Fire T2000 server.

Like the Sun Fire T2000 server, the key to good performance on a Sun Fire T1000 server is throughput. Maximum throughput is achieved when the application keeps many hardware threads busy in parallel. If an application is multi-threaded and performs well on a Sun Fire V240 server, it should also do well on a Sun Fire T1000 server. You may need to run multiple instances of the application if it does not scale well across multiple cores. Solaris Containers provides the mechanism to do this.

See: http://www.sun.com/software/solaris/utilization.jsp

Sun Fire T1000 and T2000 servers are fully SPARC binary compatible for all Sparc architectures that Sun supports, v7, v8 and v9. An application does not need to be recompiled or modified in any way to run on a Sun Fire CoolThreads server. Binaries created by older versions of the Sun Studio compiler will also work well. All the current compiler optimizations also work well.

We recommend using the latest Sun Studio 11 compiler which is available free for download from

http://developers.sun.com/prodtech/cc/index.html

        32-bit  Studio 7,8,9,10 -fast -xO4 -xtarget=generic
        32-bit  Studio 11       -fast -xO4 -xtarget=ultraT1
        64-bit  Studio 7,8,9,10 -fast -xO4 -xtarget=generic64
        64-bit  Studio 11       -fast -xO4 -xtarget=generic64 
	

If your Oracle installation involves the following, please implement the instructions below:

• installing Oracle 10g Release 2 on T1000/T2000
• using Oracle 10g Release 2 JVM on T1000/T2000

Description:

Use of the Oracle JVM involves an uncommon software trap which was not implemented correctly for sun4v platforms (T1000/T2000) and will cause the calling process/thread to enter an infinite loop and appear to be hung. Spinning processes/threads respond normally to UNIX signals and can be killed.

If the Oracle JVM is selected for installation, the JVM will be called and installation will appear to hang at that point.

If Oracle is installed on another SPARC system and that installation is copied to a T1000/T2000, processes will spin when they first attempt to call the JVM.

Workaround:

1. If your application makes no use of the Oracle JVM, when installing Oracle, de-select the option to have the Oracle JVM installed.
2. If your application uses the Oracle JVM, download and install patch 118833-18. To install this patch and update your system follow the process documented here: http://www.sun.com/servers/coolthreads/tnb/updates.jsp

Sun has a range of "Cool Tools" for application development and performance optimisation. Please see:

http://cooltools.sunsource.net/nonav/index.html

        prstat(1M)
        mpstat(1M)
        vmstat(1M)
        iostat(1M)
        cpustat(1M)
        lockstat(1M)
        plockstat(1M)
        ps(1M)	

In the Studio 11 software distribution there is also the Workshop Analyzer which is an excellent tool for evaluation of Application performance. This is described at
http://developers.sun.com/prodtech/cc/analyzer_index.html

The latest version of the JVM also has JVMTI built in that gives a rich profiling environment via the -Xrunhprof option. More information available at
http://java.sun.com/j2se/1.3/docs/guide/jvmpi/jvmpi.html

/usr/sfw/bin/openssl speed rsa1024 rsa2048 -engine pkcs11 -multi 32

[ intermediate output snipped..... ]

                                  sign    verify    sign/s verify/s
                                  rsa 1024 bits   0.0001s   0.0000s
12871.3 
45148.1
                                  rsa 2048 bits   0.0004s   0.0000s 
2299.6 
20425.3

 # mpstat 1
CPU minf mjf xcal  intr ithr  csw icsw migr smtx srw syscl  usr sys  wt idl
  0    0   0  294   329  227  117   60   12 5     0     20   10  90   0   0
  1   11   0    0   141    4   73   41   12 5     0     20   11  89   0   0
  2    0   0    0   140    2   64   37    1 5     0     20   10  90   0   0
  3    0   0    0   130    0   49   32    2 5     0     20   10  90   0   0
  

  PID  UNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
  4448 root      12  44 0.0 0.0 0.0 0.0  43 0.5  2K 460 .1M   0 prstat/1
  4447 root     1.2  11 0.0 0.0 0.0 0.1  14  73  54  65 .2M   0 fbench/27
  4447 root     1.1  10 0.0 0.0 0.0 0.1  15  74  57  52 .2M   0 fbench/29
  4447 root     1.1  10 0.0 0.0 0.1 0.0  15  74  64  53 .2M   0 fbench/19
  4447 root     1.1  10 0.0 0.0 0.0 0.4  14  74  49  55 .2M   0 fbench/7
  4447 root     1.1  10 0.0 0.0 0.0 0.2  14  74  51  44 .2M   0 fbench/17
  4447 root     1.1 9.9 0.0 0.0 0.0 0.3  14  74  48  57 .2M   0 fbench/14
  4447 root     1.1 9.9 0.0 0.0 0.0 0.3  14  74  42  61 .2M   0 fbench/9
  4447 root     1.1 9.8 0.0 0.0 0.0 0.1  15  74  51  49 .2M   0 fbench/25

                set consistent_coloring=2
        	    in /etc/system.
        

Containers can be combined with Resource Management to dedicate cores and Hardware threads to Zones. More information is available at
http://docs.sun.com/app/docs/doc/817-1592/6mhahuolc

There are a number of ways to consolidate Applications on a Sun Fire T1000 and Sun Fire T2000 server.

The first is to let all the applications run unrestricted on the server. This can achieve the highest utilization of the system but runs the risk a single application may consume all the resources.

The second is to create multiple Zones and assign an Application to each. This has the advantage of Application isolation but again can leave to resource starvation.

The above two techniques can be combined with processor sets or resource pools to avoid starvation. Hardware threads can be bound to the Zone for instance to ensure quality of service.

As an example say an existing application farm is has a load balancer that load balances incoming requests into a rack of 16 two-way system with an ip-address in the range of 192.168.1.2 ..... 192.168.1.18 Each of these two-way system runs a Servlet/JSP application in a web container.

First pick the interface on the Sun Fire T1000 or T2000 server that will connect to the load balancer - let it be ipge2 we plumb the ipge2 interface and assign a ip-address to it - 192.168.1.1. We then create 16 logical interfaces and assign the ip-address of each of the two-way systems.

        ifconfig ipge2:1 plumb
        ifconfig ipge2:1 192.168.1.2 netmask + broadcast + up
        ...
        ...
        ifconfig ipge2:16 plumb
        ifconfig ipge2:16 192.168.1.17 netmask + broadcast + up
		

Install a SPARC (R) /Solaris (TM) version of the web container on the Sun Fire CoolThreads server. Create 16 instances of the server, (with Tomcat for instance, simply untar the Tomcat package in 16 different directories, tomcat1...tomcat16, or use the same binary location for 16 different instances)./p>

Each web-application server can be bound to a specific ip-address, configure each of the server to bind to one unique logical interface from the above list, start the servers and we have just consolidated 16 two-way systems in a single Sun Fire T1000 or Sun Fire T2000 system.

More information at

http://blogs.sun.com/roller/page/waterfalls/

Two Sun Blueprints have recently been released that offer some helpful insights into application consolidation on CoolThreads Servers.

The first, Consolidating the Sun Store onto Sun Fire T2000 Servers documents the process of migrating the online Sun Store from a Sun Enterprise 10000 with 38 400MHz CPUs onto a pair of Sun Fire T2000 servers (they used two for high availability). The resulting environment of Solaris 10 Containers. They saw an overall reduction of approximately 90 percent in both input power and heat output. The space savings were even more significant.

The second blueprint, Web Consolidation on the Sun Fire T1000 using Solaris Containers, gives a detailed hands-on description of the process of building a web-tier consolidation platform on a Sun Fire T1000 with Containers.

See also the blog on Consolidating Applications Onto a CoolThreads Server.

The following Linux distributions are available for the Sun Fire CoolThreads servers:

Ubuntu Linux 6.06
Gentoo Linux 2006.1

Linux support is provided by the distributions' respective communities

For more information on running Linux on Sun Fire CoolThreads servers, please click here: http://www.sun.com/servers/coolthreads/tnb/linux.jsp