University of Coimbra

Success at a Glance

Like all scientific pioneers, researchers at the Center for Computational Physics at the University of Coimbra needed a more powerful computing infrastructure. The university's existing 108-processor supercomputer, Centopeia, could not deliver the performance scientists needed to maintain a prominent position in the scientific community and advance analysis in numerous disciplines such as astrophysics, biomedicine, and condensed matter physics.

In addition, because Centopeia was the university's only supercomputer, the school was unable to participate in computing grids that support some of the world's most advanced scientific projects. For example, scientists at the European Laboratory for Particle Physics (CERN) are working on the biggest scientific instrument on the planet: the Large Hadron Collider (LHC). To process the expected 15 PB of data produced by the LHC annually, researchers will rely on a virtual computing infrastructure known as the LHC Computing Grid (LCG). However, participation in the LCG and other grids such as the Enabling Grids for E-sciencE (EGEE) requires computers to be customized in a way that can restrict parallel-processing capabilities.

Not wanting to handicap researchers on campus and at other institutions that rely on Centopeia, the university sought a way to acquire another supercomputer. In 2006, the school entered a re-equipment program competition promoted by the Portuguese Science and Technology Foundation — and won.

The university invested its winnings in a new supercomputer built on scalable, high-performance technologies from Sun Microsystems. Purchased through Sun Advantage Partner IDW, the supercomputer includes 132 Sun Fire X4100 servers. Chosen for its price/performance, the Sun Fire X4100 server is up to 43% faster than Xeon 5160–based servers from Dell, IBM, and HP. In addition, the Sun Fire X4100 provides for more network expandability than any other major competitor, with four integrated Gigabit Ethernet ports in a 1 RU form factor. Enterprise RAS features like hot-swappable, redundant power supplies and disk drives facilitate high levels of system availability.

Two of the X4100 servers are configured as management nodes, and the other 130 servers are nodes in a compute cluster. Together, the servers provide 520 dual-core AMD Opteron 275 processors. In addition, each server delivers 8 GB of RAM — or 1.04 TB collectively — and runs the Community ENTerprise Operating System (CentOS) 4.4.

Storage is provided by an existing Sun StorageTek 3511 SATA array. Connected to management nodes by fibre-channel links, the array can currently hold up to 6 TB of data and can be expanded to accommodate 54 TB. The Sun servers proved to have the best price/performance ratio among the competitors. The performance evaluation used both industry-standard benchmarks like SPECfp and Linpack but also scientific applications used by University researchers.

In May 2007, the new supercomputer — named Milipeia — went into production. Not only can the supercomputer scale to more than one-thousand nodes, the nodes can operate in a coordinated way, just as millipedes' legs do. Milipeia delivers 8 times the performance of Centopeia and is the largest parallel-processing computer in Portugal. Over the next two years, the university plans to upgrade Milipeia to approximately 2,000 processors so that it can reach speeds over 10 TFLOPs. Right now Milipeia has a peak performance of 2.29 TFLOPs and a sustained performance of 1.6 TFLOPs. In spite of the fact that Milipeia has over 5 times more processors than Centopeia, it consumes only around 3 times as much electric and cooling power in peak usage.

In June 2007, the supercomputer Centopeia was integrated into the EGEE and into the LCG grid network by the Coimbra branch of LIP (Laboratory for Instrumentation and Particles). Available to researchers 24 hours a day, the EGEE includes 41,000 CPUs and maintains 100,000 concurrent jobs. The LCG encompasses approximately 100,000 CPUs distributed across 500 international research institutes and universities. More than 5,000 researchers will rely on the LCG grid to process data from the CERN particle accelerator, which is scheduled to go into operation in May 2008. Also in 2007 the University joined the PRACE (PaRtnership for Advanced Computing in Europe) consortium, aimed at building a pan-European HPC infrastructure lead by Petaflops-class supercomputers.