SoundCloud

Sun Servers, Open Storage and Startup Essentials Program Help Support Monthly Growth of 25% at Web 2.0 Startup

Founded in 2007, SoundCloud provides a Web-based platform that is transforming the way music is shared. Over 250,000 customers — including artists and other music professionals — already rely on SoundCloud to send, receive, and distribute music. Based in Berlin, Germany, the company employs 12 people.

Customer Challenges

Support exponential growth
Speed IT productivity
Accelerate site response times
Increase availability

Solution

SoundCloud took advantage of Sun Startup Essentials to migrate its Web server stack to a solution that includes virtual servers and a hybrid storage pool built with Sun technologies. Hosted and managed by London-based EveryCity, the new architecture quickly scales with growth, delivers the raw performance customers expect, and gives SoundCloud employees the ability to focus on expanding the business.

Business Results

Reduced application maintenance by 50%
Averaged a 25% increase in customers each month
Scaled to support a tenfold increase in Web traffic
Increased site performance
Realized 99.99% application availability

Products and Services

Story Details

In 2008, SoundCloud launched a service that gave artists and other music-industry professionals an easy way to share music from a central location on the Web. SoundCloud built its initial offering with a custom Web application that made use of commodity hardware running Linux and a database built with MySQL Community. All sound files were stored remotely using the S3 and EC2 services from Amazon Web Services, an Amazon.com company. Although the architecture supported the first few months of business growth, by the end of 2008 the hardware supporting the Web application and database neared capacity and could no longer meet performance requirements. Availability became an issue, and IT personnel had to dedicate too much time managing the Web application and data flow. As performance and reliability decreased, so did the number of customers.

Realizing that its business depended on IT performance, SoundCloud evaluated its options and found an ideal solution with Sun Startup Essentials. Through the program, SoundCloud could take advantage of discounted Sun technologies as well as hosting services through EveryCity, a managed services provider in London. “EveryCity presented a really attractive deal with virtualization,” says Sean Treadway, chief architect at SoundCloud. “We could have scalable CPU nodes that we could quickly bring up or down to meet our application needs.”

" Before, it was difficult to scale our storage. I had to use some NFS tricks to distribute where files were stored, which cluttered our Web application so that it became more difficult to maintain. With our Sun solution, we have a good strategy for scaling different application bottlenecks and are no longer limited by a fixed storage-pool size. We can grow our storage pool as required without having to worry about where the space is or how it will work. "

— Sean Treadway, Chief Architect, SoundCloud

EveryCity hosts customer environments on virtual servers built with Solaris Containers, housed on Sun Fire X4150 servers. To alleviate I/O bottlenecks and meet cost constraints, each virtual server stores data in a ZFS hybrid storage pool located on a Sun Storage 7210 Unified Storage System, with Solid State Disk technology. This is accessed via the iSCSI protocol, and Quad Gigabit Ethernet links help to accelerate communication between components. In addition, the built-in analytics, leveraging Solaris DTrace and the Sun Fault Management Architecture (FMA), quickly identify issues, speeds resolution, and provides specific information that can be used to tune architectures and applications.

“In terms of performance, we can use the advanced analytics on our Sun hybrid storage solution to see which IP nodes are processing higher transaction volumes than others, and then identify which Solaris Containers need to be moved around to distribute the load more evenly,” explains Duncan Malcolm, commercial director at EveryCity. “When new servers are created, scripts automatically provision storage.” To further enhance performance and availability, EveryCity also makes use of file system snapshots, clones, and data compression. “Unlike traditional compression that can slow things down, data compression in ZFS actually speeds things up,” explains Alasdair Lumsden, technical director at EveryCity.

In January 2009, a team from SoundCloud and EveryCity began the migration. Their timing was fortunate. Early in the process, the database server had reached capacity and the SoundCloud Web site went offline. “We pulled the data from a memory dump out of MySQL,” explains Treadway. “EveryCity imported the data, and 10 hours later, we finished the migration.”

Today, customer requests are processed by multiple instances of the Web application running within a cluster of Solaris Containers. Sound files are processed and sent to Amazon Web Services. Customer information, images, and artwork is tracked in a database that runs on a Sun Fire X4150 server. The database data is stored on eight 15k RPM SAS disks, striped and mirrored using the ZFS filesystem. Web application data and the Solaris Container boot images are stored on the Unified Storage System.

As a result of its new solution, the Web application driving SoundCloud maintains 99.99% availability and an Apdex score of 0.961.0 — while supporting rapid growth. Over the last 11 months, the number of site users has grown from 20,000 to 250,000, an increase of about 25% each month. In addition, the site processes about three million dynamic page requests per day. “Today, our systems are in a different league altogether in terms of performance,” says Eric Wahlforss, founder and CTO of SoundCloud. “We haven’t had any unscheduled downtime, and our systems are handling more than 10 times the traffic.”

Employees at SoundCloud are also more productive. “It used to take 50% more time to handle routine application maintenance,” says Treadway. “Now, at the flip of a switch, we can ask for more resources and can quickly allocate those without adjusting the application itself.”

Currently, SoundCloud is re-evaluating its outsourced solution with Amazon due to its high cost and inconsistent performance. “We’re getting to the point where we’re storing so much data on S3, and our availability requirements are so high, that the S3 solution just isn’t enough for us,” says Wahlforss. “We are currently looking at moving our sound files off S3 and on to a Sun storage solution sometime over the next three months.”