Olympic Gold
MSNBC.com wins big in Salt Lake City
by Michael Hurwicz
June 2002In 1997, NBC was gearing up to cover the 2000 Summer Olympic Games in Sydney, Australia, and was making plans for the 2002 Winter Games in Salt Lake City, UT. While the company obviously had expertise with television coverage, executives quickly realized that they weren't ready to produce Web sites for sports events of Olympic magnitude. At the time, NBC hadn't yet created its interactive arm, NBCi. And MSNBC.com was primarily a news site with limited experience covering sports events. NBC officials knew that any Olympics sites they built would have to feature up-to-the-minute results and related news content, serving as portals to immense libraries of statistics and related information.
Rather than try to build the sites on its own, NBC made a deal with Quokka Sports, a popular sports portal and development firm. For each of the 2000 and 2002 Games, Quokka would produce two sites for NBC—Olympics.com and NBCOlympics.com. Olympics sponsorships are big opportunities for vendors to get noticed. So, in exchange for the rights to produce the sites, Quokka agreed to front millions of dollars in equity, absorb all production and hosting costs, and split ad revenues with NBC in a 50/50 deal.
From the start, Quokka knew that it would be hard to make money on the Sydney Games, largely because of limits on the site content. Olympics Web sites are deliberately restricted to protect television advertising revenues. For example, live streaming video of events is limited to just minutes per day, and can be shown only after the footage has been broadcast on TV. About 3.7 billion people—well over half of the Earth's population—watched some portion of the 2000 Olympic Games on television. In contrast, an estimated 20 million people would log onto the Sydney sites—a minuscule revenue opportunity by comparison.
The Sydney sites were widely praised for their features, but not their profits. When the games were over, only 5.6 million unique visitors had logged on, according to IDG. Quokka wrote off this disappointment as practice, and hoped that the 2002 Salt Lake Games would be more profitable. However, the underwhelming performance of the 2000 Games Web sites made it hard to gain sponsors for the 2002 sites. Before the games even began, Quokka declared bankruptcy. It was April 2001, and the news came as a blow to NBC.
The second big blow came just weeks later. Logictier, the official Internet operations sponsor for the 2002 Games, decided that its business model wasn't working and backed out of its contract to provide hardware, security, facilities, bandwidth, and other infrastructure services for the sites. With less than a year left, NBC and the Salt Lake Organizing Committee (SLOC) suddenly found themselves without anyone to build and host the Web sites for the 2002 Games.
MSNBC.com Steps Up
NBC and the SLOC began a furious search for a replacement Web development agency. During May and June, the committee met with representatives from Yahoo, CBS SportsLine, and Ignite Media (the producer of the National Football League Web site). They were unable to strike a deal with any of the companies. With the Olympics only eight months away, time was running short, and no one was eager to take on a job with a compressed and inflexible timeframe, high visibility in case of slip-ups, millions in up-front expenses, and dubious financial prospects after that.
Finally, near the end of June, the SLOC found a rescuer in a company that had, in a sense, been by its side all along. It was NBC's joint venture with Microsoft: MSNBC.
MSNBC.com, the company that hosts and produces the popular news Web site of the same name, agreed to host and produce the Olympics sites. No financial details were made public, but industry buzz has it that MSNBC.com paid nothing for the rights, agreeing only to absorb the costs of the sites, reported to be around $10 million. (Normally, an Olympics sponsorship goes for $50 million and up.)
With fewer than five months before the planned November 2001 launch, and fewer than eight months before the opening of the Olympic Games, MSNBC.com had to build three sites—Olympics.com, NBCOlympics.com, and an Olympics version of its own MSNBC.com site (see the figure). Together, the sites would contain over 5,700 pages of content and would have to support a bandwidth of 4Gb/sec.
"It was a short window in which to get everything done," said Mike Corrigan, director of technology for MSNBC.com. "We thought we could handle it, based on our experience publishing MSNBC.com on a daily basis." However, the Olympics sites posed challenges that the group had never experienced. For instance, MSNBC.com normally receives between 4 and 5 million unique visitors per day. The Olympic sites combined with MSNBC.com—all running on the same infrastructure—would, at their peak during the games, receive 8.8 million unique visitors per day. And because of the nature of the games, Olympics traffic often comes in surges, as news of triumphs, skullduggery, and controversy emerges. The group decided that the sites had be able to handle a combined 300,000 concurrent visitors, compared with MSNBC.com's usual 35,000.
"At the beginning, this seemed nearly impossible," said Corrigan. "Having to design and build three sites...building an infrastructure for the largest sports event ever, and having to launch in five months and finish in eight seemed an incredibly tall order."
To top it off, the SLOC contract with MSNBC.com called for near-real-time results and online voting—two applications with more demanding timing and traffic characteristics than anything MSNBC.com had done before. For instance, the online voting application had to be able to handle 2.5 million votes in the space of a few minutes. Full results had to be online in less than five minutes, with critical events in less than one minute.
To ensure that the sites remained online and accessible in the midst of intense traffic, MSNBC.com planned to grow its physical infrastructure by about 40 percent, increasing the number of servers from around 30 to more than 50. The group added 100 contractors to its existing staff of 200. It also added a second data center to ensure more bandwidth and fault-tolerance in the form of redundant Web servers, application servers, and databases.
For load balancing between the two data centers, MSNBC.com bought two 3DNS systems from Seattle, WA-based F5 Networks. For local load balancing among servers within a data center, the group added four pairs of F5's BIG-IP controllers. The controllers are typically sold in pairs for redundancy.
Glory or Grief
Because Microsoft is one half of the MSNBC partnership, it isn't a coincidence that the sites would rely on Microsoft software. It marked the first time that Microsoft technology would be at the heart of Olympic Web systems.
At previous games with Web sites—in 1996, 1998, and 2000—the projects were completely IBM based. From the AIX operating system, RS/6000 SP Unix servers, and HTTP Server (a modified version of the open source Apache Web server), to the WebSphere Application Server Advanced, DB2 Universal Database, and Lotus Domino for content management, everything came from Big Blue. Even the load balancing was done with IBM Network Dispatcher software.
To date, only Unix had proven that it could reliably scale to Olympic proportions. Microsoft had encountered limited success in going after the biggest sites on the Internet. Of the five most-trafficked sites on the Web (in descending order, according to www.netratings.com: AOL Time Warner, Yahoo, MSN, Microsoft, and Google), only the two controlled by Microsoft (MSN and Microsoft) use Microsoft technology. The other three use some variant of Unix.
Success at the Olympics would be a public-relations coup for Microsoft. Any significant glitch would strengthen the perception that Unix was king on the high end.
MSNBC.com planned to expand the infrastructure that supported the main MSNBC site to clusters of Compaq ProLiant 8500 servers. Each server had eight 700MHz Xeon processors and 4GB of RAM, and was running Internet Information Server, SQL Server 2000 Enterprise Edition, and Windows 2000 Datacenter Server.
The Marathon Begins
In late August, the group hired KeyLabs to start load testing the site. KeyLabs devoted three employees to the project full time. "It was a two-year project, and they had six months to get it done," said Mike Fahnert, president of KeyLabs.
KeyLabs started with a discovery process. A project manager and senior test leads came on site and talked to MSNBC.com to learn more about the traffic patterns and critical paths, as well as both current and projected requirements for scalability and load.
KeyLabs then began devising a test plan and test cases. Early on, KeyLabs decided to bring in the WebLoad testing tool from RadView Software. The WebLoad tool let KeyLabs simulate heavy traffic more efficiently, allowing it to drive more traffic from the lab to measure realistic loads on MSNBC's site. If the testing software had been less efficient, KeyLabs might have misinterpreted a slowdown in the test machines as a slowdown on the MSNBC.com site.
Learning from a Setback
KeyLabs ran its first tests in early September. It was just in the preliminary stages when world events brought an unplanned stress test. The September 11 terrorist attacks pushed the MSNBC.com site well beyond any foreseen usage levels. Although servers didn't actually crash, they were only 43 percent available for about two hours following the attacks. During those two hours, the group quickly switched over to Akamai Technologies' EdgeSuite service, which caches entire pages on edge servers in Akamai's content delivery network. Normally, MSNBC.com serves only GIFs via the Akamai content delivery network (CDN). Switching the entire site over to the CDN let MSNBC.com remain online during the sudden traffic spike.
The switch to EdgeSuite took hours because MSNBC needed its servers configured and Akamai needed additional setup. Once these arrangements were in place, however, if a similar switchover ever became necessary again, it could be completed in minutes.
Akamai's EdgeSuite service was used in the Olympics disaster-recovery plan. "If there had been a major disaster, we could have turned on the Akamai EdgeSuite within a few minutes and hosted the Olympics site," said Corrigan. "This was our insurance policy." Although the system was tested, it was never needed for the Olympics.
The MSNBC.com group knew it couldn't miss its deadlines for the Olympics, despite the September 11 tragedy. It continued adding servers over the next few months. KeyLabs developed and refined JavaScript-based test scripts, generating traffic from over 1,000 computers at its lab in Lindon, UT. To see how well the testing platform itself would scale as loads increased, the company verified script functionality, tuned and profiled the scripts, and examined the impact of scripts on the machines driving the loads.
In October, KeyLabs pushed the same scripts out to 240 systems in five data centers around the country. This system was capable of inducing the MSNBC.com site to transfer data at 2Gb/sec. Under the pressure of these tests, MSNBC became proficient at analyzing results and reconfiguring its backend systems in hours rather than days. This was particularly necessary, as stress testing could only be performed for the approximately six off-peak hours each day.
In late November, KeyLabs pushed its test scripts out to a distributed testing network. It consisted of a million peer-to-peer nodes around the world and was managed by United Devices (UD). The UD network is an established group of users who have loaded the UD agent on their systems. When a system is idle, the automated UD scheduler (jointly developed by KeyLabs and UD) can remotely initiate site testing. If a UD user starts using his or her PC, the scheduler immediately initiates a backup agent on another node.
