SUN's Efficient Data Centers

SUN Microsystems completed a major consolidation of its global data centers --- nearly 40% reduction in space. A green benefit was an impressive reduction in energy consumption.


SUN’s strategy was to define and deploy a standardized and modular data center computing unit. A single design effectively scales-up to handle a broad range of profiles from small lab to mega data center.

Dean Nelson, Sr. Director of Global Lab & Datacenter Design Services provided me a tour and tutorial recently. Dean and his team have generously shared their best practices, which are available for free download from the SUN site. A few may also be found in the resource section at the end of this post.

The technical concepts are highlighted by a gray box and may be skipped without diminishing one's understanding of the business approach and benefits.

Goals
SUN started on its data center journey about five years ago. The primary goals were to

1. Reduce operating expenses: Use less real estate and less energy
2. Enhance flexibility: Eliminate the data center “speed bump” that keeps the business from rapidly deploying equipment to satisfy changing business requirements
3. Future-proof: Ensure DC infrastructure can grow to handle two times the load.

New technical strategy needed
SUN realized that the traditional data center approach had served its useful life since

• Raised floor cooling was unable to deal with higher heat loads created by more powerful computers.
• Adding equipment or reconfiguring was difficult, slow and expensive.
• Upgrading data center infrastructure such as power, cooling, or other environmental equipment was usually prohibitive, which was a major factor in the proliferation of data center spaces in the first place.

The traditional data center design could no longer be enhanced, a new approach was required.

Modular DC
As with all elegant designs, the new design is deceptively simple. SUN created a modular data center design by deploying standardized “pods”. As Dean Nelson, explained “A pod is an energy-efficient building block that can be duplicated easily to create a data center of any size in any location.”

Modular Data Center Specs

• Group 20 – 24 servers together to create a computing unit or POD.
• Move cooling to the POD. SUN continues with air cooling, the raised floor is eliminated. A/C is delivered from above the POD or integrated within the pod next to the servers. Raised floor can still be accommodated, but is not necessary for this design.
• Provide flexible and dynamic shared infrastructure for power. Different power requirements are built into the infrastructure upfront. To change the power to the pod / server rack, it is simply plugged into a different circuit without any downtime.
• Move patch panels and cabling to the pod.
• Standardize the design
• Improve server utilization
• Build infrastructure to 50% capacity to allow for 3 tech refreshes
• Virtualize applications

Implementation
SUN made a sizable investment of $250 million over several years, or approximately $300 per square foot average across new and existing space which included server upgrades, environmental and power infrastructure equipment and improved server utilization.

• Year 1: Understanding the portfolio (over 1M square feet with a variety of computing needs), take baseline measurements and invent a new way to build data centers
• Year 2: Complete smaller projects around the world. Perfect the design and create the deployment strategy.
• Year 3 -5: Execute on the major consolidations. California and Colorado being the largest efforts.

BENEFITS

Less real estate
As part of the consolidation, servers were upgraded to gain more compute power in a much smaller footprint. From 2,000 the average compute power has tripled.

In addition to DC plant improvements, SUN looked at the server utilization since it drives the number of servers required which in turn drives the size of the DC. Better utilization results in less real estate and less energy. A recent McKinsey study confirms that server utilization is still abysmal at most data centers and improvements can yield 20 – 30% return.

Results

• In the U.K. data center, real estate was cut by more than half
• 60% reduction of real estate was achieved in the California consolidation
• Overall, real estate was reduced by 40%. From 1.4 million square feet to 860,000 square feet.

Energy Efficiency
Next generation servers use less energy for comparable amount of work, but they provide a great deal more compute power. As compute power increases, more heat is generated in a smaller area. Consequently the heat gain is more sizable and the cooling more challenging. This is analogous to a space heater that is more effective in a small room than a larger one.

Without dramatic changes in cooling strategy, the energy consumed for cooling the identical compute power would have been greater. Rather than cool the entire data center as one unit, the cooling break-through was to decentralize cooling and cool each POD individually. Dean Nelson explained that “the POD improves cold air delivery and allows the DC to operate at higher temperatures”.

Equipment - technical approach
Infrastructure was upgraded to the most energy efficient available. Enhancements included a flywheel as an uninterruptedly power supply (UPS) systems, energy-efficient transformers, and a variable primary chiller plant.

Results:

• In California, SUN lowered data center utility costs by a remarkable 50% and reduced Sun's global carbon emissions by over 1% in just three months. As a result of this effort, Sun received $1.2 million in energy rebates plus a $250,000 "Innovation Award" from its local utility, Silicon Valley Power.
• In the U.K. data center, Sun cut energy consumption by 70%.

DC Efficiency Metric
More of the energy in the DC is now used to power the equipment rather than expended on cooling and other activities. A typical DC according to Department of Energy consumes 53% of its energy in cooling, other environmental conditioning and lost power. SUN is now only consuming 22% for these other activities which is 58% better than average. (Some statistics for the DC operators: Average PUE 2.13; SUN’s PUE 1.28)

Flexibility
One other benefit of the modular data center design is faster and less expensive installation to handle on-going changes to servers and racks. Dean Nelson reviewed how “It is now much less expensive to “snap-in” extra compute power since the power infrastructure was pre-configured to handle a range of power requirements.”

Results:

• It took just 6 hours to install 3,000 servers in San Jose facility
• Since SUN maintains the variety of equipment for testing of current customer install base and new products, there is constant reconfiguration of equipment. This activity once required weeks and with modular DC, it is just hours to reconfigure.

Future-proof
Traditional data centers usually build out the power and cooling for the maximum forecasted load. Dean Nelson shared that SUN took a different approach where “the infrastructure is also modular allowing it to grow to satisfy new requirements. This saves both capital and operating expenses.”

Results:

• The implementation includes the ability to easily double average compute power within the existing footprint.
• Planned ahead for potential new power and cooling approaches
• SUN has already added 30 kw server racks without engineering work; they were simply snapped into place.

Technical Details on building for the future

• Double the average density, power, cooling, and cabling requirements.
• Server electrical requirements range for 1 to 30 KW
• Change power distribution including three-phase power supply.
• Incorporate new cooling designs such as systems integrated directly into racks. The chilled water or refrigerant lines are already running to SUN’s pods

Conclusion
Now that SUN has worked out the kinks, these strategies and techniques can be used for any size data center with any computing profile. Not only new data centers, but an existing data center can benefit.

SUN’s Bangalore data center is such a scenario. As Dean Nelson reviewed, “Bangalore presented the challenge of working within the existing office building’s power, space and cooling . Also no servers were replaced during the project. Despite these constraints, we were still able to reduce our space consumption by slightly more than half while reducing our power use by 17 percent.”

Even more benefits could be achieved if combined with a tech refresh. A business is then able to extend the life of its current data center and operate it at a lower cost.

Resources:

• Aligning Business Organizations – Sun’s approach to building energy efficient data centers
• Video on California consolidation - Project approach plus great footage of DC
• The Role of Modularity in Datacenter Design – White paper
• Solution briefs