How DreamWorks Animation Improved Its Production Process Using Red Hat Enterprise Linux
Teaming with Red Hat, DreamWorks Animation improved production processes using open source and cloud computing, lowering hardware costs by 75% and reducing annual software licensing costs by 60%. I&O leaders standardizing on open source can adopt this framework to achieve comparable results.
- As early as 1997, DreamWorks Animation knew it had to migrate away from costly proprietary software and Unix RISC processor-based workstations to more efficient scaling and a reliable and stable infrastructure.
- When DreamWorks Animation chose to generate three-dimensional (3D) computer-generated films, it found that it needed a high-performance compute level of digital resource that would scale to support new special effects and a higher level of imagery than it used on previous films.
- DreamWorks Animation needed to utilize a stable platform to develop its proprietary software.
- "Type A" organizations should exploit open-source software (OSS).
- "Type B" organizations should test OSS for specific-use benefits that are complementary to proprietary alternatives.
- "Type C" organizations must use caution, as IT procurement can also become locked into OSS vendor subscription fees.
- All organizations should secure senior executive support to help establish new governance and procurement policies when choosing to standardize on OSS.
Film studio DreamWorks Animation transitioned from traditional animation to computer-generated films, as the company sought to move from creating images on costly proprietary solutions, such as Silicon Graphics Inc. (SGI) workstations, and on to platforms running OSS. Studio artists needed to render visual images with increased complexity and better performance on commodity hardware. At the time, using open source was still a vague hope, fraught with risk centered on reliability, support and consistency.1
DreamWorks Animation's executives chose to standardize on OSS for production on x86 workstations and servers, and teamed with Red Hat and HP to make the transition. DreamWorks Animation has been implementing Red Hat Enterprise Linux (RHEL) for more than a decade, as it uses Linux to scale its digital resources and accelerate its go-to-market process. DreamWorks Animation also decreased operating expenditures and increased efficiency in creating an improved visual product. The collaboration led to a "technology revolution" within the film studio that eventually led to rendering significantly more detail, from strands of hair to hundreds of dragons. According to DreamWorks Animation, it considers itself both a technology and animation company because of the high degree of innovation and technology its artists use to produce computer-generated animated films.
During the late 1990s, DreamWorks Animation developed film images using proprietary hardware and software solutions. The company turned to RHEL on HP x86 servers.2 The 13-year relationship between Red Hat and DreamWorks Animation created a mutually beneficial relationship through technology exchange. For industry I&O leaders looking to transition to open-source technologies, we offer a framework from one company that reaped benefits using this approach.
Early on, DreamWorks Animation recognized the benefits of using computers to drive animation, as it recognized that commodity computing was overtaking proprietary workstations and it needed to deal with that inflection point. The team evaluated other options, including Microsoft software and variants of Unix. However, it did not want to be locked into proprietary solutions and chose an open-source framework to save on production costs, increase agility and interoperability with other systems and reap a larger ROI.3 DreamWorks Animation does not rely solely on open source; its executives say that it will use proprietary technologies where appropriate. DreamWorks executives believe that choosing an OSS vendor is easier due to advantages of mutual technology transfers and innovative technology collaboration.
DreamWorks Animation works in three- to five-year time frames to produce an animated film. Moving to OSS and RHEL:
- Lowered hardware costs, in year 2000, from $40,000 per unit to just below $10,000 per unit as a result of moving from Unix-based SGI ($40,000 per workstation) to commoditized HP x86 hardware and OSS ($10,000 per workstation).
- Reduced testing costs by an estimated 50% as DreamWorks Animation needs to do less testing of its tools and hardware platforms during upgrades. DreamWorks Animation only needs to retest when upgrading major RHEL distribution versions and not for each dot release (as with SGI's IRIX).
- Slashed compute hours of render time on "Shrek" with Red Hat support; both on-site and remote; and weekly and daily updates over 2.5 years. With added flexibility of the OSS SW, DreamWorks Animation could devote more time to new developments of its proprietary software, as artists could render images and receive animation feedback faster at their desktops without needing to wait for a nightly render cycle.
- Red Hat middleware enabled DreamWorks Animation to develop a distributed rendering system.
DreamWorks Animation evaluated RHEL for its synergistic approach. For industry I&O leaders making a strategic shift, we recommend securing the full backing of your CTO and CEO. DreamWorks Animation benefited from strong executive support, as it could not make such a shift without a corporate commitment for directional change. DreamWorks Animation also secured Red Hat engineers and consultants to work on site as needed on a weekly or daily basis. That resulted in DreamWorks engineers improving their own proprietary tools. Red Hat engineers customized open-source tools so they would work appropriately for DreamWorks, enabling Red Hat to engage in productizing projects, such as its grid computing solution. Both firms continue to collaborate on file access optimization. The timing was particularly beneficial to both firms but may not be an available option to other IT leaders without mutually beneficial returns.
For DreamWorks Animation, the pieces were in place to ramp up scaling for production. And in 2007, DreamWorks chose to adopt open-source virtualization to consolidate servers. By 2009, it began implementing Red Hat Enterprise Virtualization, which helped the studio to avoid a $20 million-plus data center build-out and stay in its current footprint. This virtualization implementation makes DreamWorks one of the early adopters of Red Hat Enterprise Virtualization.
Additionally, DreamWorks Animation chose Red Hat Messaging and Grid (MRG) computing solution to replace its former proprietary Load Sharing Facility (LSF) grid computing software. MRG enabled better batch scheduling of millions of render jobs that run daily; these average 500,000 per day. This breakthrough resulted in:
- Reduced costs by eliminating Platform Computing LSF licensing costs, saving the studio 60% of its annual costs.
- Better management of system tasks, as Red Hat MRG enabled DreamWorks Animation to transition from an architecture where it manually distributed work between different clusters to a single multiple-site cluster where the system distributes work based on available resources. Shipping racks of servers between the firm's Glendale and Redwood City, California, sites had cost up to $50,000 per year.
Red Hat based the productization of its virtualization and clustering tools on the successful launch of Linux in the entertainment industry. Had DreamWorks Animation stayed the costly proprietary path for a few more years, it may have lost momentum from its "Shrek" success and its industry competitive edge. Using open-source development tools and libraries, DreamWorks Animation also can contribute to an open-source volumetric data library, OpenVDB, for the industry. That change also spearheaded techniques such as:
- Huge explosions and atmospheric effects
- Millions of characters in a crowd
- A single dragon in "Shrek" to many 3D dragons in "How to Train Your Dragon"
Each image improved the moviegoer (DreamWorks Animations customer) experience, especially in the breakthrough film, "How to Train Your Dragon," where open source made it possible to create hundreds of digitally driven and visually appealing dragons, rather than just the one in the early "Shrek" films. "How to Train Your Dragon" could not have been made more than 10 years ago, since the cost to deliver 60 million CPUs in production would have been prohibitive. Open source enabled DreamWorks Animation to adapt its proprietary software to new hardware quickly, and the continuity and stability of the platform facilitated the expansion of toolsets. Reduced licensing costs now allow the team to grow its resources at an incremental price and render more complex imagery.
Most importantly, improved animation imagery contributed to improved box office revenue. DreamWorks improved its box office revenue from "Shrek" to "Shrek 2" and "Shrek the Third," and even more significantly for "How to Train Your Dragon." Worldwide revenue grew as follows:
- Shrek — $484,399,218
- Shrek 2 — $919,838,758
- Shrek the Third — $798,958,162
- How to Train Your Dragon — $495 million in gross revenue
For some I&O leaders looking to transition to open-source technologies, we recommend that you team with a strategic vendor with whom you can cultivate a strong working relationship with the open-source community. DreamWorks Animation worked with Red Hat at many levels, including having Red Hat engineers and consultants on site.4 In addition, DreamWorks Animation teamed with Red Hat at the distribution level. For example, when an important graphics driver did not integrate properly, Red Hat provided engineering depth to ensure seamless integration.
In addition, DreamWorks Animation focused its participation around select open-source projects where requirements, challenges, and approaches resonated for the studio's business and utilized Red Hat to engage with the open-source community. To help express those desires and own the implementation, DreamWorks Animation worked with Red Hat's RHEV virtualization team and the open-source libvirt, KVM (or Kernel-based Virtual Machine), and oVirt projects for virtualization.5
For industry I&O leaders, drive only solutions that are already well-tested and developed; forgo nascent technologies. Gartner believes that I&O leaders should leave development to the open-source partner.
DreamWorks Animation leverages cloud computing options for various areas and is a proponent of grid computing and private cloud. Moreover, DreamWorks Animation artists remotely render 10% to 20% of its compute business in a private cloud. DreamWorks Animation uses the HP Flexible Compute hosting facility in Las Vegas to augment computing resources when spikes in production occur as the firm spends about 60 to 80 million compute or render hours on each film. With 10 movies in production at any given time, DreamWorks Animation uses existing compute cores throughout global sites for rendering. If one site needs additional resources, it can tap another site. Instead of acquiring more servers, DreamWorks Animation expands its resources with HP Flexible Compute. DreamWorks developed a methodology for file caching that enabled artists to use servers from other studio sites in the rendering process. While that worked well for the private cloud, that process is cost-prohibitive for the public cloud. DreamWorks plans to continue to develop proprietary animation software on top of open-source technologies and increase compute hours to the cloud as production needs increase. OSS will facilitate its cloud deployment. With MRG, DreamWorks Animation has an elastic rendering facility that alleviates licensing costs.6 OSS also enables hardware access on a fast turnaround basis.
DreamWorks Animation, creator of computer-generated animated feature films, television specials and live entertainment properties, has released more than 20 feature films, including "Shrek," "Penguins of Madagascar," "Kung Fu Panda," "How to Train Your Dragon" and "Puss in Boots." DreamWorks Animation, based in Glendale, CA, releases two to three films per year, with each film taking approximately three to five years to produce. DreamWorks, with 2,500 employees, teamed with Red Hat to stay leading-edge as it chose to standardize on Linux for smooth transition capabilities and interoperability with other tools (see Note 1).
Additional research contribution and review provided by Gartner analysts Andrew Butler, Mike Chuba and Tony Iams.
2 As of 2013, DreamWorks Animation has 900 servers, 2100 render farm compute servers and over 1,800 workstations with 90% of servers and all workstations on Red Hat. Thirteen specialists support servers, desktops and other digital operations. Five petabytes of storage capacity exist with 200 to 300 terabytes per year of growth in the archival tier. All servers are HP.
5 All storage is provisioned and made accessible globally. Virtualization is based on RHEV-M. Bare metal is used only for the high-performance computing (HPC) clusters or core services. Cloud management is employed using OpenStack, VMware and a large collection of in-house tools.
It takes three to five years to produce one of the DreamWorks Animation films. In another example of technology progress, "Megamind" took a lot of production time in the final scene where the spaceship explodes. In a recent film, "The Croods," the cave destruction scene is composed of 11 distinct special effects. The render hours have grown from 10 million render hours to produce "Shrek 2" to 80 million render hours to produce "The Croods."