How Pixar Saved Toy Story 2 from Almost Complete Deletion
Proper data protection during production is critical, and Pixar almost learned this the hard way with Toy Story 2. Here’s the story.
I recently stumbled across a story from the history of film from YouTuber Austin McConnell that I had never heard before. In 1998, less than a year before the slated release date of Toy Story 2 an animator working on the film entered a command to delete files into the Unix-based, shared-network project system, intending to run a routine file cleanup. Little did this poor animator know that the hasty command entered into the system would result in an avalanche of errors that would nearly bring Pixar to its knees.
Today, we are looking at the story of Toy Story 2 and how it was saved from 90 percent deletion.
Saving Toy Story 2
In 1998, Associate Technical Director Oren Jacob was working with Larry Cutler and Larry Aupperle when they began to notice elements of Woody’s outfit disappearing.
‘That’s when we first noticed it, with Woody,’ recalls Oren Jacob.
More assets kept going offline, and “Directory No Longer Valid” errors stopped the work on Toy Story 2 in its tracks.
Eventually every animator and every TD, everyone working on the show, goes, ‘Oh, all machines down. Let’s go to lunch.’
When Oren and the rest of the crew returned several hours later, the network drive was brought back online and it was discovered that only 10 percent of the film’s assets remained. Despite this, panic hadn’t broken out yet. Supervising Technical Director Galyn Sussman requested tape backups of the film from the IT department.
The IT department furnished the archived work, which the team then used to restore the project and all of its assets. Once the department heads felt confident in the restoration, the rest of the production team was invited back to work on the film.
Unfortunately, Pixar didn’t have regular testing of backup protocols in place at the time, and as a result, they were unaware of a critical error that prevented the tape backups from properly archiving the work done on the film. As the dust settled, the production team on Toy Story 2 began to realize the nightmare scenario they were in.
‘We had restored the film from backups within 48 hours of the [deletion command], run some validation tests, rendered frames, somehow got good pictures back and no errors, and invited the crew back to start working. It took another several days of the entire crew working on that initial restoral to really understand that the restoral was, in fact, incomplete and corrupt.’
The crew was again sent home, and a do-or-die meeting was called with Pixar executives and department heads to assess the situation and decide the fate of Toy Story 2. The film would have to be delayed, restarted, or scrapped altogether.
‘Now sadly, what’s happened is that there is zero confidence in any solution, because the restoral is bad, the work on it is bad, the deletion was horrible, and the backup tapes are busted.’
}
The somber atmosphere of the sentiments being expressed in that board room must have been palpable. That’s when Galyn Sussman remembered the backup version of the film she kept when she was working from home to be with her newborn son.
Sussman’s backups were the last chance Toy Story 2 had to be completed on time. A plan was hatched, and Oren and Galyn carefully retrieved her computer and brought it back to the studio.
Here is an awesome, firsthand account from Oren and Galyn about the stress they were under while retrieving and returning the computer to the studio. After the stressful journey, the computer was delicately transported to the studio.
‘We dupe’d that data immediately, then set about the task of trying to verify and validate this tree, which we thought might be about two weeks old. We compared Galyn’s restoral with a much older one (from two months prior) and couldn’t determine a clear winner; there were too many inconsistencies. So, instead, we set about the task of assembling what effectively amounted to a new source tree, by hand, one file at a time. The total number of files involved was well into the six figures, but we’ll round down to 100,000 for the sake of the rest of this discussion to make the math easier.’
From there, the team went to work verifying each asset from all versions of the film in order to determine what assets to restore to the project. Approximately 70 percent of the project was easily verified as restored, but that still left some 30,000 assets to sort through manually.
The team pulled together, got their sleeping bags, and worked in 8-hour shifts around the clock to restore the film’s remaining assets over the weekend.
‘In the end, human eyes scanned, read, understood, looked for weirdness, and made a decision on something like 30,000 files that weekend.’
But, as a result of the effort, Toy Story 2 was saved.
There is so much more to the story than I can cover here, so check out an in-depth article with Galya and Oren about all of the circumstances surrounding the deletion and recovery of the film.
The Importance of Good Data Management
I’ve talked before about the importance of proper data protection techniques. Recovering deleted or corrupt data is always extremely stressful and arduous. The prospect of losing over a year of work on a 100 million-dollar film is an important lesson, and it highlights some of the biggest ways in which any project can be brought to its knees without a highly systematized backup and archival process.
To me, the lesson of Toy Story 2 is one of the most important an aspiring filmmaker can learn. Don’t get caught without a plan; it could mean the death of your project.
All images via Disney.
Looking for more on film and video production? Check out these articles.
