AR Athletes

For its first foray into showcasing humans in AR, Google tasked Framestore with capturing the athletes’ movements at the highest visual quality, while maintaining the nuances of each unique performance. Since the precision with which they complete each movement is what separates these athletes as the best in their respective fields, the team went to extraordinary lengths to nail the most accurate representation possible.

Framestore collaborated closely with Google’s immersive arts team, allowing for a level of problem-solving that was essential as each athlete's capture presented its own set of unique challenges.

Using Google’s immersive Scene Viewer to reliably support 3D and AR experiences, Framestore worked alongside photogrammetry and motion capture partners to oversee the authentic likeness and performance capture of each athlete. To avoid the intrusiveness of standard facial motion capture techniques, the team opted instead for multiple high resolution reference cameras that would allow animators enough footage to recreate accurate facial moves and gestures by hand in Maya.

“Every athlete presented a different challenge; for example, the convenience of having a soccer player perform soccer tricks within a fixed motion capture studio volume was not possible for a gymnast, who needed to be captured in a full gymnasium with a foam pit,” said Marco Marenghi, Animation Supervisor at Framestore. “Likewise with a swimmer, who could not be motion captured at all! The swimmer needed a completely different approach and a technique that had previously been untested in its complexity.”

Since conventional motion capture technology was not possible underwater, a reference shoot for the swimmer’s AR performance involved extensive high resolution video capture in an Olympic sized swimming pool. The footage from nine high-speed cameras, six of which were underwater, was then stitched together to form imagery that enabled Framestore to recreate the athlete’s performance seamlessly.

“The entire process of acquiring and using the underwater footage was full of challenges,” explained Todd Herman, Senior CG Artist at Framestore. “The typical technique used to align footage from multiple shots was not going to work as expected when the cameras were underwater -- with a waterproof housing covering each lens, there were multiple refractions of the light taking place. Successfully aligning those cameras was itself an achievement.”

Another area of enhancement the team addressed was for a sprinter. As the sprinter was captured in a volume of only 80 feet, they were unable to reach her top speed before having to "hit the brakes." Framestore took footage of their record-breaking runs and modified the stance and speed of the mocap to match that of the reference.

In many instances, training schedules, locations, and the distance or space required to complete each skill altered the capturing process, so Framestore’s LA and Montreal-based animation teams stepped in to convert movements into a format that would fit the AR environment. This meant manipulating some of the performances into slow motion or slightly altering the ramp-up movements for athletes performing at high speed.

After much modelling, rigging, texturing, and animation work, the team converted the 3D assets into an AR format. Creating assets to fit within the restrictive parameters of mobile AR required a workflow and methodology outside of Framestore’s established pipeline, challenging Senior CG Artist Todd Herman to write custom Python scripts that would ensure maximum efficiency.

With motion captures taking place at the height of the global pandemic, Framestore’s immersive, animation and production teams coordinated globally to overcome scheduling, talent zones, quarantine requirements, and last minute location changes, safely producing multiple international shoots.

Framestore Pictures producers Anne Vega and Kevin Clark oversaw US-based shoots, in addition to shoots taking place across the UK and Europe.

“My role as director differed from a traditional live shoot because of this project’s vast creative and technical parameters,” said Fernando Cardenas, Director at Framestore Pictures. “Consider the particular needs of this project; to juggle complicated schedules, capture a diverse set of talents, and procure locations for complex motion capture performances in the US and Europe. That's when collaboration between Framestore’s various divisions becomes our strength. Understanding the vision and needs of the Google team was essential, and the amount of trust their team showed in us made for a truly amazing outcome and experience.”

The 3D and AR athletes are now accessible globally via the Google app on mobile devices.