LogInDerek Potter, Head of Product Management at Vicon, talked about their finger tracking system Shōgun High Fidelity, its potential influence on the game industry, and their collaboration with Epic Games.
In case you missed it
You might find these articles interesting
About Vicon
Vicon was founded over 35 years ago when several engineers recognized the potential of analyzing human movement with optical cameras. Vicon’s earliest products were used in the medical field to help health care professionals analyze sets of basic body movements during clinical assessments. These systems were composed of a small number of lower resolution cameras and ran on very old Unix-based computer systems. As the technology progressed, the system was able to capture and analyze more complex movements, the whole body, people walking and eventually highly complex and detailed movements, like multiple subjects interacting. This is when (more than 25 years ago) the technology caught Hollywood’s attention as production studios were interested in using human movement captures to make more realistic visual effects for movies and video games. Since then technology has continued to evolve with new use cases across many fields, including engineering and sports science.
Finger Movement Tracking
As motion capture moved from niche to core usage and acceptance in the VFX world, the demands from the user base for high detail and expressive nuance from actors grew. The hands interact with others, with the object, and with the world – they connect us. Most importantly they express. But their size, flexibility and ability to interact with each other and objects in the world have made human fingers one of the most challenging tracking elements of the human body. So, capturing very detailed finger movements has been an important goal of VFX users for many years.
Our investment in this new level of finger tracking, Shogun High Fidelity, was driven by both detail level and the costs that have traditionally been associated with finger tracking. The detail level (Fidelity) of our new Shogun finger tracking is unmatched in the industry today. The fingers create hundreds of complex configurations. But what makes them so expressive also makes them the most challenging. Previous finger tracker solutions often required a grueling amount of manual intervention in post-production. Our new finger tracker means more detailed and higher quality finger data in the first pass. Better the first time, more accurate and less time in post. The reliability and robustness of the solution also result in a significant reduction in the cost.
Process
The process of capturing finger data is not drastically different from what we have seen in the past. The difference lies in the approach we’ve taken training our models and creating our processing algorithm. So, in short, it’s a new smarter approach that has created this big jump forward. We started the process with the benefit of having more than 15 years of experience and understanding of finger tracking: what makes it challenging, what works, what doesn’t and what do VFX artists want and need. Additionally, modern cameras are very high resolution compared to 15 years ago, and PC processing power is much more affordable today. Our development team’s approach involved a combination of our strong understanding of hand anatomy and very large data sets. We invested heavily in creating challenging finger tracker data sets, painstakingly went through every frame to ensure it was correct (something that many animators need to do) and then used advanced statistical analysis techniques to understand how to improve the marker sets and tracking reliability. In short, human anatomy, lots of data, lots of care to ensure correct data and then lots of statistics.
Collaboration with Epic Games
We have a great relationship with the people at Epic Games and their understanding of motion capture and the creation of realistic digital humans is second to none. It has been very exciting to work with them on several of these projects.
Were there challenges? Yes, absolutely, but when you’re pushing boundaries that’s what you need to expect. The greatest challenge was transitioning our brand new hand models and techniques from theory to production under high pressure. Baptisms of fire such as this often produce some of the best results. The amazing thing for us was that during the preparation for these demonstrations, you go through an intense period of real-world testing with expert input from your partner – who is your target audience. And when you have the opportunity to work with groups like Epic Games who have a combination of world-class VFX artistry and technical ability, the progress you can make is astonishing.
Future Prospects
One of the things we’re most excited about with our new finger tracking technology is that it can continue to improve and evolve when more and more data is input into it. That’s the way these types of large data/learning algorithm approaches work. You need data and an understanding of where any remaining fine tweaks need to be applied. The more users who use this, the more data and input we have to continue to evolve the tracking.
How could this change video games? I think for the bigger studios this type of tracking can be used to capture a level of mocap actor expressiveness that hasn’t been seen before, which helps the movie-goer or video game player become even more immersed in the experience. The increased ease of capture and reduction in post-process costs also means the finger tracking will be accessible to smaller teams who wouldn’t have even considered this in the past.
When will it become commonplace? Finger tracking is something everyone who mocaps has wanted to do. The constraints have been the ease, the quality, and the cost. We’re hoping that what we are offering now will mean that this becomes commonplace in the very near future.