Teaching Virtual Production in Unreal Engine

Introduction

Hi! I'm Deepak Chetty, I'm a Filmmaker, Educator, and emergent cinematic arts explorer based out of Austin, TX. I studied Film Production at Pratt Institute in Brooklyn, NY where I received my B.F.A and The University of Texas at Austin where I received my M.F.A. I consider myself a Generalist in the broadest sense of the term as it pertains to media creation as I've jumped all around over the course of my career over the last 17 years since my first gig. Regarding post-production, I've done motion graphics for HBO docs, been a compositor on projects such as Warner Media/Rooster Teeth's gen: Lock, handled invisible VFX on a multitude of feature films and shorts as a freelancer, and have led post-production on live-action XR work for projects such as Respawn/EA's Medal of Honor: Above and Beyond (The live-action 360-3D Gallery content), Stanford Medical Center's Project Braveheart VR Therapy Program, and special content for The Washington Post among others. 

On the production side, I initially trained as a camera operator and eventually became a cinematographer on shorts that have played at festivals such as SXSW, Slamdance, Hamptons International Film Festival, AFS, and more. The first feature I shot as a cinematographer, The Love Inside is available to stream free on most VOD platforms. My Masters Thesis at UT Austin, a sci-fi action film I wrote and directed called Hard Reset was shot natively in stereoscopic-3d and is available on Gunpowder and Sky's DUST platform and has been optioned as a feature film.

These days I spend most of my time developing new skills, working on my own projects, and contributing to education. Besides my position as a mentor for CG Spectrum College of Digital Art & Animation, I am an Assistant Professor of Practice at The University of Texas at Austin with a joint appointment in the Film and Fine Arts departments where I teach and develop academic workflows for Virtual Production, Motion Capture, VFX, and Interactive. I am also the Producer for the Film/TV/Virtual Production track at Epic Games’ Unreal Online Learning platform.

I've felt that the variety in my career and the in-depth work I've done in various disciplines has allowed me to get a really nice high-level understanding of content creation in almost any arena, and I've gathered all this experience and honed these skills as I move towards goals of creating larger projects of my own! 

Getting Into 3D Art

I come from traditional film production and fine art background. Pratt Institute is a very old-fashioned art school where I was exposed to all formats and mediums of art during my undergrad years. I owe a lot of what I consider my aesthetic sensibilities to the other classes I took outside of Film Production which included Light Color and Design (affectionately known then as LCD), Drawing, Photography, and Art History.

My education at UT Austin was a bit more specific towards Film Production, but I happened to be there at the exact time interest in XR and immersive was being supported on a University level with an internal program called UT3D. This sparked my interest in immersive and emergent cinematic arts and that's how I was able to make my Master's Thesis, which had over 200 VFX shots – in live-action stereoscopic 3D! It took almost the better part of a year to complete them after filming.

I was not too familiar with using DCC's for 3D asset creation, or integration at the time – my main experience in visual effects was with After Effects and layer-based workflows. That entire film, Hard Reset, was done using a custom and highly templated pipeline I created in Adobe After Effects. I used the Element 3D plugin from Video Co-pilot to integrate 3D models into AE's ecosystem in lieu of a traditional 3D pipeline. Without a doubt, my traditional art education helped and continues to help me in all my pursuits. It taught me not only the fundamentals but the ways in which art has evolved over time and how different mediums have influenced one another and converged as time moves on. 

Virtual Production

Virtual production, just like the production itself, is a broad term that covers an entire spectrum of disciplines and talents. Its usage is cross-discipline, industry, and format. Due to the convergence of traditional framed cinematic or linear content and interactive or immersive content – the definition of what constitutes Virtual Production to many could boil down to the process or procedures in which content is created.

Real-time rendering is commonly associated with virtual production, and for a good reason! It's a large part of the projects that have popularized the term virtual production in this recent era of innovation – but I believe one can exist without the other. Many projects that employ virtual production techniques still are rendered offline and not in 24 FPS (or whichever desired framerate) real-time. A very simplified way to look at the categorization of virtual production, which may truthfully to some be too simple, is to imagine a skillset or category of physical/practical production and think of how that works analogously in the world of CGI or hybrid production. The cinematographer is the virtual cinematographer, the art department is the virtual art department, etc. The aspect of sound might be the only aspect of content creation that remains relatively untouched in regards to training or skillset since in most cases it is such a function of post-production, which is in a sense all virtualized now that we do not cut our linear cinematic content on reel to reel machines but computers! 

Teaching at CG Spectrum

My term in CG Spectrum’s Realtime 3D Technical Art & Virtual Production Diploma is designed to lean alongside my strengths as a practical filmmaker who understands how the various disciplines and attributes of practical filmmaking may translate into the virtual world. As an example – my experience as a cinematographer allows me to explain why using real-world values for your camera settings, such as sensor sizes that line up with real cameras, lens properties that line up with real lenses, etc. can actually help in a sub-conscious or even very overt way to your audience feeling like what you are creating in-engine is cinematic. It's principles like that, or how I'll sometimes operate a motion-tracked camera rather than keyframed one – so the sense of human operation is imbued into the camera movement.

I understand not all scenarios call for human operation of a camera, but in many cases – it's that human element, with micro-movements and slight timing delays, etc. which are spontaneously crafted within each recording that allows for the uncanny valley to shrink as it pertains to the virtual representation of cinema and reality.

Each week of the virtual production course is carefully crafted to not only engage with these techniques across the certain discipline or subject we are covering but discuss how best to achieve the feeling of cinema within. That's not to say that fully keyframed cameras are bad, absolutely not – many camera moves we see these days, especially in larger budget productions are not physically possible – but it starts the discussion for when and where different methods are appropriate in regards to how we are telling the story and captivating the audience.

By the end of the term, students should walk away with a fully completed short film – that is the goal! Not a shot, or a sequence – but an end-to-end narrative made using virtual production techniques within Unreal Engine. Hard Reset took me almost a combined total of six weeks to render on a single PC (not including errors and crashes), now I'd peg it exponentially less if I had conceptualized and executed the visual effects in a real-time environment. It's an unimaginable difference. The opportunity to utilize what I like to call rapid creative iteration, which allows a creator the flexibility to modify all aspects on the fly and the fact that this is democratized with such wonderful results is what is truly extraordinary about this technology and its utilization.

Unreal Engine

In my experience as an educator, the difficulty of getting started with lights and cameras in 3D is proportional to how much experience, understanding, and affection you have for those values in reality. An individual skilled or well versed in photography and/or cinematography who inherently understands real, physical cameras and light properties will likely have an easier time onboarding into representing these values in software. In virtual production, most of the time – our frame is created and/or represented by the field of view of a camera. So understanding values such as manual exposure, shutter, aperture control, focal length, sensor size, and how the last two values contribute to the field of view, etc. all contribute to how you might block, light, and shoot a scene.

If one is not familiar with those principles, I would highly recommend researching them or even practicing with a DSLR camera or things like the director's viewfinder apps on one's phone. Sometimes it's hard to explain with just words how a close-up of a character can feel emotionally different if shot physically close to the subject with say a wide-angle lens, or shot with the camera placement farther back with the relatively same framing/field of view on a longer focal length. But when visualized, which is very easy inside of a program like Unreal Engine – the difference becomes immediately clear. As such, using Unreal Engine is a perfect way to onboard people into Virtual Production as the ability to be physically accurate with settings and values is built-in to many parts of the software and allows for a natural transition into the virtual world. Now, that's not to say that you can't just jump in and make things up – and use non-standard values, just don't expect them to be as recognizably cinematic from the get-go. One of the uses of Unreal outside of content creation that I've used over the years as an educator is just demonstrating cinematography techniques in-engine – as though I was doing the same on a sound stage. 

Challenges

I always, always, always refer to the films and art that have inspired me. Visual expression rarely emerges from a vacuum – mine certainly does not. What is an individual artistic creation if not an evolution of what has come before? I wholeheartedly embrace what has inspired me and I look for ways to integrate that into my work. While the iconic shot of Chief Brody on the beach in Jaws might be one of the most well-known Hitchcock zooms, if not the most, it's still called the Hitchcock zoom right? 

I love this comparison video of Star Wars: A New Hope's climactic trench run and a similar scene clearly used as inspiration in a WW2 film called The Dam Busters. What a genius idea of taking the visual representation of an earthbound aerial dogfight and transposing that into a space battle, making the latter feel altogether much more real and familiar. The trick is that homage is not a dirty word or concept, it's a stepping stone to your own creation. Whether you like it or not – the subconscious is always pulling reference – you might as well be conscious and in control of how you implement your inspirations! 

Conclusion

I know filmmakers who never started well into their 30s. I know architects and designers who have switched over to gaming or film in their 40s! The list goes on and on. The shared characteristic of these people that I have observed is that they are relentless in their pursuit of knowledge and the creative application of that knowledge. Success may not come quickly, or easily – it has to be earned. 

The last piece of advice that I'd give is that no matter what your age, and well, especially if you are older and making a career change – your attitude goes hand in hand with your ability to succeed. I've witnessed many talented people flounder because they can't take criticism and never grow as an individual nonetheless an artist. Whereas on the flip side I've seen people whose talent has not reached such a level yet go on to great success because they are open to learning and are incredibly collaborative. Very few positions in this industry require less than a constant amount of collaboration between either individuals or departments and having a good attitude and a willingness to collaborate outside of how purely talented you are going a long way towards setting yourself up for success.

You can learn more about me through my website or check out CG Spectrum’s online Realtime 3D Technical Art & Virtual Production course which I helped create.

Deepak Chetty, Epic Games Unreal Online Learning Producer and CG Spectrum Realtime Mentor

Interview conducted by Arti Sergeev