Creating Photoscanned Props With RealityCapture & Unreal Engine

Introduction

Hello, 80 Level readers. My name is Xiaoyu Cui and I'm a 3D Environment Artist from China, currently working at Tencent.

I liked playing video games when I was a child, so I always had a deep interest in video games and was very curious about how the worlds in the games were made. It wasn't until college that I had a real opportunity to understand and learn how games are made.

I spent a lot of time learning 3D modeling on my own, watching tutorials on YouTube, and communicating with some industry insiders. They helped me solve a lot of problems, and I am very grateful to my friends who helped me.

My first job was at Virtuos. I also worked as a 3D modeler on such games as The Outer World, New World, Spyro Reignited Trilogy, L.A.NOIRE, and many others.

Getting Into Photo-Scanning

The reason I started photo-scanning is very simple, the traditional way of making a model would take me lots of days or weeks manually modeling and texturing, and may not get the good variation and convincingness of the small detail.

But if you use photogrammetry, you may get all of it in one day. Also, I noticed that many AAA games adopted photoshop-scanning technology to make the final models very realistic and the production pipeline short. Therefore, I was very curious about how to implement this workflow. Then, I searched for relevant information on the Internet and started trying to scan some models by myself, which may be helpful to my own projects and job.

To start photo-scanning real-life objects, you certainly need to have a camera first. If you don't have an SLR camera, a mobile phone is fine. To get better scan data, I highly strongly recommend a high pixels camera capable of taking pictures in .raw format, it can preserve the information of the original picture to the greatest extent, which is convenient for us to remediate and process in the later stage.

If the pixels are low, the captured model will be very blurry regardless of the ploy count or the resolution of the textures.

To avoid the model with obvious light and shadow information or color deviation, it is recommended to scan in an overcast shooting environment, which can avoid hard shadows and strong reflections on the models or textures. If there is no suitable weather, then try to take pictures in a shady place without direct sunlight.

There are many objects that cannot be scanned, such as translucent objects, highly reflective objects, objects with Albedo that is too black or white, objects that are too thin or too small, self-illuminating objects, etc.

The Scanning Process

Firstly, press the camera shutter as fast as possible to avoid handheld motion blur. Secondly, use a high f-stop to avoid blurry background. Finally, in terms of ISO sensitivity, make sure you have enough light so you don’t have to rely on too much ISO, avoid excessive noise.

Once you get your camera set up, better don’t change those settings because it’ll just throw off the process later if you start zooming in or adding ISO for darker areas.

While capturing, it is better to rotate around your object, once the first loop is complete, you can adjust elevation a little bit, making sure to maintain your distance from the object, then shoot another series of photos all the way around, move a step every 15 degrees. Once complete, do another pass from a slightly higher elevation and so on until you’re nearly shooting straight down onto your object. Actually, the more you capture, the better.

I made a brief conclusion of some precautions during the scanning process:

• Make sure to use the .raw file format.
• Recommended focal length range is 24-35mm, and do not change the focal length during capturing.
• Make sure the shutter speed, ISO sensitivity, and aperture size values are appropriate.
• Better turn off the automatic white balance.
• Try to choose cloudy weather to capture, avoiding direct sunlight and hard shadows.
• When you capture, make sure each capture has an overlap.
• Make sure every photo is in clear focus, and blurry photos can be deleted, they don't help to generate the models.
• The more you capture, the better result, and it’s better to warm up before capture.

Image Processing

Next, I would use Lightroom to process the photos, shot with .raw, we have the ability to tweak and change our image a little bit to help us get not only better models but better texture details. Remove the highlights and shadow information from the photos before generating the textures. Also, you can do a subtle sharpening and a little bit of a noise reduction, just to help get a slightly cleaner image, but don’t overdo it.

Generating the Model with RealityCapture

Import all the processed photos into RealityCapture. By aligning the images and calculating the model, you can get the following results:

Just a bit of a warning, when you use lots of photos, aligning can take a while depending on how beefy your computer is, so if you’re using a low-end PC, it can take a while.

If your project uses Unreal Engine 5, you can directly generate models, UVs, and textures in RealityCapture and export these models directly into the engine. Nanite easily supports millions of polycounts. But keep in mind, very high poly models will take hours to import into UE5.

For this project, I utilized UE4 and used a traditional production process. I made a low poly in 3ds Max, and manually topology to get the low poly of the entire tire, the reason for doing this is better to show the details I want to express. Also, it’s convenient for me to unwrap the UVs later.

Once done, bake the Normal Map and the Diffuse Map in 3ds Max. Of course, using ZBrush's Decimation master is also a good option to get the low poly.

Ultimately, here is what I got:

Rendering

For the final rendering, I chose an existing lighting environment scene from the Unreal Engine Starter Content > Advanced_Lighting. I didn't do any extra lighting because it was good enough for me to achieve the effect that I wanted. Just made a simple curtain in 3ds Max to make the background cleaner:

Place all scanned models together:

In Post Process Volume, I just adjusted the intensity of the AO, the saturation, and the contrast of the overall picture.

And here are the final results:

Conclusion

Photo-scanning is not a very complex technology, and it can speed up our production process. The final model is also very realistic. As a beginner, you can try to scan some relatively simple objects. If you try more, you will definitely get good results. Finally, I also recommend William Faucher's videos to learn more.

Finally, I am very happy to share this project with you and hope it can help you. If you have any questions, you are welcome to communicate with me on ArtStaion. Thanks!

Cui Xiaoyu, Level Artist

Interview conducted by Theodore McKenzie