How to Create Realistic 3D Oak Tree

Introduction 

Bonjour! I’m Quentin Rapicault, a French digital gardener – or vegetation artist – currently in my fifth and final year of graduation. My passion lies in crafting all kinds of vegetation, whether realistic or stylized, for both current and next-gen game engines. Each project presents a unique set of challenges, and solving these problems is what excites me the most.

In this article, I’ll take you through my vegetation workflow, the techniques I have developed over the years, and the steps I followed to create a game-ready, highly detailed oak tree. My goal is to provide technical insights and, hopefully, inspire other artists looking to push their foliage creation skills further.

Project Goals

With this project and future ones, my goal was to achieve a beautiful and realistic render that’s still game-ready. I chose not to use a Nanite workflow because I wanted to master this technique first. There are many elements and constraints to consider in the process.

I decided to model an oak tree because it’s one of the most interesting trees to use for practice, and it allows me to learn how to create something that looks natural while also being aesthetically pleasing. I also added some unique details like mushrooms and ivy roots because a tree is never just a tree in a biome there’s always a lot happening around or on it. I couldn’t just create a lonely tree; it would be a bit sad if I did.

Software Used

• SpeedTree (for modeling and procedural generation)
• Maya (for mesh refinement and LODs)
• Substance 3D Designer (for bark and foliage texturing)
• Unreal Engine 5 (for final shading, wind effects, and in-game integration)

I have a specific workflow for vegetation that involves SpeedTree, Substance 3D Designer, and Maya. I begin by creating the basic structure of the tree with placeholder leaves and sometimes hand-drawn atlases to visualize what I’ll need. Then, I focus on adding the necessary details and ensuring the tree looks appealing from multiple angles. After that, I create the texture atlas, either from scratch or using preexisting atlases from Resources. Before applying the textures, I clean them in Substance 3D Designer. Before creating anything, I start the research. Research plays a significant role in my workflow, as nature often offers beautiful inspiration.

Reference Gathering

The reference part is crucial because it’s always best to start with an existing tree shape. When creating my reference board, my goal wasn’t to gather hundreds of pictures of trees, but to find a small number of interesting and consistent images, thinking about how each part could be used for my tree. I also referenced scientific books, particularly for atlases. At my parents' house, there's an oak tree, and it was useful to observe it up close to notice subtle details, like why moss grows in certain areas or why the tree looks darker in some spots.

A tree has a story to tell, and while I took a more practical approach here to hone my skills, I plan to push the narrative aspect further in my future work.

The balance between realism and artistic style for vegetation is tricky. Sometimes, when I walk outside and look at trees, I assess them – but they’re realistic, natural. However, in video games, we don’t want to recreate reality exactly; we want to enhance it. For this tree, my job was like that of a gardener: to make the tree gorgeous rather than just realistic and make my work as invisible as possible.

Modeling

First, I create a blocky shape for my tree to establish its bounds and understand what I’ll need in terms of textures and atlases. I like to draw a schematic of the tree’s structure with color coding before I start; it helps me figure out how many levels of branches I need.

Once I find the general shape I want (a basic trunk with cut branches at the bottom), I balance and tweak them, first by adjusting the node parameters, then using the free mode to move them individually. This is a time-consuming task, but it’s worth it because it gives the tree character and mood.

I then create the branch cards for the leaves. For some trees, this part can be meshes or a combination of mesh and leaves, but for a more fluffy effect, I prefer using cards. The mushrooms were added with the mesh node. I modeled them in Maya with some variations, created five individual mushrooms, unfolded the UVs, textured them, and then assembled them into a mushroom pack to place on the tree. They’re not highly detailed but are good enough for my render target.

For the ivy, I used the mesh decals node to apply it along the trunk. To add more volume, I also placed some leaf cards on it so it looks good from the side or up close (don’t mind the topology of the root, it was a WIP).

After finishing the first version of the tree, I iterated on a second shape with a more massive structure and multiple trunk starts. I prefer this version by far.

I adjusted the card spawn placement and, while working on optimization, I focused on avoiding overdraws. Be sure to be on your player view when you check your tree because it may be seen from a point of view that causes different problems than in the editor view. When using masked textures, overdraw can occur when one texture shows through another, affecting performance, so I try to minimize it. Additionally, I wanted the branches to be visible, not hidden by excessive leaves, so I carefully controlled the card spawn points to get the look I wanted.

Once I finalized the shape and structure, I created variations, like older or younger versions, broken branches, etc. In SpeedTree, I also baked the AO and vertex color to use in the engine.

Texturing

SpeedTree

After completing the basic model, I created the texture atlas using a similar pattern to the one shown in the image. This method is useful because it allows me to cut out entire branches or just medium and small parts, as the leaves don’t overlap the middle sections. I also love the realism of this texture, and it was a great challenge for me to recreate it.

As with my Abies pine post, I used vertex color to create a mask, which I then used in Substance 3В Designer. This saves time and maintains a non-destructive workflow.

I added subtle color details to the leaves. For the base, I started with Quixel resources, but I found them too neutral, so I tweaked some parameters like the baked AO in SpeedTree, gradient variations on each leaf, and added brown color variation.

Substance 3D Designer 

For the trunk, I wanted to make something visually striking with just the base color, so I added noise patterns to create lighter and darker areas, enhancing the height details on the base color. If you want to achieve something unique for the trunk, you can divide the texture into two parts – one for the base of the trunk and the other for the rest of the tree.

My workflow in Substance 3D Designer allows me to iterate quickly and export all the necessary maps. I’ve set up options to extract roughness and subsurface maps from the base color, though I prefer using dedicated textures for those maps.

Integration in Unreal Engine 5

Materials

I’ll explain my vegetation material and trunk material here. I use two master materials: one for the leaves and one for the trunk. In the future, I may merge them and use more material functions with static switches.

For the leaves material, I have options for color tweaking, like gradients or color variation, using vertex color masks. I also packed a mask for subsurface scattering (SS) and applied it only to the leaves to avoid affecting the branches. Additionally, I included roughness and AO maps in the mask pack.

For the trunk material, I have basic PBR settings with control over the AO. I used blue vertex color as a mask to add unique bark textures, like orange spots for the pines. I also included a material function for moss and used the same mask for snow effects, as they apply to the same areas.

To manage wind parameters and seasonal effects, I used a parameter collection, which is useful for adjusting parameters across the entire scene and making them dynamic with a Blueprint for things like a seasonal system.

Lighting

I started with basic neutral lighting to check the normals and avoid rendering issues when iterating on my mesh (it’s important to check in the target engine since SpeedTree’s render isn’t a good reference). Then, I added moody lighting – still clear and visible, but with a bit of vibrancy, like a fall setting or dawn. Something I like to do too is to test my assets in a sample scene in Unreal and see if they look realistic and credible or not. 

I also played with the elements around the tree and the shadows it cast. Shadows from leaves on the trunk or the ground always add character to the render. For beauty shots, I focused on specific parts of the tree, making sure it looked good from all angles but emphasizing the view I wanted to render. I also integrated the tree into a mini-biome because trees are never alone. As a vegetation artist, I don’t just make trees; I think about what’s around them too.

Conclusion

I learned a lot from this project, including optimization tricks, refining my SpeedTree workflow, and improving my engine integration. I also learned the balance between realistic rendering and optimization, such as avoiding overdrawing and unnecessary geometry.

With Nanite now available, I’ll continue to explore creating trees using mesh-only techniques without masked textures. I’ve already started experimenting, and my next project, which will be posted soon, will focus on stylized trees with sculpted and hand-painted textures along with a scene inspired by Zelda-style vegetation!

I hope this article was interesting and helpful. Don’t hesitate to contact me on ArtStation or by email. I look forward to working as a vegetation artist!

Quentin Rapicault, Vegetation/Environment Artist

Interview conducted by Gloria Levine