User:Gazorpo/How to Bake with Blender and xNormal for Skyrim

warning: turns out im bad at words so your probably better off using a different guide

I am a beginner 3D artist trying to do Arcane University's 3D course. However, the course strangely recommends against the method of baking that it teaches, and it does not teach the basic steps and settings for baking with its recommended free tools. Thus, I will try to share what I have learned to help to fill this knowledge gap.

Requirements:

• Blender 3.2+
• xNormal 3.19+

Stage 1: retopology

At this stage, make a low poly model based on your high poly model. I will not give steps for this here. Just two notes:

• You might want to revisit this stage after finding issues with your bake
• I found the snapping and Shrinkwrap tools mentioned here in this video very helpful: Retopology for Beginners in Blender

Stage 2: mark sharp edges

Here, mark edges that cause strange shading between faces at angles more than about 80 degrees, which can otherwise result in baking problems.

1. In Blender, click the Layout workspace tab
2. Of the four Viewport Shading options, select Solid (the second option, which shows basic shading but not materials)
3. Select the low poly object
4. In Object Mode, right click the object, and click Shade Auto Smooth (Blender 3.3+) or Shade Smooth (Blender <3.3)
   • If your model does not have sharp angles, you can try skipping to the next stage after this step
5. In the Navigation Bar (right side of screen by default), click the tab with the upside-down green triangle (Object Data Properties tab)
6. In the Normals section, ensure that Auto Smooth is ticked and set to 180 degrees
   • This lets you see sharp edges affect the smoothing on the object and set all sharp edges manually
7. Change the object interaction mode from Object Mode to Edit Mode
8. Of the three Select Mode options, select Edge (not Vertex or Face)
9. If you see any edges between faces at an angle more than about 80 degrees that have strange shading, select them, right click them, and click Mark Sharp
   • This removes the smooth shading at those edges and hopefully makes the shading more natural and cause a better bake
   • You might need to experiment here and see what the resulting bake looks like to get results that you are happy with

Stage 3: UV unwrap

This stage is for making a layout for the texture, normal map, etc. I will not list all of the steps here, but take note: you need to mark all sharp edges as seams before unwrapping! You can do this with these steps:

1. In Edit Mode and Edge Select Mode, select a sharp edge of your low poly model
2. With the sharp edge selected, select all sharp edges by pressing Shift+G and clicking Sharpness
3. With all sharp edges selected, right click one of them and click Mark Seam

Stage 4: prep high poly model

You only need to do this stage if your high poly model has multiple objects. Here, we merge them into a single high poly model for export to xNormal.

1. Recommended: save your high poly objects before making a merged version, in case you want to change them later
   1. Duplicate the high poly objects by selecting them and pressing Shift+D then Esc
   2. Hide the duplicated objects by clicking the eye icons next to their names in the list of objects (top-right of screen by default)
   3. Use the unhidden objects for the next step
2. Merge the high poly objects:
   1. Apply each modifier in order in the Modifier Properties tab (blue wrench icon): click the down arrow in the top right, and click Apply
   2. In Object Mode, select the objects
   3. Right click one of them, and click Join

Stage 5: prep low poly model

We now prepare the low poly object for export by triangulating faces and applying any modifiers.

1. Recommended: save the low poly, in case you want to use it later without triangulated faces and with modifiers
   1. Duplicate the low poly object by selecting it and pressing Shift+D then Esc
   2. Hide the duplicated object by clicking the eye icon next to its name in the list of objects (top-right of screen by default)
   3. Work with the unhidden object as the low poly object for the remaining steps & stages
2. Triangulate faces:
   1. In Edit Mode with the low poly object selected, change the Select Mode to Face (third option, after Vertex and Edge)
   2. Select all faces: press A
   3. Right click a face and click Triangulate Faces
3. Apply modifiers (examples: Mirror, Shrinkwrap):
   1. Set the object interaction mode to Object Mode
   2. With the low poly object selected, click the Wrench Icon (Modifier Properties) in the Navigation Bar (right side by default)
   3. From top to bottom, for each modifier:
      1. With the low poly object selected, click the down-arrow at the top right of the modifier (left of the X button)
      2. Click Apply from the drop down menu
   4. If you used Mirror and Shrinkwrap modifiers: ensure vertices along the middle of the mirror axis are welded and centred correctly
      • I found that modifier combo had this kind of issue
      • In case it helps, if your keyboard has a num pad, you can centre vertices along an axis like this:
        1. Select the vertices you want to centre
        2. Start scaling them along the axis along which you want to centre them by clicking and holding the correct part of the gizmo
        3. Press 0 on the num pad and unclick
           • This should align them along that axis
        4. In the Sidebar (vertical tabs right side of viewport by default), click the Item tab
        5. Under Transform > Location, set the value of the desired axis to 0

Stage 6: make cage

In this stage, we create a kind of container object called a "cage" that helps baking. If your model is simple and has no sharp angles, then you can try skipping this stage and baking without a cage.

1. In Object Mode, ensure that the low and high poly objects are visible and overlapping
2. Select the low poly object
3. Duplicate it by pressing Shift+D then Esc
4. With only the new object selected, change to Edit Mode
5. In Edit Mode, change to Face Select Mode
6. In Face Select Mode, press A to select all faces
7. Expand the cage:
   1. Press Alt+S to start expanding
   2. Move the mouse to inflate the cage object so that it just barely contains the low and high poly objects entirely, then click to finish

Stage 7: export meshes

Here, we export the low and high poly meshes, and the cage if applicable, so that we can import them into xNormal in the next stage.

Do this for the low poly object, high poly object, and cage separately:

1. In object mode, select the object
2. Click File > Export > "Wavefront (.obj)"
   • Not "Wavefront (.obj) (experimental)"
3. In the export settings of the export window (right side), ensure that Include > Limit to > Selection Only is ticked
   • Otherwise, it will export your whole scene instead of just the selected object, which ruins the bake and can crash Blender
4. Optional: ensure that Geometry > Write Materials in unticked
   • Otherwise, it will export an unneeded material
5. Select an export folder and unique file name (example: "lopoly.obj" for the low poly mesh)
6. Click the Export OBJ button
   • When exporting the high poly mesh, expect Blender to load for a long time and be unresponsive during that time
7. Save your Blender scene

Stage 8: bake

Now, we use xNormal to bake the normal map (and possibly other maps) using our exported meshes.

1. Open xNormal
2. Import high poly mesh:
   1. Click the "High definition meshes" button on the right
   2. If there are any table rows except the empty row and header row: right click each row and click "Remove mesh from list"
   3. Right click the empty row and click "Add meshes"
   4. Select the high poly mesh OBJ file
3. Import low poly mesh and cage:
   1. Click the "Low definition meshes" button on the right
   2. If there are any table rows except the empty row and header row: right click each row and click "Remove mesh from list"
   3. Right click the empty row and click "Add meshes"
   4. Select the low poly mesh OBJ file
   5. If you made a cage:
      1. Right click the filled row, and click "Browse external cage file"
      2. Select the cage OBJ file
      3. Ensure that Use Cage is ticked
4. Set options:
   1. Click the "Baking options" button on the right
   2. Next to Output File, select an output folder and file name
      • xNormal will add "_normals" to the end of the normal map file name before the file extension
   3. Next to Size, choose the number of desired pixels for the width and height of the map
   4. Next to "Edge padding", enter 4
      • See here for an illustration of what edge padding does: https://content.invisioncic.com/Mseclife/image_extract/152096iF87CDC3971417E55.png
   5. Under "Maps to render", ensure that "Normal map" is ticked
      • You might also want to tick "Ambient occlusion" and "Curvature map" for texturing later, but it will take longer
   6. Next to "Normal map", click the three-dot button
   7. In the normal map window, next to Swizzle Coordinates, ensure that the second box has Y- selected (not Y+)
      • The difference between Y+ and Y- is that the green channel of the normal map is inverted
      • Skyrim uses Y- (because it uses software called "DirectX"), but Blender uses Y+ (because it uses software called "OpenGL" instead), so in Blender we will invert the green channel in the material to preview the normal map
   8. Click the Close button of the normal map window
   9. Optional: under antialiasing, choose a higher number for less pixelated edges
      • Higher numbers take much longer, so consider using 1x or 2x first and then 4x later for a final bake when the bake looks good otherwise
   10. Keep other settings at their defaults unless you know what you are doing
5. Click Generate Maps, and wait until it finishes rendering the map(s) on the screen before going to the next stage

Stage 8: preview

Finally, we preview the resulting normal map in Blender (and Skyrim if desired), and then we go back to a previous stage if we notice any issues caused by what we did at that stage.

1. Open the scene in Blender from which you exported the low and high poly objects and cage
2. Hide the high poly object and cage
3. Select the low poly object
4. Optional: make a new material for the normal map, so you do not overwrite any other material
   1. In the Navigation Bar, click the tab with the circular red icon near the bottom (Material Properties)
   2. At the top right of the tab contents, click the + button
   3. If there are any rows beside the button other than the newly created one, remove them: select them and click the - button
5. Click the Shading workspace tab (top of screen by default)
6. In the Shader Editor viewport (bottom right by default):
   1. Add the normal map as an Image Texture node:
      1. Press Shift+A and click Texture > Image Texture
      2. In the new node, click the Open button
      3. Select the normal map file generated by xNormal
      4. In the drop-down menu under the file name, ensure Linear is selected
      5. Next to Color Space, ensure that Non-Color is selected
   2. Invert the green channel by adding additional nodes with the connections and settings in the image below:
      • You can make any of the new nodes by by pressing Shift+A, clicking Search, typing the title at the top of the node in the image above, and clicking the result
      • You can connect nodes by clicking and dragging from one input or output dot to another
        
   3. Connect the Normal Map node output to the Normal input of the Principled BSDF node
7. View the baked normal map in Shading workspace's 3D viewport (top right by default) or the Layout workspace with Material Preview selected out of the four Viewport Shading option buttons (top right of the viewport by default)
8. Optional: tweak the Principled BSDF material settings for a clearer preview
   • I like to just turn Metallic all the way up and leave the other settings at their defaults
9. Optional: preview the low poly model in Skyrim with the normal map by following these instructions