You should do piece by piece baking (and combine the result in Photoshop later on).
*Tip - when baking Normal Map in Xnormal - change the background for the nromal map from purple to black, and in AO - from white to red, so its ALOT more easy to select with one click of magic wand later on PS.
One thing to consider, the hole process of normal map baking is combination of topology, uvs, geometry similarity, normal angles, xnormal settings and its all a mess if you have no idea how to do it.
- Good idea is always to check normals before exporting low poly OBJ. Select your model piece - DISPLAY - Polygons - Face normals, to check in which directions normals are going.
- A quick way to flip the normals would be to use "Conform" which will make all of the normals face the same direction and then you can flip them all together. Then you can use average normals to get them to even out over the surface.
- If normal bakes look ugly (messed up) - try to Unlock Normals in Maya and resave lowpoly obj model for re-bake.
Sometimes you can use the "use exported normals" in Xnormal (on low poly setting).
Edit your low poly model in Maya - if you see DARK EDGES after applying Normals - Smooth Edges - correct with -> Vertex Normal Edit Tool (under normals Menu). Export your obj and bake with "use exported normals" / you may also re-bake with "average normals" and compare the results.
There is a set distance calculator you can use in Xnormal. Tools - Ray Distance calculator (unfortunatelly that never works for me - even I played with those - it never gave me the desired results.).
- I DON'T know if thats true, but i noticed on my normal map renders that Scale Mesh 1 / Ray 0.1 = Scale Mesh 5 / Ray 0.5 = Scale Mesh 10 / Ray 1 all those settings give me the same result.
- removing small artifacts on bakes - use 0.1 (smaller number)
- 0.1 - rays casting distance - very close
- 1 - rays casting distance - broad (far)
- also depends on Scale Mesh - for the ray casting - but always use SAME Scale Mesh for Default Mesh Scale and Mesh Scale BOTH on low poly and hi poly, then play with different Ray castings.
Two ways to make a cage
1. Using Xnormal 3D Viewer
2. Using your software program (Maya, Max) to make a cage and export it as additional .obj
(To manually add an external cage in Xnormal - on your low poly - check Use Cage, the under "external cage file" right click - choose "Browse external cage", you may also reset external cage from right click menu, using cage also overrides the Rays settings, even thoug they are shown).
3D VIEWER in XNORMAL:
Export obj. then open Xnormals and load the low res and high rez piece.
Click on the "3D Viewer button." Once inside you will see at the near top left of the viewer - there is a "Show Cage" button which will show the cage for your low res mesh. Once you click that, it brings up a few options of which one is a slider called external cage. You can then use the "expand cage" slider to make it bigger, basically you want it to encompass the model piece entirely. You can also edit small points manually by selecting the "Edit Cage" button and then selecting verts with the middle mouse button and using the x, y, and z sliders. When you are happy with what you have click "Save Mesh" and save it where you want it to and then click yes to have xNormal load the maps for you. Go to the low poly meshes and make sure next to the rays that the "Use Cage" checkbox is selected. Then check your setting and bake.
3D Viewer is great because you dont have to manually create cages in maya and it saves off separate meshes. The only trick you may have to watch out for, is if you ever want to use the cage meshes again, make sure to click "use cage" in the low poly settings in xnormal. This should give you much cleaner bakes!
- AWSD - are the keys to navigate in 3D Viewer. Right click button also rotates around the camera. Its a little annoying to navigate, just use combination of AWSD and right click.
- When you edit the cage - yous elect the verts with MID mouse button. Use sliders for global or X, Y, Z to move the selected vert.
- Why you have to load the high poly (as far as i know) - because the cage HAS to be OVER the high poly (othewise you get artifacts) - so the cage has to completely over -hug the high poly.
- To exit temporalily without closing 3d Viewer - hit Windows button on keyboard (if you hit ESC it will CLOSE the 3d Viewer). To go back in side ALT + ENTER.
- once you click Save Mesh and YES it EXCHANGES the high poly and low poly (and saves that separately in SBM file) so its save, you can always go back to your previous meshes!
- you have to click USE cage, BUT Xnormal does that auto after you exit 3dViewer.
EXTERNAL CAGES done in MAYA:
One thing to consider while doing external cage in Maya - is never to exctrude faces (or duplicate faces) because it changes the vertex ID and the WILL NOT work in Xnormal. TOPOLOGY HAS to be the same as the low poly and it has to be simply - scale it, move a couple of faces/ edges/ vertexes around and thats it. ( in Max there is a push modifier and it just works differently than maya plus you can export various file types).
- GREAT trick to INFLATE in Maya is >>> Select model > hit W for move tool, change setting to "move normal" - grab N handle and drag back and forth along the vert normal to the distance you prefer.
- To Inflate a cage out of original low poly. make a copy of the low poly - select the verts -> Edit Mesh -> Transform Component - drag arrow and it inflates it for you.
In Maya - you can only export the .OBJ so it works in Xnormal.
Another trick i use is ... for very complicated close-together parts - in Maya I selected a couple of faces and duplicated those tough sections - like small piece - re-bake those SEPARATELY in Xnormal and copy pasted the normal map (parts) in PS.
A link to tutorial Zbrush: