Robotics - Isaac SIM Image Synthesis for A Custom Object - Rico贾若童的博客

Prepare Object

Convert STL to USD

The path is:

STL
  -> convert to USD
  -> load USD into Isaac stage
  -> attach semantic label: motor_sleeve
  -> create camera + light + table
  -> BasicWriter saves RGB + 2D bbox

If STL is probably in millimeters. So in Isaac, we probably want scale = 0.001. Then,

cp "$HOME/Downloads/J3 motor_sleeve.STL" ~/isaac_projects/assets/motor_sleeve.stl

Image Bounding Box Dataset

# class_id, bbox X, bbox Y, bbox width, bbox height
0 0.498437 0.482292 0.648438 0.966667

We describe distance as object extent: extent = max(max_x - min_x, max_y - min_y, max_z - min_z). so camera distance that’s 2.5x will be 2.5 * largest_object_extent

The basic objects you see are from UsdGeom.mesh, USD/pixar mesh primititive class imported from pxr in Isaac Sim python. It’s a largef 4-vertex quad positioned under the object each frame by randomize_background().

Isaac Sim replicator (SDG, Synthetic Data Generation / perception pipeline). has annotators, writers, sensor simuatliopn, and can randomizing scale, texture, lguiting, and background. We don’t label chairs, tables, etc. and they are just distractors (cuz they are not important to our mission). You can find assets in content browser for

environments/  
 bathroom.usd  
 living_room.usd  
 warehouse.usd  
 lab_table.usd  
distractors/  
 bottle.usd  
 laptop.usd  
 box.usd  
 tools.usd

Synthesize Images using Isaac Sim

Here is a minimal pipeline to synthesize images given the CAD of an object, in the YOLO dataset format

CAD/USD object
  → Isaac Sim headless renderer
  → randomized camera / lighting / material / background / distractors
  → RGB image
  → semantic segmentation mask from Isaac Replicator
  → YOLO bbox from that mask
  → RF-DETR-style dataset folder

Tools being used:

Isaac Sim 6.0 / SimulationApp: This is the main Omniverse/Isaac Sim app in headless mode that we use.
Omniverse Replicator omni.replicator.core: Creates camera, render product, RGB annotator, semantic segmentation annotator. RGB annotater outputs RGB images; semantic segmentation annotator gives per-pixel object / class IDs
- Replicator randomizers: randomize poses, lights, materials, textures, colors, backgrounds
MDL / MaterialX / PBR materials → realistic material definitions: metal, plastic, rubber, wood, concrete, etc.
USD / Pixar pxr Usd, UsdGeom, UsdShade, Gf, Sdf: Creates/edits scene objects, materials, lights, meshes, transforms. USD (universal scene description) is a 3D framework for complex lightweight scenes and rendering, whereas CAD stores exact, highly accurate physical dimensions.
- USD stores meshes, transforms, object hierachy, materials, textures, cameras, lights, joints.. properties of multiple objects in a scene
- CAD stores extrudes / cuts/ fillets, precise dimensions, exact geometry like cylinders instead of meshes
RTX Renderer → turns geometry + lights + materials + camera into RGB images
Isaac Sim semantics utils: Adds semantic class labels to the motor sleeve prim/mesh
Custom extensions / shaders / post-process → underwater, fog, turbidity, sensor noise, blur, distortion
Warp: GPU kernel backend used internally by Replicator; script redirects cache to /tmp

Sample workflow (skipping operations that are less relevant)

# Launches Isaac Sim without a GUI.We geenerate fixed size images. 
simulation_app = SimulationApp(
    launch_config={
        "headless": True,
        "width": 640,
        "height": 480,
        # this is quite photorealistic
        "renderer": "RayTracedLighting",
    }
)

# Create annotator with placeholder render_product (basically output from the camera)
rgb_annotator = rep.annotators.get("rgb")
rgb_annotator.attach(render_product)

sem_annotator = rep.annotators.get(
    "semantic_segmentation",
    init_params={"semanticTypes": ["class"], "colorize": False},
)
# the USD prim has semantic labels ALREADY.
sem_annotator.attach(render_product)

stage = omni.usd.get_context().get_stage()
# Load the object's CAD/USD reference and finds its mesh and bounding box
stage.DefinePrim("/World", "Xform")
sleeve_prim = stage.DefinePrim("/World/MotorSleeve", "Xform")
sleeve_prim.GetReferences().AddReference(USD_PATH)
mesh_prim = find_first_mesh(stage)
# calculate 3D bounding box of the object in world frame
mn, mx, center, extent = compute_bbox(mesh_prim)


# Now the fun begins: add randomizable material to the object
mat_shader = setup_material(stage, mesh_prim)
for i in NUM_FRAME: 
 # randomize diffuse color, roughness
 mat_name, mat_diffuse, mat_roughness, mat_metallic = randomize_material(mat_shader, mat_idx)
 # randmoize dome light, key light, and fill light
 dome_prim, key_prim, fill_prim = add_lights(stage, center, extent)
 # randomize camera pose 
 rep.functional.modify.pose(
     cam_prim,
     position_value=(cam_x, cam_y, cam_z),
     look_at_value=(look_x, look_y, look_z),
 )
 
 # ── Semantic segmentation -> motor_sleeve-only mask ───────────────
 seg_raw = sem_annotator.get_data()
 mask, seg_info, unique_semantic_ids = semantic_mask_for_label(seg_raw, LABEL)

Most important aspect is camera pose randomization. We randomize the camera pose on a hypothetical sphere ( the azimuth - elevation - radius model, a.k.a spherical coordinates).

Here is an example workflow of physics based object settling simulation.

randomize_dome_light - chooses an HDR texture and intensity for the dome light.
randomize_distractors - samples positions, rotations, scales, and display colors for the distractor prims
randomize_pallet - picks one of the pre-created materials and binds it to the pallet.
randomize_camera - samples an orbit position around the pallet and points the camera back at it.
randomize_boxes - writes per-box poses just before the timeline plays so PhysX settles the boxes over NUM_SIMULATION_FRAMES ticks.

Domain Randomization

Synthetic data often suffers from a sim-to-real gap, where models trained in simulation do not perform as well on real-world images. This gap usually comes from two main sources:

Appearance Gap

The appearance gap refers to pixel-level differences between synthetic and real images.

Different object materials, textures, colors, or surface details.
Unrealistic lighting, shadows, reflections, or exposure.
Simplified rendering that does not fully match real camera behavior.

This gap can be reduced by:

Improving rendering fidelity. like adding photorealistic materials, lighting, and camera effects.

Content Gap

The content gap refers to differences in scene composition between synthetic and real-world environments.

Too few object types or scene variations.
Limited object poses, scales, and placements.
Missing background details or environmental context.
Synthetic scenes that are too clean or repetitive.

This gap can be reduced by increasing scene diversity during dataset generation.

Domain randomization intentionally varies simulation parameters so the model learns to handle a wide range of conditions instead of overfitting to one synthetic setup.

Useful parameters to randomize include:

Object appearance: texture, color, material, roughness.
Object pose: position, rotation, scale, and viewpoint.
Lighting: brightness, direction, shadows, reflections.
Camera settings: focal length, noise, blur, exposure.
Backgrounds: floors, walls, clutter, outdoor or indoor scenes.

By exposing the model to many simulated variations, domain randomization helps it generalize better to real-world images and unseen environments.

NVIDIA Isaac Sim Replicator

NVIDIA Isaac Sim includes Replicator, a core Omniverse extension for synthetic data generation and domain randomization.

Replicator supports on-the-fly randomization of scene attributes, such as:

Materials
Textures
Lighting
Camera poses
Object placement
Background elements

Because these changes can be applied without repeatedly reloading assets, Replicator makes large-scale randomized dataset generation more efficient.

Robotics - Isaac SIM Image Synthesis for A Custom Object

Isaac SIM