GETEX: Intra-Class Geometric Variation via 2D-to-3D Generative Models for Reducing the Synthetic-to-Real Gap

Patrick D. Vibild, Christian Mai, Miguel Ozalla Sánchez, Jesper Liniger, Simon Pedersen AAU Energy, Aalborg University, Esbjerg, Denmark · IROS 2026

This repository contains the code, configuration files and input photos for our paper, plus a download of the GETEX (GEometric & TEXture) real-world test set.

Target objects are photographed, and a 2D-to-3D generative model (SAM 3D) turns each photo into a different 3D mesh of the same object. Several meshes per class introduce intra-class geometric variation into domain-randomized synthetic training data. With generative geometry alone (GEN-PBR), detection matches a CAD-based pipeline (CAD-PBR). Combining CAD and generative geometry (COMB) improves mAP50 by 4.2% over CAD-PBR on GETEX.

Important

This code is built on SynMfg_Code by Zhu et al. (ICRA 2025). The domain-randomization and rendering pipeline (scene composition, distractors, backgrounds, lighting, camera, post-processing, Blender rendering and YOLO annotation) is the work of Xiaomeng Zhu, Jacob Henningsson and co-authors, released at https://github.com/jacobhenningsson95/SynMfg_Code under GPL-3.0. We kept their framework and changed only how objects are imported and textured (see What we modified). If you use this code, please also cite their paper (citation below).

Contents

Path	What it is	Origin
generation_main.py	Starts the parallel Blender generation from a config file	SynMfg_Code, unchanged
Blender/blender_run.py	Per-instance Blender scene randomization and rendering	SynMfg_Code, modified
Blender/opencv_check.py	Checks OpenCV inside Blender	SynMfg_Code, unchanged
config-sample.json, data/Objects/README.md, docs/SynMfg_README.md	Original sample config and docs	SynMfg_Code (docs: short GETEX notes added)
configs/	The configs used to generate our training sets (CAD-PBR, GEN-PBR, GEN-HSV)	GETEX
scripts/generate_variants.sh	Renders one dataset part per mesh variant (5 × 1,200 = 6,000 images)	GETEX
scripts/merge_split_yolo.sh	Merges parts into one YOLO dataset with an 80/20 split	GETEX
tools/prepare_2d3d_meshes.py	Renames and rescales raw 2D-to-3D GLB outputs to class labels	GETEX
Blender/convert_glb_to_obj.py, Blender/glb_dir_to_obj.py, Blender/decimate_glb_dir.py	GLB → OBJ conversion and mesh decimation	GETEX
images/2d_bases/	The 126 input photos used to generate the 3D meshes	GETEX
Release v1.0 → GETEX_test.zip	GETEX real-world test set (475 images, 1,955 instances)	GETEX

The git history starts with an unmodified import of SynMfg_Code (commit 3a0ed92), so git diff <first commit> HEAD -- Blender/blender_run.py shows exactly what we changed.

What we modified in SynMfg_Code

Upstream SynMfg_Code imports only .obj files. After import it UV-unwraps each object and replaces its material, so the pipeline can apply random RGB, image or PBR textures. That works for CAD models, but it throws away the texture that 2D-to-3D models bake into their meshes. Our changes to Blender/blender_run.py are:

1. OBJ + GLB/GLTF import. models_path may contain .obj, .glb and .gltf files, and the file type is tracked for every object and its duplicates. GLB imports may contain empties, so only mesh objects are kept.
2. Keep the embedded texture of GLB objects. GLB/GLTF objects are not UV-unwrapped. Their imported materials are never overwritten, never cleared between scenes and never replaced by the random texture step. OBJ objects keep the upstream behaviour (random PBR material etc.). The same config can mix both.
3. HSV colour randomization of the native texture (GEN-HSV). When randomize_glb_hsv is true, a Hue/Saturation/Value node is inserted in front of the Principled BSDF Base Color of each GLB material and resampled for every scene: hue rotation U(0, 1) (0.5 = unchanged, i.e. a full ±180° shift), saturation × U(0, 2), value × U(0.2, 3), at full blend strength (Eq. 1 in the paper).
4. Rotation fix for GLB. GLB objects import with quaternion rotation, so they are switched to Euler XYZ. Otherwise the random object rotations would have no effect.
5. Environment. The active conda env's site-packages are added to Blender's Python so cv2 can be imported. environment.yml was updated (Python 3.10, conda-forge, pygltflib).

The rest of the DR framework (scene composition, camera pose, illumination, post-processing, labels) is unchanged. New config keys (all optional):

Key	Default	Meaning
randomize_glb_hsv	false	Apply random HSV to the embedded texture of GLB/GLTF objects (GEN-HSV)
glb_hsv_hue_min / glb_hsv_hue_max	0.0 / 1.0	Hue range of the HSV node (0.5 = no shift)
glb_hsv_sat_min / glb_hsv_sat_max	0.0 / 2.0	Saturation multiplier range
glb_hsv_val_min / glb_hsv_val_max	0.2 / 3.0	Value multiplier range

All other parameters are documented in the original SynMfg README.

Setup

1. Python environment: conda env create -f environment.yml && conda activate getex
2. Blender 3.4: download it from https://download.blender.org/release/Blender3.4/ and set export BLENDER_PATH=/path/to/blender-3.4.1-linux-x64/blender
3. Textures and backgrounds: follow the SynMfg instructions to fill data/Background_Images, data/Texture_Images and data/PBR_Textures (blenderproc download cc_textures data/PBR_Textures).

Pipeline

1. From photos to meshes

images/2d_bases/ holds the input photos, named <object>_<variant>.jpg. There are 14 physical objects and 9 photos of each. Variants 1–5 were used for the main results; 6–9 were added for the N7/N9 mesh-count ablation. Two physical cows (cow1, cow2) share the class Cow, and three physical monkeys (monkey1–monkey3) share the class Monkey:

Photo prefix	Class		Photo prefix	Class
cow1, cow2	Cow		monkey1, monkey2, monkey3	Monkey
david	David		sheep	Sheep
disco	Disco		tiger	Tiger
goat	Goat		venus	Venus
horse	Horse		winged	Winged
moai	Moai

The target object is segmented in each photo, and the masked image is passed to SAM 3D. For the backbone ablation we used TRELLIS, Hunyuan3D and Hi3DGen instead. Each photo gives one textured GLB, so there are several different meshes per object. Keep the file name (cow1_3.jpg → cow1_3.glb).

The pipeline requires every mesh object to be named after its class label (see data/Objects/README.md). All meshes of the same class also need a consistent scale.

SAM 3D meshes (main results).

• GEN-HSV uses the GLBs directly. Put them in data/Objects/GETEX_SAM3D_GLB with their mesh objects renamed to the class label.
• GEN-PBR uses OBJ copies. The OBJs have no material, so the pipeline assigns random PBR materials. The converter takes each class label from the file name:

$BLENDER_PATH --background --python Blender/convert_glb_to_obj.py -- data/Objects/GETEX_SAM3D_GLB data/Objects/GETEX_SAM3D_OBJ

Other backbones (ablation, Table IX). Rename the raw outputs and rescale them to the SAM 3D sizes. Next, decimate them, because they are far denser than the SAM 3D meshes. Finally, convert them to OBJ:

python tools/prepare_2d3d_meshes.py meshes/TRELLIS data/Objects/GETEX_TRELLIS_GLB --reference-dir data/Objects/GETEX_SAM3D_GLB
$BLENDER_PATH --background --python Blender/decimate_glb_dir.py -- data/Objects/GETEX_TRELLIS_GLB data/Objects/GETEX_TRELLIS_DEC
$BLENDER_PATH --background --python Blender/glb_dir_to_obj.py -- data/Objects/GETEX_TRELLIS_DEC data/Objects/GETEX_TRELLIS_OBJ

2. Generate the synthetic training sets

Every regime has 6,000 images at 720 × 720. The configs in configs/ are the ones we used; only the input and output paths were renamed.

Regime	Meshes	Texture	Command
CAD-PBR	Ground-truth CAD / 3D-scan OBJs	Random PBR	python generation_main.py --config configs/cad_pbr.json
GEN-PBR	5 SAM 3D meshes per object (OBJ)	Random PBR	BASE_CONFIG=configs/gen_pbr.json MESH_DIR=data/Objects/GETEX_SAM3D_OBJ EXT=obj OUT_PREFIX=Render_GEN-PBR scripts/generate_variants.sh
GEN-HSV	5 SAM 3D meshes per object (GLB)	Native texture + random HSV	BASE_CONFIG=configs/gen_hsv.json MESH_DIR=data/Objects/GETEX_SAM3D_GLB EXT=glb OUT_PREFIX=Render_GEN-HSV scripts/generate_variants.sh
COMB	–	–	Equal 2,000-image subsets of CAD-PBR, GEN-PBR and GEN-HSV
GEN-COMB	–	–	Equal 3,000-image subsets of GEN-PBR and GEN-HSV

generate_variants.sh renders 1,200 images per mesh variant (VARIANTS="1 2 3 4 5"). For the mesh-count ablation, change VARIANTS and SAMPLES_PER_VARIANT so the total stays at 6,000 (e.g. VARIANTS="1" SAMPLES_PER_VARIANT=6000). Merge the parts into a YOLO dataset:

PREFIX=Render_GEN-PBR PARTS="1 2 3 4 5" OUT_DIR=datasets/GEN-PBR-6000 scripts/merge_split_yolo.sh

3. Train and evaluate

We used YOLOv8x from Ultralytics 8.4.7:

yolo detect train model=yolov8x.pt data=datasets/GEN-PBR-6000/data.yaml imgsz=720 epochs=500 batch=16 \
  optimizer=SGD patience=100
yolo detect val model=runs/detect/train/weights/best.pt data=GETEX_test/data.yaml split=test \
  imgsz=640 conf=0.05 iou=0.40 augment=True

GETEX test set

Download GETEX_test.zip from the v1.0 release. It has 475 real photos with 1,955 hand-annotated bounding boxes (YOLO format) over 11 classes. The classes form two groups:

• Geometric-dominant: 3D-printed statues (David, Disco, Moai, Venus, Winged)
• Texture-dominant: toy animals (Cow, Goat, Horse, Monkey, Sheep, Tiger)

Group	Subset	Images
Simple	A – single target	110
	B – single class + distractors	101
Complexity	C – non-uniform texture	64
	D – multi-class, low occlusion	77
	E – high occlusion	59
Context	F – complex (printed) backgrounds	64
Total		475

GETEX_test/data.yaml evaluates all subsets together, and each subset folder has its own data.yaml.

Citation

If you use this code or the GETEX dataset, please cite our paper and the SynMfg_Code paper it builds on:

@inproceedings{Vibild2026GETEX,
  author    = {Vibild, Patrick D. and Mai, Christian and Ozalla S{\'a}nchez, Miguel and Liniger, Jesper and Pedersen, Simon},
  title     = {Intra-Class Geometric Variation via 2D-to-3D Generative Models for Reducing the Synthetic-to-Real Gap},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2026}
}

@inproceedings{Zhu2025ICRA,
  author    = {Zhu, Xiaomeng and Henningsson, Jacob and Li, Duruo and M{\aa}rtensson, P{\aa}r and Hanson, Lars and Bj{\"o}rkman, M{\aa}rten and Maki, Atsuto},
  title     = {Domain Randomization for Object Detection in Manufacturing Applications Using Synthetic Data: A Comprehensive Study},
  booktitle = {2025 IEEE International Conference on Robotics and Automation (ICRA)},
  year      = {2025},
  pages     = {16715--16721},
  doi       = {10.1109/ICRA55743.2025.11128647}
}

License

The code is released under the GNU General Public License v3.0, as required by SynMfg_Code (Copyright (C) 2025 Jacob Henningsson, Xiaomeng Zhu). Our modifications are Copyright (C) 2026 Patrick D. Vibild and are also GPL-3.0. The photos in images/2d_bases/ and the GETEX test set are released under CC BY 4.0.

Acknowledgements

We thank Xiaomeng Zhu, Jacob Henningsson and co-authors for releasing SynMfg_Code, which this work builds on. This work was supported by Innovation Fund Denmark through the U-SHIELD project (Grant No. 4354-00018B).