SynthGen

The Goal

Build out a synthetic data generation system in UE5 with a focus on visual accuracy and tool design in the context of a greater pipeline.

• Easily usable and installable
• Can be parallelized and batched
• Very high visual accuracy and photorealism

Design

Config -> UE5 Renderer -> Post Processing

The design of the system assumed the user is an ML Engineer. Given this, the system needed to be both configurable and also easily installed and used on a variety of machines. This led me to attempt to design a self-contained system that could be run from the command line while running UE5 headless.

In order for the system to have author-configurable spatial and domain randomization, I opted to put all the necessary information of each scene into a single config. Therefore each config would render out a deterministic scene and always produce the same output. This in turn would allow for easy downstream debugging.

Given this, the sole responsibility of the headless UE5 system would be to act as a renderer. It just needs to read in the config, render out all the declared actors, configure the scene environments, and render out all the configured cameras.

Afterwards there will be a post processing step that organizes the final renders into a system ready to be ingested by ML training pipelines.

Config File

A config is just a JSON file that is a serialized version of the scene. I designed it to have 3 core parts. A header for high-level scene configurations, a list of actors, and a list of cameras.

Here is an example of what the structure of a config file looks like:

header:
  scene_name: "convoy_column_04"
  season: "spring"
  seed: 42
  # ...

actors:
  - type: "Terrain"
    position: [120.0, 0.0, 340.0]
    rotation: [0.0, 45.0, 0.0]
    properties:
        grid_size: 1000
        grid_resolution: 1.0
        # ...
  - type: "BMP2_APC"
    position: [135.0, 0.0, 320.0]
    # ...

cameras:
  - position: [200.0, 800.0, 300.0]
    rotation: [-90.0, 0.0, 0.0]
    fov: 60
    # ...
  - position: [150.0, 1200.0, 350.0]
    # ...

One consideration is how actors are parsed and correctly created. I'll talk more about this in the UE5 system, but there's essentially just a registry of types -> blueprint classes. So for example "Tank" -> "BP_Tank".

Config Generation

A config JSON can be generated by a variety of systems. I opted for various Python recipe scripts. Some examples would be "convoy_column", "forest_patrol", "trench_network" and "occupied_village".

For example, a "forest patrol" recipe at a high level would generate a forest, clear out a section with a road, and add a convoy of vehicles through the clearing. Things like environment, season, lighting and spatial randomization would all be handled by the recipe and made easier with a library attached to the tool.

Bundled with the tool is a library to make writing and constructing these recipes easier. Making it feel more like writing a script for a 3D scene rather than writing a JSON file. The tool comes prebuilt with 5 different recipes but also allows users to write their own.

One design aspect of the JSON config system being the sole source of truth is that this is not the only way to generate configs. For example there could be a visual painting config tool. Potentially even an automated system to ingest a real image and output a recipe to generate similar synthetic data. The possibilities are endless.

Another design aspect of this is the easy configurability of the scene. By doing this outside of Unreal and in Python instead, it allows for much faster iteration times as well as the ease of use for people unfamiliar with Unreal.

UE5 Parser

The UE5 system has a script that gets pointed to a JSON where it reads it in and parses it into an internal representation of the config structure. It uses this to iterate through each of the actors and initializes them. Each actor is of type "ASynthGenBaseActor", where it sets its basic information but more importantly contains an entry point allowing inherited classes to implement their own custom initialization logic from the parsed actor config and the scene header config (for things like season). Extra data is passed through via the properties field. Some examples of child classes would be "SynthGenFoliageActor", "SynthGenMeshVariantActor" and "SynthGenRoadActor".

From there, each type gets registered as a blueprint class parented to one of these SynthGenActors. For example both "BP_Grass" and "BP_Bushes" are parented to "SynthGenFoliageActor", where it configures itself through the BP class.

UE5 Renderer

Nothing too special or groundbreaking here. A handful of notes.

• Certain material graphs had to be modified to support per instance color tints for domain randomization.
• The grass foliage system was pushed to its limits to give the ground a nice texture. Significant time was put into adding FBM Perlin noise and various other visual improvements to make the grass anchor the realism of the scene.
• The config parser was hooked into various environmental systems such as the directional light and fog.
• The segmentation masks were initially rendered through unlit materials, but eventually I had to switch to a crude per actor render approach as the materials never applied consistently in headless mode.
• Seasons were accounted for per actor, where each actor has unique behavior according to the season. For example in the winter, there is almost no grass.
• Trenches are carved into the terrain. But given the terrain resolution, the trenches have to also have their own mesh in order to be jagged. Given the terrain modification, the trenches are loaded first, before the terrain. Currently this is hard-coded but eventually I'd want a more modular ordering system.
• The exposure is configured per config. This is to allow for domain randomization but also to allow for deterministic outputs as auto exposure is not deterministic.
• Renders out the RGB image, depth map, and segmentation layers. But also outputs a metadata file with all the information such as actor segmentation IDs and camera settings.

Post Processing

This step does two major things.

• Degrade the images with post processing effects to match real world aerial imagery. This involved adding in image artifacts, adjusting saturation and contrast, and lowering the resolution.
• Organize and process the data into a COCO format ready to be ingested. This involves using the segmentation masks and depth maps to create bounding box annotations.

This process also has the extra option to output visualization images to debug the bounding boxes to make sure they are correct.

After image degradation with the segmentation mask annotated

Orchestrator

This step ties everything together, allowing you to easily run the full pipeline. Buildable as a self-contained CLI tool, allowing you to easily install and run the system on personal computers or cloud instances. Each step has caching allowing you to adjust various settings and only rerun the necessary steps.

synthgen run forest_patrol --seed 1 --count 10 --editor

synthgen run --all --seed 1 --count 30 --only postprocess --force postprocess -j 16 --stagger 2

Future Improvements

A few core improvements I would make.

• Support animations. Currently the system only supports static scenes. Allow cameras to have paths defined in their configs and allow actors to have simple animations and movement behaviors.
• Add IR sensor support. Create custom materials that either accurately attempt to simulate their thermal properties, or at least create materials to paint in simulated IR signatures.
  • This would be a much bigger technical challenge to get working accurately under all the various lighting conditions and seasons.
• Add in more different scene objects. Things such as SAM sites, flowering meadows, rivers, power lines, etc.
• Add in more foliage variety. Currently the foliage system has one general look. Extending the foliage system to support more variety and biomes would add a lot more visual variety.
• Add in cloud shadows
• Add more environments, such as cities, deserts, beaches, etc.
• Explore various other ways to build config JSONs.

Training

While not originally part of my project spec, I wanted to do a quick test for my own curiosity. I fine-tuned a YOLOv8 model on a dataset of ~3000 images. Here are some results.