SynthDrive

The problem

Testing how an autonomous vehicle perceives edge cases usually means capturing those situations in the real world, which is expensive, slow, and in some cases dangerous. The scenarios you most need to test, like a pedestrian at night or heavy glare or an occluded object, are exactly the ones you cannot arrange whenever you want.

What I built

I built a pipeline that chains four AI systems together. GPT-4o takes a one line prompt and expands it into three scenario variants that each target a different perception challenge. World Labs Marble turns those into navigable 3D worlds. YOLOv8 runs object detection on the generated scenes. Then GPT-4o reads the detection results and produces a Perception Difficulty Score from 0 to 100, along with a risk level and recommendations.

Why generate the world instead of filming it

The perception failures worth testing are the rare ones. If you want footage of a pedestrian stepping out in low light with glare on the windshield, you have to wait a long time for it and accept some real risk to get it. Generating the scene instead means the hard case is available whenever you want it and it is reproducible, which is what a test suite actually needs.

Four models, one contract between each

Each stage hands the next one a narrow artifact that I can check on its own. First text scenarios, then world assets, then bounding boxes with confidence values, then a score. Keeping those handoffs small is what makes a four model chain debuggable. When a run looks wrong, it is usually obvious which stage produced the bad output.

Detection as a separate service

YOLOv8 runs in its own FastAPI microservice instead of inside the Next.js app, so the computer vision runtime and the web runtime can scale and deploy separately. The detection server URL is configurable, which is what allowed me to move it onto Render without changing any app code.

Making the output a decision, not a dump

Raw bounding boxes do not tell an engineer whether a scenario is actually dangerous. The final GPT-4o pass turns the detections into one Perception Difficulty Score with a risk level and specific findings, so the result of a test run is something you can act on rather than a pile of output.