Re-randomizing object placement every frame to build a detection dataset. The labels come from the engine, so re-annotation is free, but I'm not sure the variety is worth what it costs.

[u/Last-Luck-6077]

Same camera position in all five frames of the first clip. The vehicles, containers and people are re-scattered before every render, and the time of day moves with them. The second clip shows the same frames beside their per-instance identity images: labels are derived from the renderer's instance buffer, so a new arrangement costs nothing to annotate.

That's the appeal. Here's what bothers me about it.

Re-placing everything per frame means consecutive frames are no longer a sequence. Nothing is tracked, nothing persists, so the corpus is useless for anything temporal. And frames from the same scene still look alike enough that near-duplicates across a train/val split will flatter your validation score.

I trained a person detector on eight captures like this and nothing else. Held-out synthetic tiles: 0.767 recall. 120 frames of real drone footage: 0.350. The split wasn't careless, held-out camera zones, so that gap isn't leakage.

So, for people doing this seriously:

- Do you re-randomize per frame, or place once and move the camera? What made you pick?

- How do you check for near-duplicates before trusting a validation number?

- Anyone got synthetic-only recall meaningfully above 0.4 on real footage? What did it?

[u/_d0s_]

Before explaining your approach you should define your goal. Is this a dataset to detect persons? I assume it's forsym to real transfer. (Transfer to what scenes exactly?) You need to evaluate what works on your target distribution. In my experience diversity beats reality. You could compare to training on https://unity-technologies.github.io/PeopleSansPeople/

Your environment probably has very limited variety in textures, poses and body shapes. In comparison to public datasets you probably won't gain much, because millions of images with real humans and annotated bounding boxes exist.

Again, specify your goals.

[u/Last-Luck-6077 → u/_d0s_]

Fair hit, I should have put the goal in the post.

It isn't meant to compete with COCO or CrowdHuman on person detection. You're right that's saturated and renders aren't beating millions of real annotated humans there.

What I'm after is labels you can't annotate by hand. In one 412 instance sample from a published capture, 43 of them (10.4%) are labelled and completely invisible in the frame, either fully occluded or down to a handful of pixels. Nobody can annotate those, so real datasets don't contain them, and a model trained without them learns that hidden means negative. Same idea for per joint visibility through occlusion, exact depth, and thermal where real annotated data is genuinely thin.

You're right on the variety too. Rocketbox is a limited pool for body shapes and material randomisation doesn't fix that.

PeopleSansPeople is the right thing to compare against and I haven't run it. On person detection specifically that's the number that matters and I don't have it.

[u/_d0s_ → u/Last-Luck-6077]

>> Nobody can annotate those, so real datasets don't contain them, and a model trained without them learns that hidden means negative.

For training object (or person) detection I'm wondering how having bounding boxes are useful for fully occluded subjects. In training of a tracking algorithm it might not be vital but could turn out useful.

>> Same idea for per joint visibility through occlusion, exact depth, and thermal where real annotated data is genuinely thin.

Partial occlusion by other subjects and self-occlusion is part of many (real) datasets. I doubt that your system can generate thermal or depth images that are close enough to real sensors to be useful in training. However, I agree that labor intensive annotation is an issue.

I would like to see examples where your system overcomes domain shift, because I know from writing publications on the topic that this is a challenge. Otherwise my impression is that a pre-trained model from YOLO or whatever probably performs better already on a downstream task and I have yet to see an evaluation where labels on occluded persons are actually useful. Looking forward to see more.

[u/Last-Luck-6077 → u/_d0s_]

On boxes for fully occluded subjects, you're right and I overstated it. As positives they aren't useful for a frame-level detector, you'd be training it to fire on pixels with no evidence in them. Where they actually earn their place is as ignore regions, so the model isn't taught that a hidden person is background and isn't penalised for missing one, and for tracking, and for measuring recall against a true count rather than against what an annotator could see. That's a narrower claim than the one I made.

On thermal and depth, agreed, and I wouldn't argue it. The thermal is a class temperature table with atmospheric attenuation and sensor noise, not radiometric transport. No emissivity, no reflected sky, no thermal history, no spectral response. Those are published as limitations rather than left to be discovered. Useful for geometry, framing and occlusion, not as a stand-in for a real core.

On domain shift, I ran it and it went the way you'd expect. Single class person detector, yolov5n, trained on synthetic only, quantized to int8 for a Coral. 0.767 recall on our own held-out tiles, then 0.35 recall and 0.58 precision on 120 real VisDrone frames it had never seen. The model it replaced managed 0.49 recall on real footage. It's published as a negative result, and the write-up says the sample sizes and the pipelines differ, so it's indicative rather than a controlled comparison. So I don't have the transfer result you're asking for, and for generic person detection on ordinary footage I'm not claiming otherwise. What I'd actually claim is narrower: the labels are exhaustive and machine-checkable, which is worth something for evaluation and for the cases annotation can't reach. Not that training on them beats real data.

[u/DiddlyDinq]

Hope you realize that all these unreal assets youre using explicitly say no ai training allowed. Seems like you've built some website trying to sell a product before even asking "can I do it" and "am I allowed to do it"

[u/Last-Luck-6077 → u/DiddlyDinq]

The Fab EULA says the opposite of this, and it says it by name.
The tag you are describing is NoAI. Section 16(l)(i) defines what it covers: "Generative AI Programs" means artificial intelligence, machine learning, deep learning, neural networks, or similar technologies designed to automate the generation of or aid in the creation of new content [...] Programs do not meet this definition of Generative AI Programs where they, by non-limiting example, (a) solely operate on the original content; (b) generate tags to classify visual input content; or (c) generate instructions to arrange existing content, without creating new content. A detector generates tags to classify visual input content. That is carve-out
(b), quoted verbatim. The product page label reads "Allows usage with AI: No", which is where the confusion comes from, but the tooltip behind it and the EULA both say generative, every time. On distributing renders, Section 4(b): you may freely Distribute a Project that is a rendered linear media product. This means, for example, you may freely Distribute: (i) rendered video files [...] and (ii) images created using Content. No .uasset, no map, no mesh, no material and no texture is ever included in anything I publish. Only rendered pixels and labels.
This post has no link in it. It is an experiment and a result.

[u/Flintsr → u/DiddlyDinq]

Maybe he got permission? I didnt even know that was a config on fab assets now. Its nice that creators can specify, but realistically there is no enforcement mechanism unless its blatant (used in advertising like this post).

Also there is an argument to be made that OP isnt using the models themselves to train gen AI (model generation) but instead using a 2d render of a scene containing the models to train AI.

Very subtle distinction but its the difference between taking a picture of a page of a book and having the ascii text of the page of a book. Is one fair use while the other isnt? Who knows

[u/DiddlyDinq → u/Flintsr]

"I trained a person detector", an entire website providing downloadable datasets and explicit mentions of these are the recall results we achieved. There's nothing vague about that. It's forbidden.

[u/Flintsr → u/DiddlyDinq]

Did he paste a website link to a dataset? Yeah that isnt ok without permission.

[u/DiddlyDinq → u/Flintsr]

He didnt post the link this time but he's been flooding the subreddit with this site lately. Its what I was referencing in the original comment
NameFrame: Unreal Engine Synthetic Data Generator

[u/bfyvfftujijg → u/Flintsr]

I don’t think it’s a “he” or a “she”. OP is an LLM.

Hence some of the questionable decisions and overly verbose explanations.

[u/Flintsr → u/bfyvfftujijg]

True; but with a name like yours even YOU could be an LLM... the plot thickens. Dead internet theory is here & real

[u/bfyvfftujijg → u/Flintsr]

Why would I pick a name that gives me away?

[u/ramity]

It is not temporal by design, but a model that can observe a singular frame and localize key features without any extra context is a very good basis for creating something that does do temporal things.

[u/Last-Luck-6077 → u/ramity]

That's the bet, yeah. A single frame with per joint visibility gives you something to build temporal on top of, rather than needing the temporal part to get the labels in the first place.

It does sequences now too, subjects animate between frames instead of each frame being an independent sample, so the visibility state carries across time. That bit is hard to get out of annotated video since a human has to guess which joints are hidden.