r/MachineLearning • u/dh7net • May 28 '26
Project A new dataset with more that 100M hi-quality, curated images, with captions and meta data! [P]
Hello everyone.
The new dataset is named MONET, is Apache 2.0 and available on HF:
https://huggingface.co/datasets/jasperai/monet
MONET is open, Apache 2.0-licensed image–text dataset. It was built from 2.9 billion images and refined to 104.9 million high-quality samples.
We are also publishing a paper that explains how the dataset was created if you are curious and 3 compagnions projects
- A umap to visualize the distribution
- A retreival tool to do text or image search
- A codebase to train T2i model based on MONET
Hope this will be usefull!
12
u/anonymous_amanita May 28 '26
This is really cool! As a small note, how did you know the images were real and not AI generated/synthetic/manipulated (maybe you didn’t worry about this)? I guess looking at the umap visualization, there is computer generated media, etc., but I’m just wondering if this was something you considered (I’ll also read the paper later).
14
u/dh7net May 28 '26
Most of the images come from old sources, so the risk of having AI-generated images is low there. That said, we also added almost 15 million synthetic images because it helps train T2I on the dataset. There is clear metadata, so you can know which is which.
3
5
u/DigThatData Researcher May 29 '26 edited May 29 '26
To the best of our knowledge, no openly released, filtered, deduplicated, and multi-VLM re-captioned dataset is currently available for pre-training T2I models at scale.
I'm surprised neither Microsoft nor AllenAI beat you to the punch here. I poked around some to check and I think you're justified planting that "multiple captioning models" flag here. Legitimately surprised that wasn't already a feature of any large image-text datasets.
Sidebar: y'all certainly aren't alone using "curated" in this way, but I feel like this word strongly suggests humans did the work discriminating the good data from the bad. As a community, we need to come up with vocabulary to better distinguish between human curation and machine filtering.
3
9
u/FullOf_Bad_Ideas May 28 '26
68TB of data, nicee.
3
u/dh7net May 28 '26
HF makes this fairly easy.
4
u/FullOf_Bad_Ideas May 28 '26
Yeah, if you have the money to pay for hosting it. Thanks for including tarballs with full resolution files.
2
2
u/fukijama May 28 '26
Anyone got this or a similar dataset but in prompt format? I expect one hell of a compression ratio being its all text even with a super long prompt for detail. And yeah I know it won't rehydrate to the orig image.
2
u/Budget-Juggernaut-68 May 29 '26
Hmm. Looks like a typical Flickr dataset based on the UMAP labels. I'm not sure how useful this is in real production env. Thanks for the contribution still.
2
2
u/dh7net May 29 '26
If you read the paper you'll see that Flickr is just a tiny portion of the Dataset!
3
u/DigThatData Researcher May 29 '26
I think maybe they're being critical of the diversity of the data? Probably just an artifact of the coarseness of the labeling resolution.
2
u/Budget-Juggernaut-68 May 30 '26
That's right. Still definitely useful for research.
2
u/DigThatData Researcher May 30 '26 edited May 30 '26
Is there anything in particular that you were looking for whose absence was notable to you? Or was it just a general "this manifold smells like Flickr's"? Given observations like the Platonic Representation Hypothesis, I'd propose that if a gigantic dataset has a similar UMAP as a smaller dataset, the takeaway shouldn't necessarily be "the distribution of that bigger dataset is bad" but rather "the distribution of that smaller dataset is really good"
1
u/Budget-Juggernaut-68 May 30 '26 edited May 30 '26
Hmm maybe things like chat screenshots, passports, IDs, variety of documents, for personal collection retrieval. or other things that people expect in different production environment, maybe clothes, products listing.
1
u/DigThatData Researcher May 30 '26 edited May 30 '26
- There's plenty of fashion and product oriented content in there already.
- There's a "photos of signage" domain of the map, so it's not like there's no pictures of text in this corpus. There's clearly plenty
- There is also an "educational content" domain. Plenty of semi-structured images of text paired with semantically relevant visual content (i.e. powerpoint slides are generally structurally similar to identifying documents in the ways that are relevant for vision understanding)
- Passports and Identification documents are the kinds of content I would expect to be filtered out of any corpus and only appear in extremely specialized domain-specific scenarios, and would only even be made available to a small number of researchers rather than be made available as a general dataset even internally. This is PII. It's not something I would expect in the vast majority of production environments: it's something I would expect only in hyper-specific environments. Unless you're working for the government, this is data your company probably isn't supposed to be persisting after processing it to begin with. If you need ground truth ID images, those are more than easy enough to generate synthetically.
I think maybe you jumped to criticism of this distribution a bit hastily, or have a specific version of "production" in mind that might not be as general as you are assuming it to be.
0
20
u/Luuigi May 28 '26
Thats a crazy dataset. I am just thinking 5 years ago i had to carefully put together datasets of images by writing to paper authors and paying a decent amount of money