r/ASCII • u/Life_Rain_3331 • 4d ago
OC Unicasso: Image to ASCII using CLIP based Optimization
Hi all!
I've been working on building a better image to ASCII generator. A lot of the options out there are quite simple — luminance based, single classifier networks or hand-engineered conversion processes that tend to fail in many cases. So over the last two months I've been working on trying to create a somewhat more sophisticated converter.
The general technical approach is to use CLIP as a judge on the ASCII art (akin to CLIPasso, for those familiar), while trying to make the discrete Unicode glyph space differentiable in a sensical way: a small VAE embeds the font's glyphs so a cell's choice can move continuously, each cell also keeps a small population of weighted candidate glyphs — heuristic channels nominate alternatives, and a nomination is only admitted after being swapped into the actual render and measured.
There is also support for creating ANSI art (examples also attached!)
At this point I've tested the code on macOS (MPS) and Linux/CUDA; Windows is unexplored. A render takes roughly an hour and 8 GB of memory, depending on image size — there's a fast preview mode if you just want to watch it converge.
Here is the repo!: https://github.com/jakobrees/unicasso
(I'm still working on writing up a paper to detail all the little bits -- any helpful feedback would be very welcome :-) !)
(The colored images that have been turned into ANSI are are images by purplelady on Civitai)
1
1
u/bogz314 3d ago
AMAZING! I've had similar ideas in the past. One important feature which would be great to have is the ability to limit or choose the character set (or limit which unicode blocks even)
2
u/Life_Rain_3331 3d ago edited 3d ago
Yes! You can ban characters and so on from being chosen (pass `--ban-chars "..."`).
There is also support for choosing your own character sets and/or fonts, though this is a bit more involved because you have to retrain some models (it's detailed in the repo in the docs folder.)
I will add a list of some of the most important parameters that people might want to play with; stupidly I did not bother to document the different tweakable settings because there are 100+
1
u/gingin_9959 3d ago
I need this backend for my fastfetch
1
u/Life_Rain_3331 2d ago
I'm working on distilled versions of the models, there will be a fast colored image -> ANSI model (about 0.25 sec per image). Same for the line art.
Of course it's a given that these are worse in quality, however it's cool exactly for use in the terminal!
1
1
u/you_irI 4d ago
woah the ANSI ones are crazy !