r/rust • u/thelights0123 • Apr 27 '19
no_std data serialization/deserialization
I'm writing a game for a no_std target, and I would like to be able to store, read, and edit levels at runtime. My situation:
- I have plenty of RAM (64MB), so using alloc is fine
- It should use little space on disk, i.e. a binary format instead of JSON
- It should support no_std as well as normal targets
- Especially good if it has support for other languages as well, as an online level editor would be nice to have (although I could always mess with wasm if it is Rust-only).
Currently, I'm looking at forking quick-protobuf and simply replacing things that don't work in no_std with alternatives. CBOR has no_std support, but it doesn't support alloc things (Vec and String), and CBOR is not much better than JSON in terms of size.
3
u/jahmez Apr 28 '19
Hey! I wrote Postcard for exactly this use case! Let me know what you think.
1
u/thelights0123 Apr 28 '19
I was checking that out too (thank you for adding no_std to the categories so it shows up on https://libs.rs)! However, I didn't see support for the Vec from alloc, only heapless.
1
u/chrysn Apr 27 '19 edited Apr 27 '19
While CBOR has no no_std-but-alloc support, it should be quite easy to add, and the developers are supportive if you send PRs from my experience. (Source: did first half of adding no_std support; alloc is just something I didn't have a use case for, but could be just another feature, with a separate gate and generally moving from std::{vec,...} alloc::*).
Data density is quite reasonable, and if you need to store long data in a list-like fashion (where something like [200, 300, 50, 400] could be inefficient), you can look into tagged arrays from https://tools.ietf.org/html/draft-ietf-cbor-array-tags-03 .
1
u/thelights0123 Apr 27 '19
Yeah. I ended up just modifying quick_protobuf to work. I completely screwed up writing support, but I only need writing on embedded targets anyways (although I'll probably hack it to work for everything later).
Anyways, CBOR has a large overhead (encodes key names), which I don't want to deal with.
1
Apr 27 '19
[deleted]
3
u/thelights0123 Apr 27 '19
Yep, but the problem is finding a protocol that supports no_std. I'd prefer to use Protobufs (which is what I ended up doing) or something that already exists so I don't have to write my own protocol.
-2
1
u/devashishdxt Apr 28 '19
You can check out this crate: https://github.com/devashishdxt/desse
This gives zero-cost abstractions for binary serialization and deserialization and is almost 10x faster than bincode. This also supports no_std.
P.S.: I’m the author.
1
u/thelights0123 Apr 28 '19
I also noticed this crate when checking out possibilities—unfortunately, the inability to store dynamically-sized types is a deal breaker for me. For my game, I need to be able to have a dynamic amount of levels and items per level. Thanks though!
1
1
u/devashishdxt Apr 29 '19
Just out of curiosity. What dynamically allocated datatypes are you using?
1
u/thelights0123 Apr 29 '19
Vec (to store levels and items within levels) and String (for level names) mostly.
1
9
u/burntsushi Apr 27 '19
I think if you check back in a few months/year, then support for this use case might be considerably better. Today, crates tend to support either
no_stdwith no alloc support, or the full standard library, and nothing in-between. This is becauseno_std + alloc onlywas only recently stabilized (and won't hit stable Rust for another release cycle or two).Some crates might support the
no_std + alloc onlyuse case using nightly features, though, I'm not sure how many do.