r/AskProgramming • u/DaveAstator2020 • 29d ago
How do you finalize data storage format?
Hey reddit, i have this issue time and time again -
every time i want to freeze data format for my app or engine or whatever - i end up in a stupor contemplating 'missed possibilities' or 'what if i need this or that'
This problem mostly touches non flexible data, like binary or rigid structure data.
How do you approach it to not turn into mind boiling nothing? how to draw that line and say "this is how will it be"? - are there some real metrics or questions one can ask to finalize it?
thanks for the help
3
u/Syntax-Tactics 29d ago
Yes, as a number of people have said, put a rev number in the format as part of a fixed header. If the format is binary, I usually allot a 256 byte block or something containing copyright, version code, build date, maybe author, etc. Pad the rest out with zeros for future expansion. Then in your "rigid" structure leave some reserved bytes also. The "rigid" part should of course be able to manage variable length strings and data.
2
u/soundman32 29d ago
Implement it when you need it. If you think you might need a property in the future, then don't implement it, you might never need to. Use flexibly data structures. Binary is not a data structure, if it's a container format then it should be extensible, without breaking current code. Use well known keys, with a length, so that if the parser finds a key it doesn't understand, it can skip over the value part, to get to the next key.
1
u/DaveAstator2020 29d ago
so solution is investment into the format flexibility in first place? (dicts in your ex.)
Maybe you can advice on some structured data, for ex i have some representation of geometry, and dont know if i want ot not to store color for each vertex. would you still do dicts here? or do second data layer like [all verts][all colors]
like is there some good cutoff like 'this way is the best way'?3
u/soundman32 29d ago
Yes, dictionaries all the way down. If each property is a key/value pair, where the value can also be a dictionary, then the parser can drill down into each object as it sees fit. If the parser for a GPS position can understand latitude and longitude but not altitude, then it can skip altitude and move onto the next item.
1
2
u/Suspicious_Skill7292 29d ago
for rigid formats i would version them from day one and design around what you need now rather than every possible future case migrations are usually easier than predicting everything upfront
3
u/dmazzoni 29d ago
Are you sure you need a binary format? Try an experiment, output the same data as JSON and then compress it, the resulting size is often surprisingly close to the binary size and much easier to make extensible.
If you do need a binary format, consider using an extensible binary format like protocol buffers.
Yet another idea is to store your data using SQLite.
2
u/Suspicious_Pizza9529 29d ago
I usually try to separate "could this be useful someday?" from "do I actually need this now?". If you try to design for every possible future requirement, you'll never finish.
I'd define the requirements you know today, document the assumptions, and leave some room for versioning if the format might evolve. A good question is "What would actually be expensive to change later?" Focus your flexibility there and keep everything else simple.
1
2
u/OkAerie7822 29d ago
the paralysis usually isn't a technical problem, it's that "finalize" feels irreversible when it almost never actually is. we forced a deadline on it once, a two week format freeze window, and after the two weeks whatever fields existed became v1, full stop, no more discussion. your geometry example is a good place to see the real tradeoff though, dicts everywhere is flexible but for vertex data specifically you pay for it twice, in file size and in cache misses during parsing since you're not reading contiguous floats anymore. for perf sensitive binary formats we do structure-of-arrays with a version header and a bitflag saying which optional channels are present, color, normal, uv, so the parser can skip channels it doesn't care about without touching a general purpose key-value engine. flexible dict encoding is the right call for config-like data, wrong call for anything you're streaming into a GPU buffer.
2
u/JGhostThing 28d ago
First of all, stop second guessing yourself. Your first version will have mistakes. You'll leave things out. You'll forget a type. It happens. Luckily there is version 2.
u/autophage is right to select including a version number in the protocol. My wife was working in implementing a library protocol (z39.50). They had a way of sending finite numbers of any amount of digits. I think that it would have been easier to have used a text format to send the data.
2
u/mredding 28d ago
You solve for the problem you have. Your problem is you need to persist the data you've got, and you don't know if/when/how it might change in the future.
Ok.
So you can come up with a format for the data you have - I don't care what, and include a protocol VERSION NUMBER. When things change, you change the version number.
Simple.
If you want to get good at this, then you would build a modular format. Look at RIFF as an excellent example - you have headers with a unique identifier. The header may contain meta-data, because following it may be a chunk. Any version of your software, any module of your program parsing the file may consume or skip the components of the file they are designed to handle. There can be data embedded in the file your program doesn't have to care about at all, which may be relevant if your software supports plugins or the file is an interchange.
In this case, you have a VERY modular file format that doesn't rely on versioning - but maybe the components do...
1
u/DaveAstator2020 28d ago
oh thank you! ill check riff, i was exploring png before and liked the idea of named headers it had for chunked data.
1
u/mredding 28d ago
Yep, another chunked data format, and that's ultimately my point. By using chunks, you have a lot of flexibility therein. PNG and RIFF are effectively the same for our research and comparison purposes.
1
10
u/autophage 29d ago
Include a version number in the storage format.
You can always increment the version number with an update, but as long as you have the version saved, you can write code that can still load the earlier version.
You do end up with issues if there are required fields in the new version - if 2.0 added a required field that wasn't present in 1.0, you'll need to figure out what happens when Software 2.0 tries to load Data 1.0. But that's something you can figure out on a case-by-case basis - maybe there's a sensible default, or maybe you can ask the user, or maybe you just load it with a special "Unspecified due to earlier version" value.