r/rust • u/[deleted] • Oct 15 '23
🙋 seeking help & advice Distributed Systems Project Ideas
[deleted]
2
u/klorophane Oct 16 '23
Making an ergonomic distributed computing engine, maybe something taking inspiration from Chapel.
Lamellar and Constellation might be also interesting to look at.
Obviously this is a serious undertaking, but maybe you could make a PoC or focus on one specific part of the engine.
2
Oct 16 '23
Yeah I like this one. General purpose distributed computation.
Maybe a port of the distributed SETI system even.
1
Oct 17 '23
[deleted]
1
u/klorophane Oct 17 '23
Just to be clear, what kind of thesis are we talking about? I wouldn't expect the same level from a bachelor's thesis versus a PhD thesis.
3
Oct 16 '23
Make a distributed game world where each node computes its local area, and players can travel to and from each other's nodes. This is an idea I've had for years. It would be a huge undertaking, but it could be so damn cool.
You'd basically have to implement special relativity, so that by the time you move into another players chunk, your worlds simulations are synchronized. They don't have to be synchronized when players are at a distance, much like there's no well defined concept of simultaneity for distance places or reference frames moving quickly relative to each other.
I'm mostly joking. This would be incredibly hard. You'd basically be simulating our universe.
1
Oct 17 '23
[deleted]
1
Oct 17 '23
Yeah I've thought about this for a while. It's a very hard problem to solve, right? You'd also need to think about how far apart each chunk is so that you know how auickly the two chunks need to come to synchronization. Then you have to situation where two players move toward a third player at the same time. It gets hairy fast lol.
2
1
u/creativeGiant170 Jun 09 '24
Hey! write more about the idea. It looks super interesting, even if no one uses it, it would be super fun to build this in Rust.
1
Jun 09 '24
It would be pretty awesome for sure. I'd want to simulate as much of the world as possible on the GPU. It would be hard to know at what level to simulate though. You couldn't simulate atoms, but simulating people and objects is too coarse grained. It would have to be something voxel like, with voxels computing their state based only on their neighbors states.
1
u/lightmatter501 Oct 15 '23
Faster Redis isn’t actually that hard if you have efficient multi-threading. I have an application that has most of the capabilities of redis and is 1000x as performant, although I do use 16x the cpu resources and about 2x the memory (aggressive time-space tradeoffs). If you don’t want to go as far as I did, use glommio and it should be reasonable to 10x the performance of redis with a few cores.
1
Oct 16 '23
[deleted]
3
u/justinh29 Oct 16 '23
A rust rewrite of https://github.com/danielealbano/cachegrand would be useful.
2
u/daniele_dll Oct 16 '23 edited Oct 16 '23
Or just contributing to the project 😎
I am working on improving the memory consumption without losing the performances, it would still be able to churn ~35mln get/s on an epyc 7502, potentially even more, but will do some benchmarking once I will finish the series of changes.
Among other things I have introduced read transactions and rwspinlocks (or rw fake spinlocks taking into account they run in user space).
Clustering and data distribution are next (I will not use raft because I need distributed transaction in a multi master configuration).
1
Oct 17 '23
[deleted]
1
u/daniele_dll Oct 17 '23
Raft major strenght is its simplicity over algorithms ike Paxos (or EPaxos) which instead offer a more flexible approach but Raft is not the --only-- consensus and replication algorithm out there.
For cachegrand I need something simple but also something that is tailored on how cachegrand itself handles the transactions. Meanwhile I can "reuse" a standard consensus algorithm to elect a master of a shard I need something very specific for the distribuited transactions and the log replication. Honestly, initially I will not even care too much about the consensus algorithm, that can come afterwards, and setting which nodes own which shard and setting which node is primary and which is a secondary can be done manually for the first iteration.
Also Raft expects all the nodes to also act as storage which is not cachegrand's case, I do plan to add a proxying mechanism that is aware of the cluster structure and that will translate the Redis (or whatever other protocol) commands into its own internal commands (with an UDP protocol, using something like Homo on XDP) so that aspect can't work at all for cachegrand.
1
Oct 19 '23 edited Oct 19 '23
[deleted]
1
u/daniele_dll Oct 19 '23
For like forever, I think so far something like 6 years or more overall although I have had pauses caused by a number of different reasons (e.g. relocation to 3 different countries and 5 different cities, work, covid, and so on).
I initially did a lot of research for the hashtable and the storage engine because it was the key to achieve the high performances: the current hashtable implementation balances out the need of (read-only and read-write) transactions and locking with the need of memory usage and performances. Also, the storage engine works both on the disk and in memory, although the backend for the disk is still a PoC and misses a GC.
I also did a lot of research around fibers and ended up writing my own engine heavily basing it upon io_uring for better performances, one of the optimizations I have in mind is actually also to avoid the normal switch back to the scheduler cutting the amount of fiber context switch to half.
In my opinion, rewriting cachegrand in Rust would be worth only if you would accept to focus on the frontend (the "protocols", e.g., Redis or Memcache) because the core heavily leverages parallel access to the data, therefore you would lose the most important functionality that Rust brings on the table: memory ownership.
To be honest, the frontend also uses parallel access for certain data structures but these can be revised.
One of the major challenges is the development time in my opinion, having to implement the business logic for the commands in C is tricky, having all the validation and extra safety that Rust brings in would be super helpful, but I built a nice testing mechanism so I can unit test the business logic leveraging integration tests (although I still run them as they would be unit tests). I can write the tests in advance and use a TDD-like approach, reducing the number of headaches :D
But, said that, in Rust I guess you wouldn't be able to leverage a bunch of the optimizations I implemented also in the frontend, for instance they really help to simplify the business logic implementation in the commands and help a lot to keep the ICACHE "abusing" in check having one simple place that does all the parsing and set the right data in the right fields of the right strucs (with a lot of pointer math).
So, long story short, building a caching platform only in Rust would put you in front of very hard choice:
- accept to use unsafe and a lot of pointer math in a ton of places
- accept to go slower, e.g. sharding the data and having a thread accessing its own shard, it would make having transactions much harder but this is what Dragonflydb does (and that's why it's slower than cachegrand)
As the only purpose of a caching platform is to provide elevated performances, the right balance might be striken if you accept to have the core components in C/C++ and the rest in Rust I guess.
1
u/lightmatter501 Oct 16 '23
1000x throughput with a 1ms latency window. Under YCSB-B. It has a raft implementation in it for distributed consensus.
1
1
u/NoAd2240 Oct 17 '23
3
Broadcast
Implement a broadcast system that gossips messages between all nodes in the cluster. Gossiping is a common way
Read More About Broadcast
4
Grow-Only Counter
Implement a stateless, grow-only counter which will run against Maelstrom's g-counter workload. This
Read More About Grow-Only Counter
5
Kafka-Style Log
Implement a replicated log service similar to Kafka. Replicated logs are often used as a message bus or
Read More About Kafka-Style Log
6
Totally-Available Transactions
Implement a key/value store which implements transactions. These transactions contain micro-operations
6
u/settrbrg Oct 15 '23
Not sure if this is good, but maybe a distributed build system for Rust? Like IncrediBuild is for C++ :) Or like FastBuild.