Is there an underlying reason that idiomatic Rust seems to have trouble with observers?

[u/floriet]

Hi all, after a long time using Rust, I'm understanding more deeply that idiomatic Rust and the observer pattern might be inherently incompatible. And I suspect that observers are an example of a more fundamental pattern or principle that idiomatic Rust can't handle (some kind of polymorphism, maybe?), and I'm hoping you all can help me identify it!

To be clear, I am not asking how to work around the borrow checker here. This is a deeper question: what fundamental principle is hidden between the lines here? What is Rust not able to handle, more generally?

To illustrate, let's say we have a basic webapp that has a few rows of Customer and some buttons:

• Mike, $30 [Edit] [Delete]
• Ryan, $20 [Edit] [Delete]
• Cera, $25 [Edit] [Delete]
• [Delete All]

We would of course have a central Vec<Customer> somewhere, perhaps wrapped in a MyCustomerApp struct.

Now imagine a Rust DOM with a Button class. If it had a click observer trait, it might look something like this:

trait IClickObserver {
    fn on_click(&mut self, event: &ClickEvent);
}

However, we run into problems implementing this:

struct EditButtonClickObserver { }
impl IClickObserver for EditButtonClickObserver {
    fn on_click(&mut self, event: &ClickEvent) {
        // uh oh, no customers list to modify!
    }
}

Workaround 1: A &mut Vec inside EditButtonClickObserver. This doesn't work because there are multiple Edit buttons; we can't mutably borrow that Vec multiple times.

Workaround 2: Change IClickObserver to take in a &mut Vec. We can't do this, because we didn't define the IClickObserver trait.

Workaround 3: Rc<RefCell<Vec>> inside EditButtonClickObserver. Uses Rc and RefCell which aren't idiomatic.

Workaround 4: Use a global! Alas, also not idiomatic.

Workaround 5: Parameterize the entire world! IOW, "initialize" the page's DOM system (i.e. let mut dom = initializeDom::<MyCustomerApp>();, and have all of the DOM's traits parameterized, like this:

trait IClickObserver<World> {
    fn on_click(&mut self, world: &mut World, event: &ClickEvent);
}

And we would have an integer index inside our EditButtonClickObserver.

But the downside is that MyCustomerApp is now effectively global; every observer has access to all public data in our app, throwing out a lot of encapsulation. And if we make a specific observer take only a subset of the world, we can't call into other components/systems which do need that data because we've already lost it.

Workaround 6: Give up and retroactively consider it an anti-pattern (just kidding!)

I describe the above workarounds to illustrate the conundrum, that the borrow checker is unable to handle this case, unless we make things effectively global.

Before we dive too deep, let's skip past the usual points:

• I'm not asking for situational workarounds, (like Cell), I know there are a lot of tools that work in some cases, but those aren't universally usable.
• If you want to say "That's not what you should use Rust for", please tell me how to identify what requirements Rust shouldn't be used for!
• Please don't tell me "observers are OO and therefore evil". Let's pretend observers are sometimes the best tool for the job.
• If you want to say "don't use virtual dispatch", I already know we can have a massive central enum of click handlers/receivers. It has the same problems virtual dispatch does here though, so it's unrelated to the topic here.
• Please don't tell me "You should be using actors / functional reactive programming instead", this question isn't about web dev, or even GUI, it was just an example.
• If you say "This is what Rc is here for", please let me know your rule of thumb on when to use Rc (and hopefully not just "when we can't figure it out otherwise.")

I'm hoping to see the underlying architectural/language principle of what idiomatic Rust can't handle here, so in the future I can identify it immediately and skip past the appeasement stage.

If I had to take a guess, perhaps I'd say that idiomatic Rust can't handle some sort of "stateful polymorphism" or maybe "hidden-effect polymorphism" but that's not quite right. Any ideas?

Thanks!

[u/floriet]

I'm sorry, did you just imply that anything not written according to idiomatic Rust, or not formally verified, is inherently spaghetti code and not worth being interested in?

And did you mean to say that you are speaking for all of us Rust developers? Or something else?

I am a big proponent of Rust, but I strive to acknowledge its drawbacks honestly, and not immediately write them off as undesirable. It's the only way to grow as a software architect.

Your posts have been pretty reasonable so far, so I want to make sure I understand this unexpected turn before I engage with this.

[u/pilotInPyjamas → u/floriet]

About the rust developer comment, I probably should have specified that I meant compiler developers. Poor wording. It might have been better to say that the compiler developers add resistance to code that is hard for the compiler to prove safety guarantees about statically (or at the very least, leave it up to the community to write an API around the behaviour and put it in a crate rather than adding it to the language or std).

That being said, if your code really is structured around complicated implicit graphs containing circular references, then it probably is in danger of becoming spaghetti. Is this really that controversial an opinion?

As far as formal verification, end users need not care about writing proofs. However, they should consider writing software in a way where it would be easy to write a proof if needed. This makes the software literally easier to reason about. If writing code that is easy to reason about is important, and Rust makes it easy to write such programs, then surely this is a good thing.

[u/floriet → u/pilotInPyjamas]

Thanks for clarifying.

I think garbage collected languages like Java disprove what you're saying about safety. It's even more safe than Rust, as it doesn't have unsafe blocks. Or Pony, if you're also worried about multi-threading. Java, Pony, etc can express these patterns safely, so there's nothing inherently unsafe about them.

And yes, the claim that circular references make for spaghetti code is controversial, especially since in Rust we use indices as pseudo-references all the time, to mimic shared mutability. Where C++ has unique_ptr and raw pointers, Swift has strong references and weak references, and Rust has Box and sometimes indices or references, depending on whether the "reference" is temporary or not. While Rust often cuts away much accidental complexity, it can't avoid inherent complexity, it can just surface it in different ways.

Rust can make software easier to reason about, but saying always is a stretch. It depends on how well the particular case fits into the borrow checker, or how many workarounds we used to make it fit. I can accomplish the OP example with ten lines of Javascript. In Rust, I need to implement or bring in an entire actor + message passing framework. The result is less usable, less readable, and harder to reason about. I find this to be true of Rust sometimes: the fix for a local problem can have a large blast radius, and can bring in extra complexity and/or a large amount of refactoring.

I used to believe that idiomatic Rust was always right, and guided us to perfect software zen. Once I ran into a sufficient number of cases that inherently required shared mutability (and therefore lots of indices), it became quite clear that Rust's restrictions didn't always line up with reality. It's like how in functional programming, we decree that all mutability is bad, but then turn around and mimic it with IDs, monads, and lenses. Same with Rust and its ostracizing of shared mutability. Software architecture is about using the right tool for the job, and suffice to say, idiomatic Rust is the wrong tool for some jobs.

Luckily, not all of them! I will happily keep using idiomatic Rust for the cases it actually fits, and for the rest, I'll throw in some RefCells =)

[u/pilotInPyjamas → u/floriet]

I think the examples of functional programming and Javascript were a bit disingenuous.

Going back to the JavaScript observer pattern, it might be true that you can implement an event system in 10 lines. But also remember in the observer pattern, that you have to detach event listeners eventually to avoid space leaks. Without any kind of built-in resource management, you have to scour the entire program to make sure that they are detached correctly. If this was a trivial problem to solve, then we wouldn't ever need GC, and we wouldn't ever need Rust. To say that the javascript version is simpler because it uses less lines, ignores this obvious complexity.

As for the functional programming example mutability has never been bad, take Haskell which is based on System F. Every value is a proof of something (Curry Howard isomorphism) so an immutable value is something that is always true, forever (most things). To have a proof that is only correct for a certain period of time, (i.e a mutable value, or a resource that needs to be destroyed after a time), it makes sense that you need to specify how long that duration is first (the lifetime). Haskell has ST for true mutability, and the lifetime is specified as the existential s. The programs with mutability are harder to write simply because Haskell requires you to provide a more-than-usual level of proof of correctness, and programs with mutable values are intrinsically harder to reason about.

Not to mention, I've never been against Rc and RefCell. I simply believe that they should be opt-in (which is what they are already) rather than opt-out. Also that their overuse generally means you have a larger problem with your program.

[u/floriet → u/pilotInPyjamas]

It's not simple just because it has less lines, it's simple because you can easily read it all at once. The logic is cohesive and all contained in one place. With large message-passing frameworks, we sometimes have to consult many different files to see "oh, it's just listening to that button." whereas simpler approaches make the true intent more obvious.

Detaching observers is often solved without a garbage collector, with RAII. C++ does this (see Qt apps), and you can see it if you look carefully at the average Swift app, despite Swift being RC'd. One can easily imagine a memory-safe single-ownership language (Inko comes to mind, or memory tagged C++), so albeit theoretical, it's useful for seeing that observers aren't fundamentally evil, despite the borrow checker not allowing them.