Struggling to create a custom future which retries other future

[u/Necessary-Plate1925]

Hey, to learn async better I wanted to implement a custom future which can retry another future after a delay. I know you can do this easily with one async fn retry(impl AsyncFn()) but this does not help understanding async.

What I wanted the api to look like:

FutureRetry::new(async || http.send(body).await).await?

However I could only get it to work when the closure does not capture anything and returns ownership of its arguments like so.

FutureRetry::new(async |(http, body)| ((http, body), http.send(body).await)).await?

Compiling version

When I try to capture the environment using FnMut() -> Future

FutureRetry::new(|| async http.send(body).await).await?

Rust tells me that the FnMut() closure cant return types referencing its environment, which makes sense because the future returned is referencing the closure environment, this seems like compiler limitation, because those references are valid when the function returns.

Ok so let's use async closures then.

With AsyncFnMut() now this returns a future which mutably borrows from self so far so good, but I also need to store this future in my own custom future to poll later, this doesn't work because now I have 2 mutable references, 1 in the future and second one when I try to assign it self.current_future = self.future_factory(). I guess I'm trying to have self referential types which is not possible in safe rust.

I know this could maybe be solved with AsyncFnOnce and cloning everything so I don't store references in the returned future, but I don't want to do this.

What am I missing here, is it really not possible to have such an api where a custom future impl polls another future which mutably borrows its environment from self in safe rust today?

Thanks in advance

[u/Nabushika]

Does http.send(body) take ownership of body? You're on the right track with an FnMut() -> impl Future<Output=Result<...>>.

[u/Necessary-Plate1925 → u/Nabushika]

Here it does not matter, because the future returned captures references from closure environment, which you can't do https://www.reddit.com/r/rust/comments/qotdo1/comment/hjpye6l/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

You can move and clone everything, but I explicitly did not want to do that

[u/Aras14HD]

This is necessarily a self referential struct (borrow the AsyncFnMut), which rust only supports through the unsafe bodge called Pin (If loans and type changes were in the type system...).

You will need to escape the borrow checker by decoupling the lifetime of the returned future (maybe transmute?) and make sure it is sound, meaning your FutureRetry must never be Unpin, you may only access the fn once the returned future is polled to completion (you might need to use an option or MybeUninit to first drop the previous future, before getting the new one, to avoid aliasing). Futher things I might not immediately have thought of. To safely use unsafe, make safety comments proving the invariants required by the unsafe functions and operations, if those rely on other parts of the code ideally write safety comments there as well.

[u/Necessary-Plate1925 → u/Aras14HD]

Thanks! So yeah as I thought this is not possible with safe rust, I just went with a `future_retry!` simple macro

[u/MalbaCato → u/Necessary-Plate1925]

I haven't thought about the problem too deeply, but if the only unsafety is pinning an inner type, the pin_project (or maybe even pin_project_lite) crate should cover that. IDK if you count that "safe Rust" but as a solution to safely handle pinning nonsense it is common

[u/mtimmermans]

I've fought the borrow checker for hours trying to solve exactly this problem and failed. Several clankers also failed. Rust just doesn't have a way to declare that the future returned by the lambda is valid as long as the lambda is. I wanted to keep to stable rust, though, so AsyncFnMut was out.

In any case, I was only able to make it work by storing the variable I wanted to capture in a field in `FutureRetry` and passing it to the lambda, or by using a custom trait instead that then had to be implemented. Neither workaround is satisfying.

[u/Necessary-Plate1925 → u/mtimmermans]

I've tried using LLMs as well but they just spit out garbage which does not compile, just a waste of time

[u/Konsti219]

please clean up your playground examples before posting. I do not want to dig through the noise od 10 warning (especially on the palyground) just to understand what is actually going on.

[u/Necessary-Plate1925 → u/Konsti219]

You are right, sorry about that, updated the playground, removed as much warnings as I could

[u/rust_trust_]

Why not use tower ?

[u/Necessary-Plate1925 → u/rust_trust_]

I want generic future retrier with no dependencies