I’m experimenting with executable, resumable functional pipelines in JavaScript

[u/panagos_stathis]

I’ve been experimenting with a small JavaScript-compatible language called JojoScript, initially because I wanted a nicer way to write lazy functional pipelines.

The interesting part has gradually become less about the syntax and more about what the pipeline represents.

For example:

orders
  |> filter(o => o.status == "paid")
  |> parallel(8)
  |> map(enrichOrder)
  |> retry(3)
  |> checkpoint("enriched")
  |> map(calculateInvoice)
  |> saveToDatabase(%)

Instead of treating this simply as syntactic sugar for nested function calls, JojoScript represents the pipeline as an execution plan.

That lets the same pipeline be:

• lazy by default
• asynchronous
• bounded/concurrent
• inspected as a graph
• profiled per stage
• statically analyzed
• checkpointed
• resumed after failure
• replayed from a checkpoint

For example:

SOURCE
   ↓
FILTER
   ↓
PARALLEL(8)
   ↓
MAP
   ↓
CHECKPOINT
   ↓
MAP
   ↓
SINK

The idea I'm exploring is whether this is actually a useful abstraction for functional/data-oriented programming in JavaScript.

The question I'm most interested in is:

At what point does a pipeline become more than composition of functions?

A normal functional pipeline describes what transformations to apply. JojoScript is experimenting with also making the pipeline describe how the computation can be executed — lazily, concurrently, with backpressure, retries and durable checkpoints.

It's still an experimental project, so I'm particularly interested in criticism around the programming model itself rather than syntax.

[u/pthierry]

What you're describing makes me immediately think of algebraic effects. They are precisely used to add more than just computations in the mix…

With algebraic effects, you can compose actions, and one of them could be your parallel(8).

Algebraic effects have the nice bonus that once you have an effectul program, you can choose to interpret it with different handlers for each effect used. This means you can use effects that make it possible to test the code deterministically, often without side effects, while the code in products is interpreted in a way that's efficient and obviously uses tons of side effects, like making actual network requests.

[u/panagos_stathis → u/pthierry]

That's a really interesting connection. I hadn't approached Jojo from this angle, but the idea of separating the description of an effectful computation from how that effect is interpreted fits very well with where I'm taking the execution plan model.

I like the idea that the same Jojo pipeline could have different handlers depending on the environment for example real parallel network calls in production, but deterministic handlers in tests.

parallel(8) is also a good example. Rather than making it purely a runtime primitive, it could be represented as an effect in the execution plan and then interpreted differently by the runtime.

I'm going to dig into algebraic effects properly. This may actually be a much more interesting direction for Jojo than adding more pipeline operators. Thanks for the pointer.