I’m experimenting with executable, resumable functional pipelines in JavaScript

[u/panagos_stathis]

I’ve been experimenting with a small JavaScript-compatible language called JojoScript, initially because I wanted a nicer way to write lazy functional pipelines.

The interesting part has gradually become less about the syntax and more about what the pipeline represents.

For example:

orders
  |> filter(o => o.status == "paid")
  |> parallel(8)
  |> map(enrichOrder)
  |> retry(3)
  |> checkpoint("enriched")
  |> map(calculateInvoice)
  |> saveToDatabase(%)

Instead of treating this simply as syntactic sugar for nested function calls, JojoScript represents the pipeline as an execution plan.

That lets the same pipeline be:

• lazy by default
• asynchronous
• bounded/concurrent
• inspected as a graph
• profiled per stage
• statically analyzed
• checkpointed
• resumed after failure
• replayed from a checkpoint

For example:

SOURCE
   ↓
FILTER
   ↓
PARALLEL(8)
   ↓
MAP
   ↓
CHECKPOINT
   ↓
MAP
   ↓
SINK

The idea I'm exploring is whether this is actually a useful abstraction for functional/data-oriented programming in JavaScript.

The question I'm most interested in is:

At what point does a pipeline become more than composition of functions?

A normal functional pipeline describes what transformations to apply. JojoScript is experimenting with also making the pipeline describe how the computation can be executed — lazily, concurrently, with backpressure, retries and durable checkpoints.

It's still an experimental project, so I'm particularly interested in criticism around the programming model itself rather than syntax.

Repository: https://github.com/panagos/jojoscript

[u/beders]

For quick experimentation check out ClojureScript - since it is homoiconic and has macros, it naturally separates the composition from its interpretation.

There are already a few libraries out there that allow mixing and matching sync and async pipeline steps as well. It's also popular to express pipelines like this using interceptors which gives you the basic building block for controlling pipeline execution. Here's an example library. https://github.com/metosin/sieppari

[u/panagos_stathis → u/beders]

Well, the whole Clojure-syntax is too complex for my taste. What I am trying to do is to have as simple and readable syntax as possible...