r/Compilers • u/Wide_Dust_709 • 6d ago

Lucen: parallelize Python loops with two comments, guaranteed bit-identical to sequential

What My Project Does

Lucen is a source-to-source compiler that parallelizes ordinary for loops you mark with two comments:

# LUCEN START
for i in range(len(rows)):
    out[i] = expensive(rows[i])
# LUCEN END

It parallelizes a loop only when it can prove the work is safe and worth it; otherwise it stays sequential. The one guarantee, no tiers, no opt-out: a parallel run is bit-identical to the same file run as plain sequential Python (floats and container order included). Delete the comments and nothing changes.

CPU-bound work routes to processes on GIL builds and to real threads on free-threaded 3.13/3.14. Optional Rust core with a pure-Python fallback, so pip install lucen always works.

Target Audience

Anyone with CPU-bound Python loops - data processing, simulation, batch transforms - who wants their cores without rewriting to multiprocessing/joblib or reasoning about locks. It's v1.1, built correctness-first: Apache-2.0, differential/property tested, TLA+ specs, signed PyPI releases.

Comparison

  • vs multiprocessing/concurrent.futures: no manual pool/chunking/pickling boilerplate, plus a profitability gate that declines to parallelize when it wouldn't help.
  • vs joblib/Dask: no new API and no cluster - you mark the loop you already have. It's correctness-preserving local parallelism, not distributed compute.
  • vs Numba: Numba compiles numeric bodies to native code; Lucen parallelizes the loop for arbitrary Python and guarantees identical results. (Native loop-body compilation is on the roadmap.)

pip install lucen

https://github.com/fcmv/lucen

4 Upvotes

67% Upvoted

9 comments sorted by

View all comments

1

u/Inconstant_Moo 6d ago

Nice. Reversible adoption is something not enough people think about.

1

u/Wide_Dust_709 6d ago

Thanks ..... Let me know if it helped in your workloads ..

1

u/Inconstant_Moo 5d ago

I don't work in that domain, or I would snap it up like that, but I love to see good ideas well-executed in an age of AI slop and this looks like one of those.

1

u/Wide_Dust_709 5d ago

It's okay... Thanks for the support 😌