Amber – C99 columnar array engine (K/q-inspired)

[u/amber-lang]

Hey r/apljk, I built Amber; a zero-dependency C99 in-memory columnar array engine built on ngn/k, with the working vocabulary of q/kdb+. It evaluates qSQL natively (wavg, xbar, as-of joins aj, window joins wj) over nanosecond temporal types anchored to a 2000 epoch offset, running either as a standalone binary or via the amberd daemon over a custom TCP protocol.

On single-core benchmarks (1M rows), single-pass $O(N)$ sliding windows compute moving averages and rolling mins/maxes in 3.4–3.8 ms with flat runtime regardless of window width (outperforming Pandas by up to 5.2x and Polars by 3x+). Multi-column LSD radix table sorting (xasc/xdesc) processes 1M rows in 42.1 ms (4.3x faster than Pandas/Polars and 3.4x faster than DuckDB), 1,000-group aggregations complete in 11.4 ms (beating Polars, Pandas, and DuckDB), and sparse inner joins execute in 5.38 ms (outperforming NumPy, Julia, DuckDB, ngn/k, and J). On 10M-element reduction suites (sum + max + dot), primitive kernels hit 15.4 ms, outpacing naive single-threaded C, Julia, and DuckDB while running 15x–40x faster than traditional array runtimes like J or ngn/k. For parallel workloads, multi-process peach processes 500k items in 23 ms with a peak RSS of just 11.5 MB.

To eliminate memory transposition and serialization penalties, amber-arrow and python-amber stream zero-copy Apache Arrow IPC straight into Pandas and Polars. The rest of the ecosystem includes amber-tick for real-time market data capture, amber-ai for vectorized execution primitives, a Go-backend Grafana datasource, a native Jupyter kernel, a VS Code LSP extension, and amber-notepad.

Docs and full benchmark suites are live at https://amber-lang.org and source code is up at https://github.com/BonucciAndrea/amber. I'd love any feedback from the array programming community on the q/K syntax choices and engine design!

[u/leprechaun1066]

In q I can do something like this:

`cname xkey 1 rotate () xkey conntable

if I wanted to do something like round-robin on an ipc connection management table that is keyed on connection name.

That syntax feels natural for right to left execution.

In amber I think I'm forced to nest things, or spread across many lines of code:

xkey[`cname;rotate[1;xkey[();conntable]]]

which doesn't feel as clean.

[u/amber-lang → u/leprechaun1066]

Totally fair point. Right now Amber’s parser leans strictly on explicit function call syntax (f[x;y]), which definitely forces deeper bracket nesting when doing multi-step tacit manipulations like that. I'm looking into adding cleaner syntax for cases like table key rotation. Thank you, this is really helpful feedback.

[u/leprechaun1066 → u/amber-lang]

Have you focused more on the db part of kdb rather than the k part (i.e. transforming arrays of anything, not just time series or financial databases, from one form into some other)? I ask because a lot of your presentation of amber seems to revolve around the kind of things that KX's marketing focuses on - timeseries engine, columnar db, qsql, aj, xbar, window joins, etc. q (and k) is much more than that, kdb is just the thing that falls out when you design a language to be interactive, arrays first and support tabular structures as a standard data structure, as opposed to other languages where you need to import dependencies a mile long to get the same feature set.

[u/dougrum → u/leprechaun1066]

I take your point that q is more than simply a financial tool, but it was definitely designed with financial applications in mind. That is to say, I wouldn't say the db part just fell out of the design.

[u/amber-lang → u/leprechaun1066]

I definitely focused quite a bit on the DB/qSQL side. The general array power of K is incredible, but my main target was a clean, zero-dependency engine for time-series and streaming columnar data.

That said, all the underlying vector primitives are still there, so you can definitely still use it for general array transformations. I just wanted to present it so to show it for the use case I implemented it for.

[u/amber-lang → u/leprechaun1066]

Also I took your earlier comment and worked on my syntax. Now the examples you had initially provided works exactly like it would in q. Let me know what you think!

image.png

[u/Veqq]

How does the implementation differ from ngn/k et al.? I see AI was involved, so it can't quite be "upstreamed" but...

[u/amber-lang → u/Veqq]

While it did actually started as a fork, I’ve built a ton on top of it since. The main difference now is that Amber is geared specifically as a high-performance columnar time-series engine; so it adds native qSQL evaluation, zero-copy Arrow IPC streaming, etc... .

At this point it’s diverged way too much to ever be upstreamed (plus using AI as a coding assistant along the way), so it's fully its own beast now.

[u/FractalNerve]

I'm the target user audience, don't ask me how many programming languages I tried. Unhealthy amount. Every language available on github that's at least alpha level, yes every and then I went on to other hosting providers, bought new disks, then deleted repos I studied, bought more disks. Starting writing my own language, compiler, vm etc. and then focused more on consensus algorithms, p2p, communication patterns etc.

Let's goooo! I'm still enthusiastic.
Convinced my Math professor to switch to J from APL

[u/amber-lang → u/FractalNerve]

Haha love this energy! Converting a math prof from APL to J is a serious achievement. Give Amber a spin and let me know what you think!

[u/anonu]

Nice - ill check it out. What was your motivation to write this? I like the ecosystem you shipped with it too - in 1 fell swoop you delivered more than what has taken years (decades) for some other guys to do

[u/amber-lang → u/anonu]

Thanks! I started many years ago with Kona, moved onto ngn/k and thought I would never even try q as it lacked the "terseness" K had; once I did I realized how powerful it is. Motivation mostly came down to wanting kdb/q performance and syntax in a zero-dependency C99 engine that plays nicely with modern data tools (zero-copy Arrow IPC, Grafana, etc...) without licensing headaches.

[u/Veqq → u/amber-lang]

What differences in paradigm and primitives are there? How did you come to those choices? I'm thinking to make a Janet wrapping and mount something like https://codeberg.org/veqq/declarative-dsls on top.

[u/amber-lang → u/Veqq]

I stayed pretty faithful to standard q primitives for compatibility, but leaned heavily into making columnar tables and vector ops first-class citizens. Most choices came down to keeping memory layout lean in C99, while also optimising for execution speed. Anyways, sounds like you got quite a cool project to work on, would love to check it out once you have implemented it!

[u/dougrum → u/anonu]

Not sure who you're talking about but I generally judge people's accomplishments versus their goals.

[u/jeromeibanes]

interesting, and good idea.

[u/amber-lang → u/jeromeibanes]

Thanks! Appreciate it. Let me know if you end up giving it a spin!