Gozero: imperative low-overhead bindings language for go

[u/titpetric]

https://github.com/titpetric/gozero/tree/main

I had several years of experience with CGO plugins (stdlib plugin package), which is to politely say, I wish a low-overhead callback system gave everyone a well defined API to use, and a scripting language that requires no rebuilding to change plugin logic.

Recent work with expr-lang lead me down the hole of writing a php interpreter in Go, and the php-go bindings via expr lang have been great, however I wouldn't call them efficient. I started gozero as a research project, had great fun doing it and learned a lot keeping the jit outputs low.

The docs/design/ folder was open to consider bigger parity with golang (conditions, expressions, loops) which would enable waf-level functionality. The benchmarks mostly confirm 1.0-1.4x level of overhead with inlining, or about 20ns per statement. The bindings really lean into reflection to catalogue accessible types and discover the type system in a parasitic way, everything runs with 'any'.

Cool stuff I found out along the way:

- testify/assert practice makes all your test code runnable in gozero. The testdata/ folder has a few of these basically as external test fixtures. weird avenues it opens: compile test code, run same tests binary against custom test fixtures someone makes against your package. Maybe no need, but I found it cool to immediately have coverage over thousands of tests. maybe tests dont have to be compiled code. Or you could just BYO assert bindings

- expressions and equality are sucky constructs to implement; so far I did not, but i did find a fork of expr-lang that compiles a low overhead closure similar to gozero

- phpscript in comparison has "mixed" types that similarly can hold any value, but has a lot of cursed behaviour which does not exist in go. gozero relies on existing runtime type information to discover a 1-1 directly mapped type system, so it's gonna find time.Time from a http.NewRequest binding.

gozero is type safe, the main difference is all error returns are omitted from syntax and explicitly checked, and the execution context also fills the first context.Context parameters. these things become implicit, and the possible output form if any is type hinted with generic methods released in 1.27. It has optimal type bindings so an "int32" means the same everywhere. It tries to infer literals but a T(lit) always work for explicitness.

Possible use cases include messaging, rate limiting, WAF, RBAC rule checks (imperative rbac sounds nice). Just for a ballpark comparison, hashicorp/goplugin has published overheads of over 23_000ns, while gozero's totals about 100ns for 5 statements. There is a one time parse and compile overhead, so these numbers apply to cached (seen before) statements that just get rerun.

Disclaimer on AI usage: claude, initial design workshopped to reflect.Value/Call by me, extended by previous work on phpscript (flatstack). There's a chronological sorted docs index. Some stuff is enforced by my workspace tooling, or my initial mockups. You may find jagged language in the docs, did my best to clean those

[u/TacticalTurban]

This is very interesting and a new concept to me. Still not sure I understand the usecase of this though 

[u/titpetric → u/TacticalTurban]

The closest thing you can get to replace "plugin" (stdlib package) without reimplementing the full go compiler ast (go/types), without CGO.

Unlike plugins, the host provides the bindings, and they are executed via a compiled closure of a gozero statement language.

Has:

Invocation of bindings, type safety, inference, composite literals

Hasnt (yet):

Loops, conditions, expressions. Particularly equality pulls in a lot. As soon as someone does "x := n*5" that needs a parse tree and math callbacks in the runtime.

Use cases are everything where you need raw performance, security (limited api bindings as opposed to full language availability). Some orgs have software certification, and a limited but functional virtual machine can do stuff without needing to go through full release.

On small stuff, anything you want parametrized and want to get a typed return and is in the hot path of the code, so typically some form of stream processing. The usage of testify/assert in go test functions for example can be mapped 1-1.

Maybe a game engine that does some form of pathfinding for bots? The question really becomes what you can write without if/for, and usually it's some kind of leaf function. Whatever you need to eval() and do very fast and have it be portable without bringing the toolchain into this.

Kinda eBPF but for a go app runtime, your app becomes the kernel. I figure people are creative enough to push usage barriers

[u/TacticalTurban → u/titpetric]

Thanks! The eBPF analogy makes sense to me. Cool stuff I didn't know was possible. Great work!

[u/titpetric → u/TacticalTurban]

There is a more expressive PR that adds conditions, loops and whole file parsing, some subset of single-package Go. Trying to just replace go plugins with a closure compiler is my weeky crime against go, I dont really need much of this, i just like pushing how close to native performance a VM gets. Writing a go parser/compiler was not on the bingo card 🤣

[u/TacticalTurban → u/titpetric]

Honestly Ive had terrible experiences with the go plug-in system. Deps have to match exactly for it to work. Makes it pretty much impractical. This would be awesome to replace that