# Host-Invoker Pattern ## Why this exists Azul's callback typedefs pass aggregates by value: ```rust pub type CallbackType = extern "C" fn(RefAny, CallbackInfo) -> Update; pub type LayoutCallbackType = extern "C" fn(RefAny, LayoutCallbackInfo) -> Dom; pub type ButtonOnClickCallbackType = extern "C" fn(RefAny, CallbackInfo) -> Update; ``` That's perfectly C-ABI — every C and C++ user can pass a `&extern "C" fn` straight in. But every **managed-FFI** binding we ship — LuaJIT FFI, ruby-ffi, PHP FFI, koffi (Node), CFFI (Common Lisp), FFI::Platypus (Perl), ctypes Foreign.funptr (OCaml) — sits on top of **libffi**, and libffi's closure builder cannot synthesise a C-callable trampoline whose signature has aggregate-by-value args. A naive `ffi.cast('AzCallbackType', luaFn)` either silently produces a junk pointer or refuses to parse the typedef. We can't fix libffi. What we ship instead is a **C-side adapter** built into `libazul`: a per-callback-kind static thunk whose signature *is* aggregate-by-value (so the framework can call it without changes), and which *forwards* to a host-side closure whose signature is pointer-args only (so libffi *can* synthesise it). ## Architecture in one picture ``` managed-FFI host (Lua / PHP / Node / …) ┌───────────────────────────────────────────────────────────┐ │ user code: │ │ function on_click(data, info) … end │ │ │ │ registerCallback('Callback', on_click) ┐ │ │ └─► allocate id, stash fn in handles[id] │ │ └─► call AzCallback_createFromHostHandle(id) ─────────┼──┐ │ │ │ │ registered libffi closure (pointer args only): │ │ │ callback_invoker(id, *RefAny, *CallbackInfo, *Update) │ │ │ fn = handles[id]; ret = fn(...); *out = ret │ │ │ ▲ │ │ └──────┼─────────────────────────────────────────────────────┘ │ │ │ │ (registered once at module load via │ │ AzApp_setCallbackInvoker) │ │ │ ▼ │ ┌───────────────────────────────────────────────────────────┐ │ │ libazul (Rust) │ │ │ │ │ │ AzCallback { cb: az_callback_thunk, ctx: } ◄───┼───┘ │ │ │ az_callback_thunk(data: RefAny, info: CallbackInfo) │ │ -> Update { │ │ let handle = info.get_ctx().refany_to_host_handle(); │ │ let invoker = CALLBACK_INVOKER.get(); │ │ let mut out = Update::DoNothing; │ │ invoker(handle, &data, &info, &mut out); │ │ out │ │ } │ └───────────────────────────────────────────────────────────┘ ``` The arrows in red ink: the framework calls `cb` with by-value args (works because the thunk is compiled by Rust); the thunk calls the registered libffi closure with pointer args (works because libffi can synthesise that). The user's host-language function never sees the aggregate-by-value version of the typedef. ## The Rust side: `impl_managed_callback!` `core/src/host_invoker.rs` defines: * **`AzApp_setHostHandleReleaser(extern "C" fn(u64))`** — process-global hook fired when a host-handle `RefAny`'s last clone drops. Lets the host drop its `id → callable` table entry. * **`AzRefAny_newHostHandle(u64) -> AzRefAny`** + **`AzRefAny_getHostHandle(*const AzRefAny) -> u64`** — same id-keyed path serves user data, so callbacks and `refanyCreate` share one releaser and one map. * The macro **`impl_managed_callback! { … }`** — expands per kind to a static thunk, an `AzApp_setInvoker` setter, and an `Az_createFromHostHandle(u64)` constructor. A typical invocation: ```rust azul_core::impl_managed_callback! { wrapper: ButtonOnClickCallback, info_ty: CallbackInfo, return_ty: Update, default_ret: Update::DoNothing, invoker_static: BUTTON_ON_CLICK_INVOKER, invoker_ty: AzButtonOnClickCallbackInvoker, thunk_fn: az_button_on_click_callback_thunk, setter_fn: AzApp_setButtonOnClickCallbackInvoker, from_handle_fn: AzButtonOnClickCallback_createFromHostHandle, } ``` For widget callbacks that take state extras (e.g. `(RefAny, CallbackInfo, CheckBoxState) -> Update`), append: ```rust extra_args: [ state: CheckBoxState ], ``` The macro forwards every extra by pointer through the libffi closure — no aggregate-by-value anywhere on the host-facing signature. It also expands a handful of `Display`/`Debug`/`Clone` impls that the wrapper struct needs to satisfy `repr(C)`'s derive bounds. `default_ret` is what the thunk returns when: * the framework called the typedef directly without going through this path (`OptionRefAny::None` ctx), * the ctx came from somewhere that isn't a host-handle RefAny, * or no invoker has been registered yet for this kind. Pick a value that can't be confused with a "real" return — typically the kind's "do nothing" / "empty body" default. ## Where the ctx is preserved (critical) The host-handle is carried in the wrapper's `ctx: OptionRefAny` field. Three call sites in the framework would otherwise drop it: * `dll/src/desktop/shell2/common/layout.rs::regenerate_layout` — calls `info.set_callable_ptr(&layout_callback.ctx)` before invoking. * `layout/src/window.rs::invoke_single_callback` — `CallbackInfoRefData.ctx = callback.ctx.clone()` (was hard-coded `None` once upon a time). * `layout/src/callbacks.rs::Callback::from_core` — preserves ctx when reconstructing from the storage form. If `info.get_ctx()` returns `None` inside the thunk, the host-invoker falls back to `default_ret` and the user's callback never fires. That's the symptom to look for if you add a kind and the dispatch silently no-ops. ## How to register a new widget callback Three steps. Once any one is wrong the host-language side either won't codegen (silent allowlist filter) or won't dispatch (silent ctx loss). ### 1. Apply `impl_managed_callback!` next to the widget's `impl_widget_callback!` ```rust // layout/src/widgets/my_widget.rs pub type MyWidgetOnFooCallbackType = extern "C" fn(RefAny, CallbackInfo, MyWidgetState) -> Update; impl_widget_callback!( MyWidgetOnFoo, OptionMyWidgetOnFoo, MyWidgetOnFooCallback, MyWidgetOnFooCallbackType ); azul_core::impl_managed_callback! { wrapper: MyWidgetOnFooCallback, info_ty: CallbackInfo, return_ty: Update, default_ret: Update::DoNothing, invoker_static: MY_WIDGET_ON_FOO_INVOKER, invoker_ty: AzMyWidgetOnFooCallbackInvoker, thunk_fn: az_my_widget_on_foo_callback_thunk, setter_fn: AzApp_setMyWidgetOnFooCallbackInvoker, from_handle_fn: AzMyWidgetOnFooCallback_createFromHostHandle, extra_args: [ state: MyWidgetState ], } ``` The convention: `invoker_static` is `SCREAMING_SNAKE`, `invoker_ty` / setter / handle-fn keep the `Az` prefix and the wrapper name verbatim, `thunk_fn` is `snake_case`. The codegen does not parse these names — they only need to be unique within `core::host_invoker`. ### 2. Register the wrapper in `HOST_INVOKER_KINDS` ```rust // doc/src/codegen/v2/managed_host_invoker.rs pub const HOST_INVOKER_KINDS: &[&str] = &[ "Callback", // …existing entries… "MyWidgetOnFooCallback", ]; ``` Every managed-FFI adapter (`lang_lua/managed.rs`, `lang_ruby/managed.rs`, …) iterates this list, so adding the entry here is enough — no per-language edit needed. The wrapper name here is the struct name **without** the `Az` prefix and **without** the `Type` suffix. (The codegen helper strips them in `wrapper_name(cb)`.) ### 3. Rebuild the dll, rerun codegen ```bash cargo build --release -p azul-dll cargo run --bin azul-doc -- codegen all ``` That re-emits `azul.lua`, `azul.rb`, `Azul.php`, `azul.js`, `azul.lisp`, `Azul.cs`, `AzulHostInvoker.java`, `Azul.kt`, and `Azul.psm1` with the new kind wired up automatically. There's no per-language "register" step; every adapter walks the same allowlist. ## How to add a new language adapter Three tiers, picked by the host language's FFI capabilities. ### Tier A — Native struct-by-value + closures Languages that can synthesise a C-callable function pointer from a host-side closure with aggregate args: **C# / .NET (P/Invoke + delegates), Java/Kotlin (JNA Callback), Python (PyO3, compiled in), Haskell (`foreign export ccall "wrapper"`), Zig, Go (with cgo)**. These don't *need* the host-invoker pattern, but apply it anyway for uniformity. `lang_csharp/managed.rs` is the reference. The shape: * A sibling `NativeMethodsManaged` class (or namespace) holding `[DllImport]` declarations for the host-invoker C-ABI exports. * A static `HostInvoker` class with `RegisterCallback(...)` factories per kind, plus `RefanyCreate(value)` / `RefanyGet(refanyPtr)`. * Per-kind delegate types matching the libffi pointer-arg invoker signature. * GC-pinning is one static `List` so `Marshal.GetFunctionPointerForDelegate(delegate)` can't have its trampoline collected. ### Tier B — No closures, struct-by-value works Static-procedure languages — **Fortran, COBOL, Ada, Pascal, FreeBASIC, VB6, Algol 68**. Closures don't exist; the user defines a static procedure and stashes per-instance state in the `RefAny`. These don't need a `managed.rs` at all. The codegen emits ordinary `bind(c)` / `Convention(C)` / `cdecl;` declarations against the production `azul.h` and the user passes `c_funloc(my_proc)` directly. The host-invoker is unused; the framework's RefAny refcount handles lifetime. ### Tier C — Libffi-restricted (the host-invoker tier) Languages whose FFI library can't synthesise aggregate-by-value trampolines: **Lua (LuaJIT FFI), Ruby (ruby-ffi), Perl (FFI::Platypus), PHP (built-in FFI), OCaml (ctypes), Node (koffi — Bun/Deno are technically capable but ride along for uniformity), Common Lisp (CFFI), Smalltalk (Pharo UnifiedFFI)**. These need a per-language `lang_/managed.rs`. Reference: any of `lang_lua/managed.rs` or `lang_php/managed.rs`. The shape: 1. **cdef declarations** for the host-invoker C-ABI exports — splice into the language's cdef block. Reuse `managed_host_invoker::emit_cdef_block(out, ir)` for a C-syntax payload that LuaJIT, PHP FFI, koffi, and CFFI all accept. 2. **Per-kind libffi closure registration** at module load — a closure per kind whose signature is `(u64 id, …pointer args…, T* out)`. The closure looks up `_handles[id]` and dispatches. 3. **`registerCallback(kind, fn)`** — allocates a host-handle id, stashes `fn`, returns `Az_createFromHostHandle(id)`. 4. **`refanyCreate(value)` / `refanyGet(refany)`** — same id-keyed path so user data and callbacks share one lifetime story. Wire the adapter into `lang_/mod.rs` between the existing types/functions emitters and the wrapper emitter — the wrappers will reference `registerCallback` once you teach them to (see "Wrapper substitution" below). ## Wrapper substitution (idiomatic call sites) By default a Tier C user has to write: ```lua local on_click_cb = azul.registerCallback('Callback', on_click) button:set_on_click(data:clone(), on_click_cb) ``` The wrapper-emitter substitution in `lang_/wrappers.rs` lets the user write: ```lua button:set_on_click(data:clone(), on_click) -- closure handed in directly ``` The substitution rule, implemented in `lang_lua/wrappers.rs::emit_callback_pin_lines`: * For every method arg whose IR `callback_info` is `Some`, prepend `arg = azul._register_callback('', arg)` before the C call. * Special-case: when the C ABI takes the *raw fn pointer typedef* (e.g. `WindowCreateOptions::create(LayoutCallbackType)` — passes the cb but drops ctx), bypass via `_default()` + direct field assignment so the host-handle ctx survives. Tier A languages can do the same substitution targeting their native delegate type. Tier B doesn't need it — static procedures don't have an "is this a closure?" question. ## Why PHP is different Of every libffi-adjacent FFI binding we ship, **PHP is the unique outlier**: standard `php-ffi` rejects closure-to-fnpointer entirely, by design. The php-ffi authors closed this off for memory-safety reasons — a PHP closure can be GC'd while C still holds the function pointer. We can confirm: every other binding in the matrix below supports closure-as-fnpointer for at least pointer-arg signatures (the host-invoker pattern's bread and butter): | Binding | Closure-as-fnpointer | Since | |---------|----------------------|-------| | LuaJIT FFI | `ffi.cast(typedef, fn)` | LuaJIT 2.0 (~2010) | | ruby-ffi | `FFI::Function.new` | ffi gem 1.0 (~2009) | | Python ctypes | `CFUNCTYPE` | Python 2.5 (2006) | | CFFI (Common Lisp) | `defcallback` | 0.1 (~2005) | | Perl FFI::Platypus | `closure` | 0.20 (~2015) | | OCaml ctypes | `Foreign.funptr` | 0.1 (~2013) | | Node koffi | `koffi.register` | v1 (~2022) | | Bun `bun:ffi` | `JSCallback` | Bun 0.5 (~2023) | | Deno | `UnsafeCallback` | 1.30 (~2023) | | C# P/Invoke | `Marshal.GetFunctionPointerForDelegate` | .NET 1.0 | | Java/Kotlin JNA | `Callback` interface | JNA 1.0 | | **PHP FFI** | **never — by design** | 7.4 (2019) → 8.5+ | The status is unchanged across PHP 7.4 / 8.0 / 8.1 / 8.2 / 8.3 / 8.4 / 8.5. There's no version-conditional fallback to enable. ### PHP workarounds (today, all imperfect) * **`dstogov/php-ffi-callback`** — a third-party PECL-style extension by the original php-ffi author that adds `FFI::createCallback()`. Requires native compilation against your specific PHP point release. * **`php-ffi-callable`** by 7php — similar third-party extension, also source-only. * **Polling** — Azul could expose a thread-safe event ring buffer that PHP polls each tick. No callbacks at all. Reshapes the API model. * **Native PHP extension** (preferred long-term, see below). The current `lang_php/managed.rs` adapter goes through the host-invoker plumbing and works for *the non-callback half* of the API: POD wrappers, RefAny construction, `FFI::cast` for type conversion, raw FFI dispatch. Anything that reaches `Azul::registerCallback(...)` will fatal at the `$ffi->cast(typedef, $closure)` call until one of the workarounds is in play. ### Planned: `php-extension` Cargo feature (future work) The cleanest long-term answer mirrors how Python is wired today. The `python-extension` Cargo feature compiles `azul-dll` *as* a Python extension via [PyO3](https://pyo3.rs/), using the `python-extension` feature flag in `dll/Cargo.toml`: ```toml python-extension = ["build-dll", "pyo3", "use_pyo3_logger", "link-static"] ``` The codegen emits `target/codegen/python_api.rs` carrying `#[pyclass]` / `#[pymethods]` / `#[pymodule]` annotations over the same IR; `dll/src/lib.rs` includes that file under `#[cfg(feature = "python-extension")]` and re-exports `PyInit_azul`. A single `cargo build --release -p azul-dll --features python-extension` yields a `libazul.dylib` that loads as a Python extension. Closures work natively because they're dispatched inside the same interpreter Rust has access to via PyO3. The PHP analog is [`ext-php-rs`](https://github.com/davidcole1340/ext-php-rs) — the PyO3-shaped Rust crate for writing PHP extensions. The plan: ```toml # dll/Cargo.toml (planned) ext-php-rs = { version = "0.13", optional = true } php-extension = ["build-dll", "ext-php-rs", "link-static"] ``` ``` doc/src/codegen/v2/lang_php_extension/ # new emitter, mirrors lang_python.rs # walks the IR, emits #[php_class], # #[php_function], #[php_module] target/codegen/php_api.rs # generated annotated Rust source dll/src/lib.rs # gated `mod php { include!(...) }` # + `pub use php::module_entry;` ``` User-side: `php -d extension=azul.so hello-world.php`. The whole `lang_php/` FFI-based adapter becomes a fallback for users who can't or don't want to install a native extension. #### Friction differences vs. Python | | PyO3 | ext-php-rs | |---|---|---| | ABI stability | Python `abi3` — one binary works across Py 3.7+ | **No abi3** — every PHP point release (8.0/8.1/8.2/8.3/8.4/8.5) needs its own `.so` | | Distribution | PyPI binary wheels via `pip install` | **PECL is source-only**; binary distribution requires our own channel | | Maturity | Massive ecosystem | Smaller (1k★) but actively maintained | #### Proposed CI shape The `php-extension` artifact is conditionally built in a separate "full release" CI job — *not* on the per-PR test path. Per-PR PHP testing continues to use the FFI-based `Azul.php` adapter (which covers everything except callbacks). The full-release job runs the N-version PHP matrix (currently 8.0–8.5 = 6 builds) in parallel and publishes the resulting `azul-php-{version}-{platform}.so` artifacts alongside the regular release. #### Scope estimate Roughly a one- to two-day spike: * **~1 day**: `ext-php-rs` integration in `dll/Cargo.toml` + `build.rs` hook + `lib.rs` gate + a `lang_php_extension` codegen emitter modeled line-for-line on `lang_python.rs`. * **~½ day**: per-PHP-version CI matrix (separate job, full-release only) + binary publishing flow. * **~½ day**: api.json install-instructions update, hello-world.php update, internals doc cross-link from this section. Tracked separately. The host-invoker pattern's pure-FFI path stays the primary documented entrypoint for languages where it actually works (every binding in the table above except PHP). ## Generic byte-buffer invoker (fallback path) The macro-generated thunks have a second dispatch path that fires when no per-kind invoker is registered: the **generic invoker**. Hosts can register one libffi closure that handles every kind, with the wrapper name carried as a string and args carried as a pointer array. C-ABI shape: ```c typedef void (*AzGenericInvoker)( uint64_t handle, /* host-handle id */ const char* kind, /* null-terminated wrapper name */ const void* const* args, /* one pointer per by-value arg, in declared order */ size_t n_args, /* args[] length */ void* ret /* where to write the return value */ ); extern void AzApp_setGenericInvoker(AzGenericInvoker); ``` When the per-kind invoker slot is empty, the macro's thunk packs `(&data, &info, &extras…)` into a stack array and forwards through `AzApp_setGenericInvoker`. The host decides what to do per kind from the `kind` string. Two use cases this supports: * **Single dispatch site for every kind.** Hosts that prefer one libffi closure to cover all kinds (and dispatch internally on the kind name) can skip per-kind `AzApp_setInvoker` calls entirely. This shrinks the prelude in languages where every libffi cast costs noticeable code bytes. * **User-defined custom kinds.** A user in a Tier-C host can add a new callback kind by emitting an `impl_managed_callback!` invocation in a *downstream* Rust crate (or shipping their own static thunks via a small C trampoline) without ever touching `HOST_INVOKER_KINDS`. The generic invoker fires whenever the per-kind setter wasn't called. Tests in `core/tests/host_invoker.rs` cover the slot registration path; the integration path is exercised through `examples/lua/hello-world.lua` when `AzApp_setCallbackInvoker` *is* registered (per-kind path) and through any host that registers only `AzApp_setGenericInvoker` instead. ## Tests Process-global slots make these tests serialise on a `Mutex`, but the coverage is enough: * `host_handle_to_refany(id)` round-trips through `refany_to_host_handle`. * The destructor stamped into host-handle RefAnys forwards the id to the registered releaser exactly once, when the *last* clone drops. * `refany_to_host_handle` returns `None` for unrelated RefAnys (so a user-data RefAny accidentally fed into a callback's ctx slot can't free a foreign id). * The macro-generated thunks short-circuit safely when no invoker has been registered yet. ```bash cargo test -p azul-core --test host_invoker ``` The end-to-end "thunk fires invoker with the right by-value args" path is exercised through `examples/lua/hello-world.lua` — the click counter increments through `on_click → Lua → counter mutation → next-frame layout cb`, which only works if every layer of the host-invoker plumbing is wired correctly.