Dan

@thebsdbox.co.uk

Tinkerer extraordinaire 👨‍🔧 mainly eBPF 🐝, Kubernetes, Cilium and kube-vip.io

So initially I started writing a service mesh (for fun), then everyone started talking about AI workloads. So here we are, a transparent gateway that can be applied to running workloads (no restart) that allows control between agents/workloads and an LLM. kube-gateway.io

When Heptio was acquired by VMware we had to give up our downtown Seattle office. We had just gotten a new light up sign for the reception and no one wanted it so it has been sitting in my garage for the past 8 years. I have space now so just hung it up and it is magnificent.

A very large “Heptio” logo sign hanging in a workshop with plywood walls. The logo is aluminum with a green glowing backlight.

I am two people. - One, doesn't like to implement too much logging in my applications to make things clearer. - Two, the one who has no idea why my application behaves that way.. because the logs don't report anything.

Existing AI workloads? A few annotations and I can now transparently change their behaviour, swap out models in their API calls and more. (Thanks to #eBPF)

Existing AI workloads? A few annotations and I can now transparently change their behaviour, swap out models in their API calls and more. (Thanks to #eBPF)

Re-found the virtual machine manager project for Kubernetes from last year, this example has BeOS inside a kubernetes cluster ssh-ing out to the control plane to manage itself 😂. No CNI needed either thanks to eBPF.

Running BeOS in a Kubernetes cluster

Regardless of what the AI client is (GO, Python etc...) we can now ensure we jump in to the "existing" data path and manipulate any of the data in either direction 😀

Two pods using different SDKs and languages (GO/Pyton) to access AI backends, both being intercepted and sent to the kube-gateway proxy transparently.

Using eBPF 🐝 and some proxy magic 🪄 to change the model on the fly transparently to the AI workload. We can see the first responses from llama and subsequent ones from gemma (more emojis).

Bild

Two pods with their traffic being proxied (thx eBPF 🐝) through a gateway, one using kTLS and the other user-land TLS. 🔐

Transparent proxy with eBPF and kTLS for encryption.