Where Hardware Meets AI

Putting a language model behind a physical device changes what you optimize for. On the web you can hide a second of latency behind a spinner. On a device with a button and an LED, a second of silence reads as broken.

Here's what I learned wiring an ESP32 up to a hosted model for Forge.

The loop is the product

The microcontroller doesn't talk to the model directly — it talks to a thin edge function that owns the prompt, the auth, and the rate limiting. The device just sends intent and renders a response:

// ESP32 — keep the firmware dumb, keep the glue smart
void onPress() {
  led(PULSE);                 // acknowledge instantly
  String reply = ask(buffer); // round-trip to the edge
  render(reply);
  led(reply.length() ? OK : ERR);
}

That led(PULSE) before the network call is the whole trick: acknowledge the input in hardware immediately, then fill in the answer when it arrives. The user never sees dead air.

Three things that mattered

1. Trim the context. Every token you send is latency. Ship the device a tight system prompt and only the state that changed.
2. Stream if you can. Even printing the first few characters early makes the whole exchange feel twice as fast.
3. Fail loud, fail local. When the network drops, the device should say so on its own — not hang waiting for a response that isn't coming.

The takeaway

Embedded AI isn't about cramming a model onto a chip. It's about drawing the line in the right place — dumb, responsive firmware on one side; smart, replaceable glue on the other — so the physical thing always feels alive.