Autoregressive transformers fundamentally mutate state at runtime. KV cache
tensors evolve at every step, sequence lengths vary with prompts, and shape
expressions flow through
operators like Shape, Gather, Concat, Reshape, and Expand.
Today, this does not map cleanly to WebNN’s static-graph constraints.
At step 1, KV cache length is 1. At step 512, KV cache length is 512. That is not a static graph.
If this is not solved, WebNN stays limited to fixed-shape demos for many LLM use cases.
For real product workloads, people need variable prompt lengths, efficient token-by-token decode, and stable latency as context grows. Without that, local browser LLM UX degrades quickly and teams default back to heavier alternatives.
This problem is now well documented in the WebNN WG in Issue #883: Support flexible input sizes.
The issue captures exactly what we see in practice:
In other words: modern inference workloads.
ONNX Runtime WebNN has had to work around this limitation by routing dynamic-shape parts away from WebNN and into WASM execution paths.
It works, but performance is terrible for autoregressive generation because you bounce between fast backend code and slow fallback code in the hottest part of the loop.
This architecture can make demos pass, but it creates significant performance penalties in real autoregressive workloads.
In preliminary runs, keeping decode on one backend avoids the repeated fallback round-trips that dominate token latency.
So instead of accepting that, we decided to push WebNN support further.
I started prototyping this while on vacation in Malta.
What began as a small experiment quickly turned into deep changes across three repositories: converter internals, runtime shape validation, and KV-cache plumbing.
The work happened across:
webnn-graph: ONNX lowering and dynamic-input metadata supportrustnn (tarek-flexible-input): dynamic dimensions in graph/runtime + checked executionpywebnn: Python demos and loading-from-Hub workflowsI also made sure to surface rustnn through Python (pywebnn) very early.
Most ML engineers live in Python land, and I wanted this work to be reachable by
that ecosystem immediately: easier model validation, easier parity checks against
transformers, and faster feedback from people who already ship models every day.
The key was to support bounded dynamic dimensions end to end.
Why bounded and not fully dynamic? Because many backends still need strong compile-time guarantees for validation, allocation, and kernel planning. Fully unbounded shapes are hard to optimize and hard to validate safely.
Bounded dynamic dimensions are the practical compromise: keep symbolic runtime flexibility, but define a maximum range so memory and execution planning remain deterministic.
This allows the full autoregressive decode loop to stay inside the WebNN backend, without bouncing into slower fallback paths.
This is also better than common alternatives:
For example, you can bound a sequence dimension to 2048 tokens: large enough for real prompts, still finite for backend planning and allocation.
In rustnn, tensor dimensions can now be static values or dynamic descriptors
with a name and a max size, then checked at runtime:
{
"inputs": {
"x": {
"dataType": "float32",
"shape": [
{ "name": "batch", "maxSize": 16 },
128
]
}
}
}
In webnn-graph, ONNX conversion can preserve unresolved input dynamics while
still lowering shape-driving expressions needed by WebNN.
That lets us keep flexibility where possible while still emitting valid WebNN graphs.
With flexible inputs supported end to end, SmolLM-135M converts cleanly: no shape rewriting hacks, no per-length exports, no WASM fallback in the decode loop. The artifacts are published here:
Then I built a Python demo in
pywebnn/examples/smollm_from_hub.py
that:
model.webnn, model.weights, and manifest from the Hubtokenizer.jsontransformersA few extracts from that demo:
The demo defaults to the Hub-hosted WebNN artifacts:
DEFAULT_MODEL_ID = "tarekziade/SmolLM-135M-webnn"
model_files = resolve_model_files(args.model_id, force=args.force_download)
graph = webnn.MLGraph.load(
model_files["graph"],
manifest_path=model_files["manifest"],
weights_path=model_files["weights"],
)
past_key_values holds the growing KV-cache tensors returned by the previous
step. The decode loop feeds them back on every token:
def run_step(token_id: int, position: int) -> np.ndarray:
inputs = {
"input_ids": np.array([[token_id]], dtype=np.int64),
"position_ids": np.array([[position]], dtype=np.int64),
"attention_mask": np.ones((1, position + 1), dtype=np.int64),
**past_key_values,
}
outputs = context.compute(graph, inputs)
...
The demo can also run a transformers baseline and fail fast on divergence:
if args.compare_transformers:
hf_generated, hf_text, hf_prompt_ids = run_transformers_baseline(...)
...
if generated_text != hf_text:
print("[ERROR] WebNN and transformers generated different text output")
sys.exit(1)
Correctness checks against transformers are critical. Performance
improvements mean nothing if generation diverges.
pywebnn) accelerates model validation and ecosystem feedbackFlexible inputs will likely be important if WebNN is to support real LLM workloads.
Static graphs alone are not enough for modern inference. Bounded flexibility is the pragmatic bridge.
And while this work pushes WebNN forward, we are also giving a lot of love to the TensorRT backend these days, because high-performance local inference matters just as much as API design.
They don’t crash, they don’t fail tests, and they don’t stand out in code review. They just quietly turn into cliffs when the input size grows.
This post is about one such performance fix in transformers, what it revealed,
and a small experiment that came out of it: LoopSleuth, a local LLM-powered
complexity scanner.
While working on transformers, I fixed a performance issue in
convert_slow_tokenizer.py
that took a tokenizer conversion step from 4 minutes down to ~1 second
when running on very large vocabularies (100k+ tokens).
This started when CI flagged test_voxtral_tokenizer_converts_from_tekken as
the slowest test in the suite.
The test loads mistralai/Voxtral-Mini-3B-2507 and forces the fallback path to
TokenizersBackend.
That fallback triggers the slow→fast tokenizer conversion step — and that
conversion was doing repeated .index() lookups inside a sort key, turning
large vocabularies into a performance cliff.
The root cause was a classic scaling trap.
# BEFORE (simplified excerpt)
for rank, token in enumerate(bpe_ranks):
local = sorted(
local,
key=lambda x: (
bpe_ranks.index(x[0]),
bpe_ranks.index(x[1]),
),
)
(Simplified excerpt — the key issue is the repeated .index() inside the sort
key.)
At first glance this looks harmless.
But list.index() is O(n).
And the real killer is that it happens inside a sorted() key function.
Sorting local means computing the key for every element, and each key performs
two linear searches through bpe_ranks: sorted() calls the key function once
per element (O(m)), and each key calls .index() twice (O(n)), so the total
becomes O(m·n) — often a scaling trap when m and n are both large.
# AFTER (reduces key computation from O(n) to O(1))
token_to_rank = {token: rank for rank, token in enumerate(bpe_ranks)}
for rank, token in enumerate(bpe_ranks):
local = sorted(
local,
key=lambda x: (
token_to_rank[x[0]],
token_to_rank[x[1]],
),
)
The optimization is simple:
This doesn’t eliminate all sorting work (the outer loop still sorts repeatedly), but it removes the quadratic lookup cost that was dominating runtime.
The takeaway wasn’t just “use dicts” — it was that asymptotic traps often hide in perfectly valid Python idioms.
After landing that fix, I kept wondering:
How many other places in the codebase have the exact same pattern?
This wasn’t a correctness issue:
And none of the linting tools I normally rely on flagged it.
Ruff’s PERF rules catch obvious constructs like unnecessary list copies, but
they don’t reason about .index() inside a sort key.
In theory, a linter could detect patterns like:
.index() inside loops.index() inside sort keysBut most rule-based linters avoid making strong claims about asymptotic complexity.
That’s a reasonable trade-off: linters are fast, deterministic, and low-noise — but they often miss scaling issues unless you add very specific custom rules.
This is where I started wondering whether an LLM could help fill the gap.
As an experiment, I ran Claude Code over the repository with one question:
Find quadratic complexity patterns similar to the tokenizer converter bug.
The result was surprisingly useful.
It scanned ~3,000 Python functions across the codebase in a few minutes and flagged ~20 instances of the same anti-pattern:
.index() inside loops.index() inside sort keysAbout half were genuine hot-path candidates; others were technically quadratic but not performance-critical in practice.
For example:
ymls.sort(key=lambda x: results.index(x[:-4]))
Or:
aspect_ratios_ids[i, j] = supported_aspect_ratios.index(ratio)
All fixable with the same technique:
lookup = {val: idx for idx, val in enumerate(reference)}
This report was a great proof of concept.
But it relied on a large hosted model.
Instead of running a massive model in the cloud, I wanted to know:
That’s how I ended up hacking together a small prototype I called LoopSleuth.
My first instinct was to build this as a Python script on top of
transformers itself.
But I wanted this experiment to be:
A single static binary makes it easy to drop into CI, like Ruff.
And honestly, I also wanted an excuse to explore the Rust ecosystem that powers tools like Ruff and Ty.
So LoopSleuth is written in Rust and uses:
rustpython-parser to extract functionsllama.cpp bindings for local inferenceIn practice, a small model like Qwen2.5-Coder 3B (Q4) already gives surprisingly good results for this narrow task.
LoopSleuth is a CLI tool that:
Does this contain patterns that may scale quadratically?
If the model answers “QUADRATIC”, it also asks for an optimization suggestion.
This framing treats complexity as a heuristic warning (like a linter) rather than a mathematical proof.
The prompt is deliberately simple and constrained:
Classify this function as OK or QUADRATIC.
Look for list.index(), nested loops, or linear operations inside loops.
Return only one word: OK or QUADRATIC.
This makes the model focus on structural patterns rather than trying to perform full dataflow analysis, and the constrained output format makes parsing reliable.
Example output:
⚠️ Functions with O(n²) complexity: 5
🔴 QUADRATIC COMPLEXITY DETECTED IN:
• bubble_sort
• find_duplicates
• remove_elements
• string_concatenation
• nested_comparison
Because it’s a CLI, it can be used in a few practical ways:
Before building anything, I tried the usual suspects.
Tools like Ruff, Pylint, and performance-focused plugins can catch a lot:
consider-using-join)PERF rules inspired by PerflintBut none of the linters I tried really caught the pattern that triggered this whole experiment:
.index() lookups inside loops.index() inside sort key functionsThese tools are excellent at enforcing specific rules, but they generally don’t try to answer:
“Does this function scale quadratically with input size?”
That gap is what made the LLM approach interesting to explore.
One thing I wanted to sanity-check early was whether existing linters would catch the same issues.
So I built a small test file with a handful of intentionally quadratic
functions (nested loops, .remove() in loops, string concatenation, etc.) and
ran:
--select ALL)The results were pretty stark:
| Tool | Detects .index() in loop? |
Reports complexity? |
|---|---|---|
| Ruff | ❌ | ❌ |
| Pylint | ❌ | ❌ |
| LoopSleuth | ✅ | ✅ (heuristic) |
LoopSleuth flagged all 5 quadratic functions, while Ruff and Pylint flagged plenty of style and quality issues but neither directly reported algorithmic complexity problems.
This isn’t really a criticism of those tools — they’re simply not designed for that job.
I wrote up the full comparison here:
To be clear, there may be ways to approximate some of these checks with custom rules or plugins, and linters remain the first line of defense for code quality.
LoopSleuth is just exploring a different axis: scaling behavior.
LoopSleuth is not a replacement for linters.
It’s a small experiment.
Traditional linters like Ruff or Pylint excel at catching specific code smells. But most scaling bugs don’t come from a single construct. They come from composition:
Rule-based linters struggle to infer:
.index() is inside a hot path”LLMs, even small ones, can often reason about these patterns more directly.
That said, LoopSleuth runs against isolated Python functions one by one, which means it doesn’t yet understand:
Like any heuristic tool, LoopSleuth has trade-offs:
False positives:
False negatives:
The accuracy depends heavily on prompt design and model choice.
Important: LoopSleuth is a screening tool, not a replacement for profiling or benchmarking. It flags patterns that may cause issues, but only real measurements can confirm actual performance problems.
More broadly, I’m interested in whether this approach can extend beyond complexity analysis to other classes of performance issues.
One direction would be to build a small library of prompts for:
And in an ideal world, we could fine-tune a small model (like Qwen2.5-Coder 3B) to specialize on this kind of performance reasoning.
If this experiment proves useful, here are some directions worth exploring:
Right now LoopSleuth is a proof of concept, but these extensions could make it practical for real codebases.
LoopSleuth started as a simple question:
Could we catch quadratic complexity bugs automatically?
The answer is: not perfectly.
But even a small local model can spot surprising amounts of hidden O(n²) behavior.
And as codebases grow — especially ones like transformers — performance traps
scale with them.
LoopSleuth is a small experiment toward making complexity visible earlier.
If you’re interested, the project is here:
If you have examples of hidden scaling bugs or want to contribute detection patterns, I’d love to collect them as test cases. Feel free to try it locally or open an issue.
]]>I recently had Claude help me with a substantial refactoring task: splitting a monolithic Rust project into multiple workspace repositories with proper dependency management. The kind of task that’s tedious, error-prone, and requires sustained attention to detail across hundreds of files. When it was done, I asked Claude to analyze the session: how much it cost, how long it took, and how long a human developer would have taken.
The answer surprised me. Not because AI was faster or cheaper (that’s expected), but because of how much faster and cheaper.
The work involved:
This is real work. Not a toy problem, not a contrived benchmark. The kind of multi-day slog that every engineer has faced: important but tedious, requiring precision but not creativity.
Total: approximately 3.5 hours across two sessions
Total tokens: 72,146 tokens
Estimated marginal cost: approximately $4.95
This is the marginal execution cost for this specific task. It doesn’t include my Claude subscription, the time I spent iterating on prompts and reviewing output, or the risk of having to revise or fix AI-generated changes. For a complete accounting, you’d also need to consider those factors, though for this task they were minimal.
Conservative estimate: 2-3 days (16-24 hours)
This is my best guess based on experience with similar tasks, but it comes with uncertainty. A senior engineer deeply familiar with this specific codebase might work faster. Someone encountering similar patterns for the first time might work slower. Some tasks could be partially templated or parallelized across a team.
Breaking down the work:
Even if we’re generous and assume a very experienced developer could complete this in 8 hours of focused work, the time and cost advantages remain substantial. The economics don’t depend on the precise estimate.
These numbers compare execution time and per-task marginal costs. They don’t capture everything (platform costs, review time, long-term maintenance implications), but they illustrate the scale of the difference for this type of systematic refactoring work.
The efficiency gains weren’t magic. They came from specific characteristics of how AI approaches systematic work:
No context switching fatigue. Claude maintained focus across three repositories simultaneously without the cognitive load that would exhaust a human developer. No mental overhead from jumping between files, no “where was I?” moments after a break.
Instant file operations. Reading and writing files happens without the delays of IDE loading, navigation, or search. What takes a human seconds per file took Claude milliseconds.
Pattern matching without mistakes. Updating thousands of import statements consistently, without typos, without missing edge cases. No ctrl-H mistakes, no regex errors that you catch three files later.
Parallel mental processing. Tracking multiple files at once without the working memory constraints that force humans to focus narrowly.
Documentation without overhead. Generating comprehensive, well-structured documentation in one pass. No switching to a different mindset, no “I’ll document this later” debt.
Error recovery. When workspace conflicts or dependency issues appeared, Claude fixed them immediately without the frustration spiral that can derail a human’s momentum.
Commit message quality. Detailed, well-structured commit messages generated instantly. No wrestling with how to summarize six hours of work into three bullet points.
AI wasn’t universally faster. Two areas stood out:
Initial codebase exploration. Claude spent time systematically understanding the structure before implementing. A human developer might have jumped in faster with assumptions (though possibly paying for it later with rework).
User preference clarification. Some back-and-forth on git dependencies versus crates.io, version numbering conventions. A human working alone would just make these decisions implicitly based on their experience.
These delays were minimal compared to the overall time savings, but they’re worth noting. AI coding isn’t instantaneous magic. It’s a different kind of work with different bottlenecks.
Let me restate those numbers because they still feel surreal:
For this type of task, these are order-of-magnitude improvements over solo human execution. And they weren’t achieved through cutting corners or sacrificing immediate quality. The tests passed, the documentation was comprehensive, the commits were well-structured, the code compiled cleanly.
That said, tests passing and documentation existing are necessary but not sufficient signals of quality. Long-term maintainability, latent bugs that only surface later, or future refactoring friction are harder to measure immediately. The code is working, but it’s too soon to know if there are subtle issues that will emerge over time.
This creates strange economics for a specific class of work: systematic, pattern-based refactoring with clear success criteria. For these tasks, the time and cost reductions change how we value engineering effort and prioritize maintenance work.
I used to avoid certain refactorings because the payoff didn’t justify the time investment. Clean up import statements across 50 files? Update documentation after a restructure? Write comprehensive commit messages? These felt like luxuries when there was always more pressing work.
But at $5 marginal cost and 3.5 hours for this type of systematic task, suddenly they’re not trade-offs anymore. They’re obvious wins. The economics shift from “is this worth doing?” to “why haven’t we done this yet?”
Before the “AI will replace developers” crowd gets too excited, let me be clear about what this data doesn’t show:
This was a perfect task for AI. Systematic, pattern-based, well-scoped, with clear success criteria. The kind of work where following existing patterns and executing consistently matters more than creative problem-solving or domain expertise.
AI did not:
The task was pure execution. Important execution, skilled execution, but execution nonetheless. A human developer would have brought the same capabilities to the table, just slower and at higher cost.
I keep thinking about that 85-90% time reduction for this specific type of task. Not simple one-liners where AI already shines, but systematic maintenance work with high regularity, strong compiler or test feedback, and clear end states.
Tasks with similar characteristics might include:
Many maintenance tasks are messier: ambiguous semantics, partial test coverage, undocumented invariants, organizational constraints. The economics I observed here don’t generalize to all refactoring work. But for the subset that is systematic and well-scoped, the shift is significant.
All the work that we know we should do but often defer because it doesn’t feel like progress. What if the economics shifted enough for these specific tasks that deferring became the irrational choice?
I’m not suggesting AI replaces human judgment. Someone still needs to decide what “good” looks like, validate the results, understand the business context. But if the execution of systematic work becomes 10x cheaper and faster, maybe we stop treating certain categories of technical debt like unavoidable burdens and start treating them like things we can actually manage.
There’s one cost the analysis didn’t capture: my time. I wasn’t passive during those 3.5 hours. I was reading Claude’s updates, reviewing file changes, answering questions, validating decisions, checking test results.
I don’t know exactly how much time I spent, but it was less than the 3.5 hours Claude was working. Maybe 2 hours of active engagement? The rest was Claude working autonomously while I did other things.
So the real comparison isn’t 3.5 AI hours versus 16-24 human hours. It’s 2 hours of human guidance plus 3.5 hours of AI execution versus 16-24 hours of human solo work. Still a massive win, but different from pure automation.
This feels like the right model: AI as an extremely capable assistant that amplifies human direction rather than replacing human judgment. The economics work because you’re multiplying effectiveness, not substituting one for the other.
Five dollars marginal cost. Three and a half hours. For systematic refactoring work that would have taken me days and cost hundreds or thousands of dollars in my time.
These numbers make me think differently about certain kinds of work. About how we prioritize technical debt in the systematic, pattern-based category. About what “too expensive to fix” really means for these specific tasks. About whether we’re approaching some software maintenance decisions with outdated economic assumptions.
I’m still suspicious of broad claims that AI fundamentally changes how we work. But I’m less suspicious than I was. When the economics shift this dramatically for a meaningful class of tasks, some things that felt like pragmatic trade-offs start to look different.
The tests pass. The documentation is up to date. And I paid less than the cost of a fancy coffee drink.
Maybe the skeptics and the enthusiasts are both right. Maybe AI doesn’t replace developers and maybe it does change some things meaningfully. Maybe it just makes certain kinds of systematic work cheap enough that we can finally afford to do them right.
One caveat worth noting: these economics depend on Claude Sonnet 4.5 at January 2026 pricing. Model pricing can change, model performance can regress or improve with updates, tool availability can shift, and organizational data governance constraints might limit what models you can use or what tasks you can delegate to them.
For individuals and small teams, this might not matter much in the short term. For larger organizations making long-term planning decisions, these factors matter. The specific numbers here are a snapshot, not a guarantee.
The latest iterations of Claude Code are genuinely impressive. Not in a flashy
demo way, but in the quiet, dangerous way where you suddenly realize you have
delegated large parts of your thinking to it. This post is about that
experience, how Claude helped me build rustnn, what worked remarkably well,
and where I had to consciously pull myself back.
For rustnn, I leaned heavily on Claude Code. The quality of the generated Rust was consistently high. Beyond producing correct syntax, it reasoned about what the code was supposed to do. It was context-aware in a way that made iterative design feel natural. I could ask for refactors, architectural changes, or alternative approaches, and get answers that actually respected the existing codebase and long-running tests.
This mirrors what many developers have been reporting toward the end of 2025. Claude Code’s agent-oriented design and large-context reasoning make it particularly strong for repository-wide work: multi-file refactors, non-trivial debugging sessions, and architectural changes that need to fit an existing mental model. Compared to Codex-style systems, which still shine for fast edits and local completions, Claude tends to perform better when the task requires sustained reasoning and understanding of project-wide constraints.
Anthropic’s recent Claude releases have reinforced that positioning. Improvements in long-context handling, reasoning depth, and agentic workflows make it easier to treat Claude as something closer to a collaborator than an autocomplete engine.
The turning point for me was when I stopped treating Claude like a chat bot and started treating it like a constrained agent.
That is where CLAUDE.md comes in.
I stumbled upon an excellent LangChain article on how to turn Claude Code into a domain-specific coding agent.
It clicked immediately. Instead of repeatedly explaining the same constraints, goals, and conventions, I encoded them once. Rust style rules. Project intent. Explicit boundaries. How to react to test failures.
The effect was immediate. Output quality improved, and the amount of back-and-forth dropped significantly. Claude stopped proposing things that were clearly out of scope and started behaving like someone who had actually read and understood the project.
For rustnn, I went one step further and anchored development around WPT conformance tests. That gave both Claude and me a shared, objective target. Tests either pass or they do not. No bikeshedding.
Tweaking CLAUDE.md quickly revealed itself as a never-ending process. There are plenty of articles describing different approaches, and none of them are definitive. The current direction seems to be layering information across multiple files, structuring project documentation so it is optimized for agent consumption while remaining readable for humans, and doing so without duplicating the same knowledge in multiple places.
That balance turns out to be just as important as the model itself.
There is a trap though, and it is a subtle one.
Once Claude is good enough, you start routing everything through it.
It feels efficient, but it is not free. Each interaction has a cost, and when you are in a tight edit-build-test loop, those costs add up fast. Worse, you start outsourcing mechanical thinking that you should probably still be doing yourself.
I definitely fell into that trap.
The solution, for me, was to drastically reduce how much I talk to Claude, and to stop using its prompt environment as a catch-all interface to the project.
Claude became an extra terminal. One I open for very specific tasks, then close. It is not a substitute for my own brain, nor for the normal edit–build–test loop.
Reducing the context window is also critical. A concrete example is Python tracebacks. They are verbose, repetitive, and largely machine-generated noise. Sending full tracebacks back to the model is almost always wasteful.
That is why I added a hook to rewrite them on the fly into a compact form.
The idea is simple: keep the signal, drop the boilerplate. Same information, far fewer tokens. In practice, this not only lowers costs, it often produces better answers because the model is no longer drowning in irrelevant frames and runtime noise. On Python-heavy codebases, this change alone reduced my usage costs by roughly 20%.
Pre-compacting inputs turned out to be one of the most effective cost-control strategies I have found so far, especially when combined with a more deliberate, intentional way of interacting with the model.
Another pain point is session amnesia. You carefully explain design decisions, trade-offs, and long-term goals, only to repeat them again tomorrow.
A well-crafted CLAUDE.md mitigates part of this problem. It works well for
static knowledge: coding style, project constraints, architectural boundaries,
and things that rarely change. It gives Claude a stable baseline and avoids a
lot of repetitive explanations.
But it does not capture evolving context.
It does not remember why a specific workaround exists, which approach you rejected last week, or what subtle behavior a particular test exposed yesterday. As soon as the session ends, that knowledge is gone, and you are back to re-teaching the same mental model.
This is where cross-session, cross-project memory becomes interesting.
I am currently experimenting with claude-mem
The idea is simple but powerful: maintain a centralized, persistent memory that is automatically updated based on interactions. Instead of manually curating context, relevant facts, decisions, and preferences are summarized and carried forward. Over time, this builds a lightweight but durable understanding of how you work and how your projects evolve.
Compared to CLAUDE.md, this kind of memory is dynamic rather than declarative.
It captures intent, not just rules. It also scales across projects, which
matters when you jump between repositories that share design philosophy,
tooling, or constraints.
It is still early, and it is not magic. You need to be careful about what gets remembered and how summaries are formed. But the direction feels right. Persistent memory reduces cognitive reset costs, shortens warm-up time, and makes the interaction feel less like starting over and more like continuing a conversation you paused yesterday.
That difference adds up.
Claude Code is good. Very good. Good enough that you need discipline to use it well.
With a tuned CLAUDE.md, clear test-driven goals like WPT conformance, and some
tooling to reduce noise and cost, it becomes a powerful accelerator. Without
that discipline, it is easy to overuse it and slowly burn budget on things you
already know how to do.
I do not think this replaces engineering skill. If anything, it amplifies both good and bad habits. The trick is to make sure it is amplifying the right ones.
*The title is a deliberate reference to “All your base are belong to us.” The grammar is broken on purpose. It is a joke, but also a reminder that when tools like Claude get this good, it is easy to give them more control than you intended
]]>As AI moves closer to users, openness becomes a question of power. Who can inspect these systems? Who can adapt them? And who ultimately controls how they evolve?
The web offers a useful reference point. Open source software and open standards turned the World Wide Web into shared infrastructure rather than a proprietary stack owned by a single company or government, even if many tried to enclose parts of it. That openness was not accidental. It shaped who could participate, compete, and be held accountable.
Open source AI is often reduced to code availability. In practice, and as the Open Source Initiative (OSI) emphasizes in its Open Source AI Definition, it is about concrete freedoms.
An open source AI system can be used, studied, modified, and shared. Studying means inspecting behavior, limits, and failure modes. Modifying means adapting models to new domains, languages, or constraints. Sharing means deploying systems without being locked to a single vendor or API.
These freedoms must apply not only to code, but also to models, weights, and the tooling required to run them. Without that access, reuse is brittle and understanding remains shallow.
Open source enables verification, reproducibility, and portability. It allows systems to be audited, adapted, and redeployed independently. In a field defined by cost, scale, and complexity, these are not luxuries. They are prerequisites for agency.
Open access does not eliminate power imbalances. Compute, data, and expertise still matter. But it preserves the possibility of independent action, which is often the difference between participation and dependency.
Open source alone is not enough. Open standards define shared interfaces that allow independently built systems to work together.
The web proved this model at global scale. By separating interfaces from implementations, standards enabled competition without fragmentation. In AI, standards around model formats, inference interfaces, evaluation, and data documentation can lower switching costs and prevent ecosystems from hardening into silos controlled by a few gatekeepers.
Without standards, “openness” risks collapsing into a collection of incompatible artifacts, each tied to its own platform or service.
Some infrastructure works best when treated as a common good. The web’s resilience came from the fact that no single actor owned its foundations.
AI is on track to become similar infrastructure. The question is not whether it will be powerful, but whether it will be governable.
If core models, datasets, and interfaces are only accessible through proprietary APIs and cloud platforms, then “AI adoption” will mostly mean dependency. Choice will be limited to pricing tiers, usage caps, and terms of service.
Not everything should be owned and monetized by a small number of companies. Projects like Mozilla’s Common Voice show that shared assets can be built and maintained in the open, at meaningful scale.
Shared infrastructure also depends on shared spaces. Platforms like Hugging Face play a critical role by enabling collaboration around models, datasets, and tools, and by lowering the barrier to participation in open AI ecosystems.
Open source and open standards are not about nostalgia or ideology. They are about keeping the option to walk away. To inspect. To fork. To rebuild.
Once that option is gone, it is rarely recovered.
What started as an experiment to gain a deeper understanding of WebNN quickly grew into something more substantial: a working implementation that is now very close to being a usable library.
I began this project after returning from TPAC, convinced that WebNN is the future of AI in the browser, and that Firefox needs to catch up with the work that has already been done in Chromium.
We are likely still months away from matching the level of maturity Chromium has achieved over several years of development. However, in just a few weeks I was able to make significant progress thanks to a few key factors:
All of these factors made it surprisingly easy to build the library quickly using Claude Code. Once I had enumerated the 95 operators that needed to be implemented, the workflow for each one was essentially the same:
Claude consistently performed well. I was able to build a library that would normally have taken me months to write on my own. When something failed, narrowing down the problem was straightforward by iterating between the specification and the tests.
Because most of the work revolves around graph conversion and orchestrating existing inference libraries, the code generated by Claude is generally clean and easy to reason about.
The Chromium implementation was also a huge help when I started to get into weird corner cases, especially around CoreML. That code base has been developed over the years with people directly involved in the spec.
I have started adding performance tests, and there will likely be some manual follow-up work, but reaching a functional implementation so quickly is already a major milestone.
These days, adding a new API to Firefox usually means creating a Rust library that is vendored into the tree and bound to Gecko using cbindgen, unless there is an existing C++ library that already fits the need.
This gradual “oxidation” of Firefox started years ago, and major features such as WebGPU have followed this model. Gecko is still a large C++ codebase, and integrating a Rust library is not trivial, but implementing something like WebNN outside the browser engine has a major advantage: it allows a much broader community to contribute. We are already seeing the benefits of this approach with wgpu.
I am not going to rehash the Rust vs. C++ debate. There is no shortage of material on why Rust has become an attractive choice for systems programming.
My first instinct was to see whether Chromium’s WebNN implementation could be reused. In practice, that turned out to be impractical. The code is deeply intertwined with Blink and its IPC layers, making it very difficult to extract reusable components in a clean way.
We also evaluated webnn-native, a C++ implementation developed within the Web Machine Learning community. While promising, the project had been effectively stalled for about two years and lacked support for the most recent inference backends. Extending it was an option, but it quickly became clear that a fresh Rust implementation would be both faster to iterate on and a better architectural fit for Gecko.
In the end, this is good news for the Web and for WebNN. An independent implementation helps validate the specification, exposes ambiguities earlier, and ultimately makes the standard stronger.
Finally, building the core in Rust makes it trivial to expose a Python API on top of it, which opens the door to experimentation and adoption by the broader ML community.
rustnn follows a key principle: graph compilation creates a platform-independent representation; backend conversion happens at execution time.
This follows the same logic as Chromium and is a great way to make sure we can add more backends in the future.
flowchart TD
RustNN["RustNN"]
WebNNGraph["WebNN Graph"]
Executors["Executors"]
Converter["Converter"]
ONNXRuntime["onnx runtime"]
CoreMLExec["CoreML"]
TensorRT["TensorRT"]
ONNXGraph["ONNX Graph"]
CoreMLGraph["CoreML Graph"]
RustNN --> WebNNGraph
RustNN --> Executors
WebNNGraph --> Converter
Converter --> ONNXGraph
Converter --> CoreMLGraph
Executors --> ONNXRuntime
Executors --> CoreMLExec
Executors --> TensorRT
In the library, we do an initial pass on the WebNN graph to produce an intermediate representation, then we pick a converter to turn that graph into another graph that can run with an AI library. And there are executors that can run the graph using those external libraries.
This is a very powerful design. For instance, I am playing with the TensorRT-RTX library that can be used to efficiently run AI on NVIDIA GPUs and that library has full support for ONNX graphs. This means we can run networks in rustnn using the ONNX converter combined with the TensorRT executor.
I picked CoreML and ONNX as my first target runtimes because I work on a MacBook, and because they are both implemented in Chromium.
Chromium uses ONNX on Windows because that library now ships with the latest Windows 11, and it falls back to DirectML. It also has a CoreML implementation on macOS.
So I went ahead and built both CoreML and ONNX as converters and executors until I could make the image classification demo work with the Python binding.
Next, I started to add TensorRT as an executor for Windows with NVIDIA
GPUs. That one is a work in progress because I have to work on another
Windows computer and I am slower in that environment. But it’s technically
already working. I started the trtx-rs Rust library to bind TensorRT, since the
existing Rust binding was 5 years old.
rustnn exposes a Python binding (PyWebNN) that implements the W3C WebNN API on top of the Rust core. You can use it for graph validation, conversion (ONNX/CoreML) and execution of neural networks.
Installation:
# Install from PyPI with bundled ONNX Runtime (v0.4.0+)
pip install pywebnn
# Or build from source with all backends (ONNX + CoreML)
git clone https://github.com/tarekziade/rustnn.git
cd rustnn
make python-dev
source .venv-webnn/bin/activate
Version 0.4.0+ includes bundled ONNX Runtime for immediate execution support. No additional dependencies needed!
This is a very small example adapted from the repo example examples/python_matmul.py.
It shows the minimal flow:
Note: Use accelerated=False for CPU-only execution, or accelerated=True with power_preference="high-performance" for GPU acceleration.
# PyWebNN — tiny matmul example
import numpy as np
import webnn
# 1) create ML instance and context (CPU execution here)
ml = webnn.ML()
ctx = ml.create_context(power_preference="default", accelerated=False)
# 2) build a simple graph: Y = A @ B
builder = ctx.create_graph_builder()
A = np.array([[1., 2.], [3., 4.]], dtype=np.float32)
B = np.array([[5., 6.], [7., 8.]], dtype=np.float32)
a = builder.constant(A) # constant input A
b = builder.constant(B) # constant input B
y = builder.matmul(a, b) # matmul node
graph = builder.build({"output": y}) # compile the graph
# 3) run the graph and print result
result = ctx.compute(graph, {}) # returns dict of outputs
print("Y =", result["output"])
What happens:
Paul Adenot is currently extending the Firefox AI Runtime platform to add a new specialized process to run against GPUs for the WebSpeech implementation, and the WebNN API will use it when it’s added in the browser.
In the meantime I have built a patch that adds the WebNN JS API in Firefox and executes it directly in the content process, which is a big security hole.
But it was a good way to start figuring out all the pieces, in particular how
to bind the Rust library into the C++ layer using cbindgen, and how to create
the WebIDL interface to provide the JS API.
The current series of patches is just a proof of concept, but I already have a fully functional demo of all basic operators, and a clone of the WebNN JS MobileNetV2 image classifier demo — see the video.
The WebNN implementation spans six distinct layers:
navigator.ml, MLContext, MLGraphBuilder, MLGraph, MLOperand, MLTensor)dom/webnn/dom/webnn/rustnn_bridge/third_party/rust/rustnn/… and has the following flow:
Graph Building Phase:
sequenceDiagram
participant JS as "JavaScript"
participant CPP as "C++ (MLGraphBuilder)"
participant FFI as "Rust FFI Bridge"
participant RUST as "Rustnn Library"
JS->>CPP: createContext()
CPP->>FFI: rustnn_context_create()
FFI->>RUST: Context::new()
RUST-->>FFI: Context handle
FFI-->>CPP: context_id
CPP-->>JS: MLContext
JS->>CPP: new MLGraphBuilder(context)
CPP-->>JS: MLGraphBuilder
JS->>CPP: input("x", shape, dataType)
CPP-->>JS: MLOperand
JS->>CPP: add(a, b)
CPP-->>JS: MLOperand
JS->>CPP: build({ output: operand })
CPP->>FFI: rustnn_graph_build(ops_json)
FFI->>RUST: GraphBuilder::build()
Note right of RUST: Convert to backend format
Note right of RUST: Create backend session
RUST-->>FFI: Graph handle
FFI-->>CPP: graph_id
CPP-->>JS: MLGraph
Inference Phase:
sequenceDiagram
participant JS as "JavaScript"
participant CPP as "C++ (MLContext)"
participant FFI as "Rust FFI Bridge"
participant RUST as "Rustnn Library"
participant BE as "Backend (ONNX/CoreML)"
JS->>CPP: compute(graph, inputs, outputs)
CPP->>FFI: rustnn_graph_compute(graph_id, inputs, outputs)
FFI->>RUST: Graph::compute()
RUST->>BE: session.run()
Note right of BE: Execute operations
BE-->>RUST: Output tensors
RUST-->>FFI: Results
FFI-->>CPP: Output data
CPP-->>JS: Promise resolves
Again, this is not the final design since we need to run inference in a separate process and have an IPC layer between the C++ code and the Rust bridge.
rustnn started as a way for me to really understand WebNN, but it quickly turned into a convincing proof that the specification is solid, implementable, and ready to grow beyond a single browser engine. Having an independent implementation is healthy for the Web, and rustnn shows that WebNN can be built as a reusable, backend-agnostic library rather than something deeply tied to a single browser architecture.
This project is also my first substantial experience with Claude Code, and it fundamentally changed the pace at which I could work. Implementing nearly a hundred operators, wiring multiple backends, and validating everything against WPT would normally be a multi-month effort. With a strong spec, good tests, and a capable AI agent, it became an iterative and surprisingly enjoyable process. The result is not throwaway code, but a clean foundation that can be extended, optimized, and reviewed by others.
I am very optimistic about WebNN’s future in Firefox and on the Web in general. With rustnn and pywebnn, my hope is to make it easier for browser engineers, ML practitioners, and researchers to experiment, contribute, and push the ecosystem forward. There is still a lot to do, especially around performance, security, and process isolation, but the path forward is now much clearer.
In this second part, I focus on architecture. I explain why I decided to move away from my initial design, what I learned from that first implementation, and why I ultimately settled on a prefix-conditioning + LoRA approach.
This choice is driven by practical constraints. For alt-text generation, the goal is not exhaustive visual understanding, but a short, reliable sentence that conveys the essence of an image to visually impaired users. Within that scope, prefix conditioning offers a much simpler model that is easier to train, easier to deploy, and better aligned with accessibility requirements.
More broadly, the PDF.js alt-text project aims to explore how far we can push small, efficient vision–language models for accessibility use cases. Rather than optimizing for peak benchmark scores, the focus is on reliability, fast iteration cycles, limited compute, and deployable models.
DistilViT is intentionally constrained. Smaller models, fewer trainable parameters, and simpler architectures make it possible to experiment rapidly, control bias more carefully through dataset curation, and realistically target on-device or near-device inference scenarios.
My first implementation relied on Hugging Face’s VisionEncoderDecoderModel. Concretely, it paired:
distilgpt2),trained end-to-end using Seq2SeqTrainer.
Conceptually, the architecture looked like this:
flowchart TD
A[Image] --> B["Vision Encoder (ViT)"]
B --> C[Encoder hidden states]
C -->|Cross-attention| D["Decoder (GPT-2)"]
D --> E[Caption]
This worked. GPT-2 generated captions, and the system was usable. I was inspired by The Illustrated Image Captioning Using Transformers, followed that recipe, and reduced the decoder size by using a distilled version of GPT-2.
What I did not fully appreciate at the time was what choosing GPT-2 implied under the hood.
Unlike T5 or BART, GPT-2 is a decoder-only language model. In its original architecture, it does not support cross-attention or encoder hidden states.
So why did this setup work?
Because VisionEncoderDecoderModel.from_encoder_decoder_pretrained() wraps GPT-2 and injects cross-attention layers. This effectively converts GPT-2 into a seq2seq-style decoder by adding encoder–decoder attention blocks and routing the vision encoder outputs through them.
That distinction matters. These cross-attention layers are initialized from scratch, require substantial training signal, and introduce additional state to manage at inference time. Exporting the model and handling caching also become more complex.
This approach is valid, but it turned out to be architecturally heavier than expected for the scale and goals of this project. Training was slower, GPU memory usage was higher, and deployment friction increased.
Models like T5 or BART avoid this injection step because they already contain pretrained cross-attention blocks. However, those blocks were trained to attend to text encoder states and still require fine-tuning to adapt properly to vision features.
At that point, I started looking for an alternative and came across prefix conditioning.
It is worth stepping back and comparing these two approaches without framing one as universally superior.
Cross-attention gives the decoder continuous access to visual features at every generation step. This is extremely powerful for tasks that require fine-grained spatial grounding, OCR, counting, or reasoning over multiple regions in an image.
Prefix conditioning, by contrast, injects visual information once, as a sequence of projected vision tokens prepended to the text embeddings at the beginning of the text. After that, the model relies entirely on standard self-attention.
This leads to clear trade-offs:
For my use case, this trade-off is appropriate. The goal of alt text here is not to enumerate details or perform spatial reasoning, but to produce a single, concise sentence that conveys the overall content of an image to visually impaired users. Captions are short, factual, and descriptive, and they primarily require global visual context rather than continuous visual querying.
Under these conditions, prefix conditioning is often sufficient, while being far easier to train, debug, and deploy than a full encoder–decoder setup.
The architecture I use now looks like this:
flowchart TD
A[Image] --> B["SigLIP Vision Encoder (frozen)"]
B --> C["Projection Head (Linear / MLP)"]
C --> D[Vision embeddings as prefix tokens]
D --> E[Decoder-only LM with LoRA]
E --> F["Caption (25–30 tokens)"]
Instead of asking the decoder to attend to an encoder, I inject the visual information directly into the decoder’s input space as prefix tokens.
The language model only needs standard causal self-attention. Any decoder-only LLM works out of the box, without architectural changes or special forward signatures.
This restores flexibility. I can swap language models freely without touching the vision side.
I apply LoRA adapters to the language model’s attention projection matrices.
In practice, this means:
Training is faster, more stable, and far less memory-intensive. The risk of overfitting drops significantly when working with small datasets.
Deployment also becomes simpler.
There is no cross-attention graph, no encoder cache plumbing, and no exotic export logic. ONNX Runtime becomes a realistic target instead of a constant source of friction.
Cross-attention remains a powerful and sometimes necessary tool. For this project, however, it added complexity without delivering better alt text.
Prefix conditioning gives me:
Initial experiments show that the new architecture produces alt text of comparable quality to the previous one, with only a 1 to 2 percent CLIP score difference when trained on the same datasets. The key difference is training speed, which is roughly five times faster.
Next, my goal is to surpass DistilViT’s current quality by improving the training dataset, while keeping an architecture that is simple, fast to train, and flexible enough to accommodate future decoder models.
Earlier this year, I built the backend for the local alt text generation feature in Firefox. Nearly half of the images on the web still lack alternative text, creating a major accessibility barrier for screen reader users. The goal of this work is straightforward but ambitious: generate high-quality alt text entirely on device, preserving user privacy while improving access to visual content.
The first implementation focused on PDF.js, primarily as a controlled environment to validate the approach. Now that the runtime performance is good enough, the next step is to generalize this capability across the entire browser so that all web images can benefit from meaningful descriptions. Before that generalization, however, improving accuracy is essential.
From a modeling perspective, the system pairs a Vision Transformer (ViT) with DistilGPT-2, a 182-million-parameter language model that fits under 200 MB once quantized. Improving this system involves multiple, often competing dimensions: bias reduction, description accuracy, and inference speed. This post focuses on the data side of the problem, specifically dataset quality and bias. Part 2 will look at model-level improvements for accuracy and performance.
The original image captions contained several recurring issues:
To address this, I used GPT-4o to systematically transform captions from Flickr30k and COCO, removing demographic descriptors that were not visually required. The resulting datasets are available on Hugging Face (Mozilla/flickr30k-transformed-captions-gpt4o) and were used to train the current Firefox local alt text model.
For more background on this initial effort, see the Mozilla Hacks post and the Firefox blog announcement. This is the model that is currently shipping in Firefox.
Qualitative panel testing showed that the transformed captions were generally better received by humans, but that only answered part of the question. What exactly improved, by how much, and what problems remained hidden in the data?
This post documents the second round of work, which focused on building systematic measurement tools to:
When training vision-language models, dataset quality is often treated as a secondary concern compared to architecture or training tricks. In practice, the data is the foundation. If the dataset is biased, noisy, or unbalanced, no amount of fine-tuning will fully compensate.
After the GPT-4o transformation, several open questions remained:
Answering these questions requires more than a single score or benchmark.
I built a dataset quality analysis tool that evaluates four complementary dimensions. The emphasis is on improving the training data itself, rather than compensating for data issues at model time.
CLIP provides a convenient proxy for how well a caption matches its corresponding image. By embedding both modalities and computing cosine similarity, I obtain a rough but useful alignment score.
A key improvement in this round was upgrading from CLIP ViT-B/32 to ViT-L/14 @ 336 px. The larger model produces lower absolute scores, but it is significantly more discriminative, making it easier to separate strong alignments from weak ones.
Interpretation guidelines:
On the transformed dataset, I observe scores of 0.311 with ViT-B/32 (Good) and 0.284 with ViT-L/14 @ 336 px (Fair but more informative).
Removing bias should not come at the cost of semantic drift. To verify this, I used BERTScore with a RoBERTa-large backbone to compare original and transformed captions.
Scores above 0.90 generally indicate that the core meaning is preserved. The transformed dataset achieves 0.904, which falls comfortably in the “excellent” range.
Bias reduction is only meaningful if it can be measured. I tracked mentions of protected attributes across seven categories, including gender, race or ethnicity, nationality, age, religion, sexual orientation, and disability.
By comparing original and transformed captions on the same samples, I can directly quantify the effect of the transformation. On a 1 000-sample evaluation set, gender mentions dropped from 70 percent to zero, race and ethnicity mentions dropped by 97 percent, and nationality mentions were completely eliminated. Age-related terms remain more common, largely because they are often visually relevant, for example when describing children.
Finally, I analyzed object frequency to identify long-tail problems. Using metrics such as the Gini coefficient and Shannon entropy, the tool highlights severe imbalance: thousands of objects appear only a handful of times.
This analysis automatically produces lists of rare objects and sampling weights that can be used for rebalancing during training.
Beyond evaluation, CLIP can also be used to guide training directly. I experimented with a combined loss that adds a CLIP-based alignment term to the standard cross-entropy loss for caption generation.
The intuition is simple: encourage the model to generate captions that are not only fluent, but also visually grounded. Early results suggest modest but consistent gains in CLIP score, at the cost of slower training and higher memory usage.
The quality analysis tool integrates directly into the project’s Makefile:
# Quick test (100 samples)
make quality-report-quick
# Full analysis on test split
make quality-report SPLIT=test
# Custom analysis
make quality-report SPLIT=train MAX_SAMPLES=1000 OUTPUT_DIR=./my_reports
Below is an excerpt from the generated quality report for the full Flickr30k transformed dataset. It illustrates how the metrics come together in practice.
================================================================================
DATASET QUALITY REPORT
================================================================================
Dataset: Mozilla/flickr30k-transformed-captions-gpt4o
Samples: 31 014
IMAGE–TEXT ALIGNMENT (CLIP)
Score: 0.274 ± 0.036 Assessment: FAIR
CAPTION FIDELITY (BERTScore)
Score: 0.899 ± 0.023 Assessment: GOOD
BIAS DETECTION (Original → Transformed)
Gender: 67% → 0%
Race/Ethnicity: 27% → 1%
Nationality: 1% → 0%
Age: 19% → 17%
OBJECT DISTRIBUTION
Gini coefficient: 0.889
Rare classes (<50 samples): 6 210
================================================================================
The report confirms that the GPT-4o transformation is highly effective at removing demographic bias while preserving meaning. At the same time, it surfaces two remaining issues: only fair image–text alignment and severe class imbalance.
The analysis produces the following artifacts:
Directory: quality_reports/
• summary.json - Aggregate metrics in JSON format
• quality_report.txt - Human-readable summary report
• per_example_scores.csv - Per-sample CLIP, BERT, and bias scores
• ranked_by_combined.csv - Samples ranked by combined quality score
• object_counts.csv - Object frequency distribution
• objects_below_50.csv - Rare / underrepresented objects (≤50 samples)
• reweighting_probs.csv - Sampling probabilities for balanced training
• lorenz_curve.png - Object distribution inequality visualization
• top_failures/ - Top failure cases with images and captions
These artifacts make it easy to audit dataset quality, compare runs, and target specific weaknesses.
None of these improvements are shipping yet. They are preparatory steps that make future work safer and more predictable. With solid metrics in place, the next phase is to train improved models, validate gains rigorously, and continue reducing long-tail failures.
The long-term goal remains unchanged: provide high-quality, privacy-preserving alt text for the large fraction of web images that still lack it, and do so in a way that is fair, transparent, and measurable.
The first major challenge was creating a runtime that could run machine learning models directly in Firefox. We needed something that worked across platforms, respected user privacy, and didn’t require sending data to external servers.
We built the Firefox ML inference engine on top of two core technologies: the ONNX runtime for executing models, and Transformers.js to simplify the inference work. The architecture we settled on uses a dedicated content process for inference, keeping it isolated from the main browser process. Remote Settings distributes both the runtime and model configurations, while IndexedDB caches downloaded models locally.
One critical evolution was moving away from WebAssembly to run a pure C++ ONNX runtime under Transformers.js. This shift gave us significantly better performance and tighter integration with Firefox’s internals. Getting this right required deep systems-level work, and I was fortunate to work with fantastic engineers like Paul Adenot and Serge Guelton who brought the expertise needed to make it happen.
This multi-process design was crucial. It gave us stability, security, and the ability to update models without shipping new browser versions. We also created our own model hub, giving us control over model distribution while still supporting Hugging Face for developers who want broader model selection.
The API we exposed is deliberately simple. Developers create an engine instance with a task name and model ID, then run inference either synchronously or with streaming output. Behind the scenes, Firefox handles downloading models, managing cache, and choosing the right backend.
With the runtime in place, we needed a real feature to prove it worked. PDF.js alt text generation became that first end-to-end project, and I have written about it in detail before. But looking back now, it was more than just a feature. It was the template for everything that came after.
We chose a Vision Transformer paired with a distilled GPT-2 decoder, compressed to 180 million parameters and under 200MB on disk. The model runs in a couple of seconds on a laptop, generates descriptions locally, and never sends your PDF content anywhere. This shipped in Firefox 130, and it set the standard for how we approach AI: small models, local execution, and privacy by default.
The harder work was not the model architecture. It was dealing with biased training data and building a validation pipeline. COCO and Flickr30k datasets carried gender stereotypes and cultural assumptions. We rebuilt the dataset using GPT-4o annotations to generate cleaner, more neutral captions. Then we built a human-in-the-loop validation app where users could correct outputs, feeding those corrections back into retraining. That iterative cycle was what made the model genuinely useful.
Once the runtime was stable and we had proven we could ship a real feature, the next step was expanding to other use cases. Smart Tabs launched in Firefox 141, bringing local AI to tab management.
The feature is simple: right-click a tab group, select “Suggest more tabs for group,” and Firefox analyzes tab titles and descriptions to suggest similar tabs. Users can accept or reject suggestions. The AI runs entirely on-device, so your browsing data stays private.
This project showed that the infrastructure we built was flexible enough to handle different tasks. Smart Tabs did not require a new runtime or a new model distribution system—it reused what we already had. That reusability was proof the architecture was working.
After Smart Tabs, we added many other small features following the same pattern: laser-focused models running on-device for specific tasks. Each one reinforced the core principle: AI should solve real problems without compromising privacy. The infrastructure we built made it cheap to ship new capabilities, and the local-first approach meant users stayed in control of their data.
The reality is that not all AI features can run locally. Small, specialized models work well on-device, but larger language models (the kind that can handle complex conversations and broad knowledge tasks) still need server-side compute. That is where AI Window comes in.
Announced in November 2025, AI Window is an opt-in feature that brings a conversational AI assistant directly into Firefox. Unlike our local features, this required building infrastructure to support server-side inference while maintaining Firefox’s commitment to user choice and control.
Over the past several months, I have been working on the server-side LLM service and the overall architecture to make sure Firefox can reliably call external services when needed. This meant designing APIs, handling failures gracefully, managing rate limits, and ensuring the system could scale while still respecting user preferences. The work was less about the models themselves and more about building the bridge between Firefox and external AI providers in a way that gives users real control.
This hybrid approach (local AI for privacy-sensitive tasks, server-side AI for compute-intensive ones) is where the browser needs to go. But it raises important questions about privacy.
Local AI gives you perfect privacy: your data never leaves your device. But when a model runs on a server, you are trusting someone else with your prompts, your documents, and your questions. That trust model needs to change.
I am looking forward to industry standards around end-to-end encryption for running LLM inference with full privacy guarantees. The technology already exists. Flower.ai has built federated learning infrastructure with end-to-end encryption that allows large models to run on remote GPUs while keeping user data encrypted. Nvidia has Confidential Computing on H100 and Blackwell GPUs, using hardware-based trusted execution environments to protect code and data during inference. The performance overhead is minimal (often less than 5%) and the privacy guarantees are real.
But here is the problem: none of this is part of the de facto OpenAI API standard that most LLM services use today. If you want to call GPT-4 or Claude or any major hosted model, there is no standardized way to do it with end-to-end encryption or confidential compute guarantees. Your data goes to the server in plaintext, and you have to trust the provider’s privacy policy.
My hope is that it will soon be possible to run inference on the cloud with strong privacy guarantees as a standard feature, not a niche offering. The hardware is ready. The cryptographic techniques exist. What we need now is for the industry to adopt these capabilities as table stakes for AI services. Until that happens, local AI remains the gold standard for privacy, and server-side AI remains a compromise.
Building AI features in a browser is not the same as building AI features in a standalone app or a cloud service. The constraints are different. You have limited resources, strict privacy requirements, and the need to work across Windows, macOS, and Linux. Here is what made it work:
Starting small: We did not try to build everything at once. The first runtime was minimal. The first model was simple. We added complexity only when we needed it.
Privacy as a requirement, not a feature: Every decision started with “can this run locally?” If the answer was no, we either changed the approach or did not build it.
Reusable infrastructure: We built the runtime once and used it for multiple features. That meant each new AI capability got cheaper to ship.
Learning from real users: The validation app for PDF.js alt text was not just about improving the model—it was about understanding what real people needed. User feedback drove every iteration.
Two years ago, I did not know how to train a model or what ONNX was. Now I have shipped multiple AI features in production. Here is what stuck with me:
You do not need a PhD: Machine learning has a reputation for being inaccessible, but the tools have gotten good enough that you can learn by doing. I started with a pre-trained model, fine-tuned it, and kept iterating. Most of the work was engineering, not research.
Data quality beats model size: We spent more time cleaning datasets and handling bias than we did optimizing model architecture. A smaller model trained on better data outperformed a larger model trained on messy data.
Privacy is possible: The narrative around AI assumes everything needs to run in the cloud. It does not. Local models work. They are fast enough, small enough, and private by default.
Building the process matters more than building the model: The validation pipeline, the retraining loop, the distribution system. That infrastructure was more important than any single model.
This work is not finished. We plan to iterate on PDF.js alt text, expand Smart Tabs, and bring AI Window to users who want conversational AI in their browser. WebNN is coming, and that will give us even better performance for local models. The Firefox ML runtime is still experimental, but it is stable enough that other teams are starting to build on it.
The bigger challenge is pushing the industry toward privacy-preserving server-side AI. Confidential compute and end-to-end encryption for LLM inference should not be experimental features. They should be the default. I hope to see more providers adopt these technologies and for standards bodies to make privacy guarantees a core part of the AI API specifications.
On a personal level, these two years showed me that AI in the browser is not just possible—it is the right way to do it. Local models give users control. They protect privacy. And they prove that you do not need to send your data to a server farm to get intelligent features. But when you do need server-side compute, it should come with strong privacy guarantees, not just promises.
What excites me the most is running AI locally. That is where the future of open AI lies: not just open models and open weights, but truly open AI that runs on your device, under your control, without gatekeepers or surveillance. The browser is the perfect platform to make that future real.
I am proud of what we built. More importantly, I am excited about what comes next.
Running AI locally is the long game. A decade from now laptops and phones will run much larger models natively, and the best experiences won’t require sending your data off to a cloud service. WebNN is how the web gets there.
WebNN is a W3C draft specification that exposes a graph-based neural network
API to the web platform. Instead of binding directly to CUDA or Metal, browsers
map WebNN calls to whatever native acceleration they have: DirectML on Windows,
Core ML on macOS and iOS, NNAPI on Android, or a CPU path via TFLite/XNNPACK.
When a CPU path exists, the browser can fall back there. Think of it as canvas
for neural networks: you provide the graph, the browser picks the fastest safe
backend.
WebNN is a graph builder and validator. The browser takes the graph you define in JS, converts it into a static graph aimed at one of the underlying runtimes in the OS (DirectML, Core ML, NNAPI, TFLite/XNNPACK, or ONNX Runtime on newer Windows), and hands it to that native library. The heavy lifting lives there: compilation, scheduling, and kernel selection. WebNN is the portable contract that keeps your app code unchanged while the browser targets the best backend.
In Chromium, WebNN uses DirectML by default on Windows and can use the OS-shipped ONNX Runtime backend on Windows 11 24H2+, falling back to DirectML otherwise.
Libraries like ONNX Runtime Web and TF.js already use WebGPU to get more speed, but that means treating a graphics API as an inference runtime: writing shaders, managing bindings, and re-implementing scheduling. WebGPU is great for explicit control; WebNN is the spec we actually want for AI, with portable graphs, browser-managed backend choice, and no shader boilerplate.
Chromium-based browsers ship WebNN behind a flag, and ONNX Runtime Web can use the WebNN execution provider when present. According to the public implementation status (webmachinelearning.github.io/webnn-status), the 95 ops in the spec are now covered across Core ML, Windows ML/DirectML, the WebNN execution provider for ONNX Runtime, and TFLite/XNNPACK (LiteRT) with only a handful still in flight. That’s enough to make real apps: speech commands, lightweight summarization, image segmentation, and style transfer.
The momentum is similar to what we saw with WebGPU two years ago: early adopters can ship progressive enhancements now, and the API will solidify while hardware vendors line up their drivers.
The big shift is that WebNN moves backend selection into the browser while keeping a high-level graph API. It is closer to Core ML or DirectML than to raw GPU programming.
The web wins by being portable and low friction. AI has been the missing capability that pushed teams toward native wrappers or cloud-heavy designs. WebNN gives us a standard, permissionless way to run meaningful AI locally in the browser while respecting energy and privacy constraints. It unlocks the boring path to mass adoption: no installs, instant upgrades, and enough abstraction that developers can stay focused on UX rather than driver matrices.
Now is the time to experiment, measure, and ship progressive AI features. The future of AI in browsers looks like WebNN.
]]>