You Can't Grep a Product

Ask a software engineer to find every caller of a function and they'll answer before you finish the sentence. Ask a mechanical engineer to find every requirement a bracket traces to, and who is affected if a hole moves half a millimeter, and you've filled their calendar for the week.

The gap isn't talent. It's data shape. Code is structured in a way machines can walk, and that structure is what made dependency analysis, refactoring, static analysis, language servers, and the whole modern developer toolchain possible. Engineering data has structure too. It's just scattered across CAD files, BOMs, spreadsheets, requirements, analyses, and PDFs.

Almost everything I've learned building AI systems for engineering comes back to that. The work is rarely about making the model smarter. It's about making the engineering world around the model easier to represent, safer to act on, and possible to verify. Represent, constrain, verify,  that pattern showed up in every system we built, and it's the shape of the rest of this piece.

Making engineering machine-readable

Strip away the file formats and a product is already a graph. Components belong to assemblies. Requirements derive from other requirements and are satisfied or verified by parts, tests, and analyses. CAD, manufacturing data, attributes, and documents are different views of one underlying system. Engineers hold those relationships in their heads. Software doesn't.

Once those relationships are explicit, an agent stops reasoning over a pile of unrelated files. It operates over engineering objects, relationships, and actions with defined meaning.

But machine-readable can't mean unrestricted. Agents behave dramatically better given narrow, typed interfaces than arbitrary access to the underlying system. The model shouldn't decide which database rows it can touch or what context it runs under. It should choose from actions whose meaning and limits are already defined, while the surrounding software owns state, permissions, validation, and safety.

That turns a vague question, did the model understand the product?, into four answerable ones. Did it choose the right action? Did it understand which object the user meant? Did it recognize ambiguity? Did it know when not to act?

The last one is the hardest to get right. We have eval cases where a user asks to delete a sensor and multiple sensors match. A capable-but-reckless agent confidently deletes one. The behavior we want is to search, surface the ambiguity, and stop. Restraint is part of competence and an interface that makes the reckless action available is a design failure no matter how the model happens to behave on any given rollout.

From understanding the product to changing it

The problem sharpens once the agent starts changing engineering state instead of reading it. In mechanical engineering, most of that happens inside CAD.

So we built environments where an agent drives SolidWorks through the GUI itself: screenshot in, mouse and keyboard actions out. It has to interpret the visual state, execute the requested edit, preserve what shouldn't change, and leave behind a CAD artifact we can inspect programmatically. The interface changed. The problem didn't. Make the state legible, restrict how the model can act on it, then independently check what happened.

That third step is where the real lessons live.

Lesson 1: doing nothing looks surprisingly good

Our first verifier rewarded two things at once: whether the requested edit happened, and whether the rest of the model stayed intact. Reasonable design except doing nothing is extremely good at keeping the rest of the model intact.

We found tasks where a complete no-op collected meaningful partial credit. Depending on the task family, the floor for doing literally nothing ran from 0 all the way to 0.45. At that point the score stops meaning what you think it means: a 0.5 on one task is real progress, on another it's an agent that didn't break the file.

The fix wasn't a better model. The verifier was wrong. Which gives you the simplest rule we have: a no-op must look like failure, and a known-correct reference must look like success. If either anchor is off, fix the evaluator before you draw a single conclusion about the agent.

We had a reference result land at 0.9994. It's tempting to call that close enough. It isn't. If your own gold result can't reach the top of your scale, the problem is your measurement. That sounds pedantic right up until you're training against it.

Lesson 2: rigor gets expensive fast

Long CAD trajectories are expensive. A serious measurement means tens of episodes, large multimodal contexts, repeated screenshots, action histories, and persistent application state. At peak we were running a fleet of around 100 CAD environments.

That forces discipline. If something can be checked offline, check it offline. If an invariant can fail before you spend model calls, catch it before you spend model calls. And when a VM dies, the CAD system hangs, or an environment resets incorrectly, that cannot quietly become a "model failure." Otherwise you're paying a lot of money to collect bad data.

The same pressure corrupts the analysis in a subtler way. Sampling is the expensive part, so the temptation is always to stop early, and pass@k is unforgiving about small samples. A 20% pass@10 doesn't mean the model succeeds on 20% of attempts. It means that across ten attempts there's a 20% chance at least one lands, which puts the per-rollout solve rate near 2.2%. At that rate a perfectly healthy task goes 100 attempts without a single win about 11% of the time.

One of our tasks looked like a wall for seven straight attempts, best result 0.72. On attempt eight it passed at 0.939. Nothing about the task changed; we hadn't sampled enough. You can talk yourself into believing a task is broken when it's merely hard, or that it's hard when your sample is noisy,  and the run that would have settled it is exactly the run you cut to save VM hours.

So verification isn't only checking whether the final answer is right. It's verifying that the environment producing the measurement is trustworthy, and that you bought enough of it to justify the claim.

Most failures aren't "the model is bad"

This is the through-line I didn't expect going in. When an engineering agent fails, the instinct is to blame model capability. Occasionally that's the answer. More often it's one of five things: the product state wasn't represented clearly enough, the interface allowed an ambiguous action, the verifier paid for the wrong behavior, the environment produced a bad episode, or the sample was too small to support the conclusion. Fix those and the same model, unchanged, looks dramatically more capable.

Which is why the interesting substrate for engineering AI isn't the model. It's everything around it that makes engineering state machine-readable, actions constrained, and outcomes measurable.

Mechanical engineering has every bit as much structure as software, assemblies, geometry, dependencies, constraints, interfaces, requirements, design intent, and almost none of it is walkable by a machine today. Making that structure explicit is tedious. So is defining safe interfaces, building verifiers, hunting reward exploits, rejecting ambiguous actions, and rerunning experiments because the thing you called a model failure was your environment.

That tedium isn't preparation for the real work. It is the real work.

-Lokesh