Manufacturers have quietly handed AI inspection systems and their peers authority over physical decisions. Almost nobody is testing whether those systems can be manipulated, and the attack that matters won't look like an attack at all.
There's a question I've started asking when I'm walking a plant floor, and I've stopped expecting a clean answer to it.
Who tested the vision system?
Not whether it works. Everyone can answer that one, usually with a number. Scrap is down, first-pass yield is up, the line runs faster than it did with human inspectors. What I mean is whether anybody has ever sat down and deliberately tried to make it produce the wrong answer.
The response is usually a pause, followed by some version of "that's the integrator's system." Which is often accurate and doesn't really settle anything.
I don't think this is negligence. I think it's a gap that opened up faster than anyone's process could adapt to it, and it's worth being specific about what that gap actually is.
What changed on the plant floor
For a long stretch, AI showed up in manufacturing as an advisor. It produced a forecast, a recommendation, a dashboard. Somewhere downstream a person looked at that output and decided what to do with it, which meant a human was standing between the model and the consequence. If the model was wrong, there was a chance to catch it.
A lot of plants have moved well past that, and I'd argue the security thinking hasn't moved with them. Automated optical inspection now accepts and rejects parts with no human reviewing the call. Predictive maintenance models determine which assets come offline and which keep running. Robotic path planning adjusts in real time off sensor input. Process control systems tune parameters continuously.
These aren't advisory systems. They're decision systems, and the decisions land in the physical world.
What I find telling is how they got there. Very few of them came in through a security review. They arrived through operations or quality, or bundled inside a machine builder's integration package, and they were assessed the way you'd assess any capital equipment purchase. Does it meet the accuracy specification. What's the cycle time. What's the total cost. Those are all sensible questions and none of them is the question of what happens when someone works to manipulate the thing on purpose.
The result is a class of system that sits inside most modern plants, makes consequential calls without supervision, is reachable on the network, and has never once been examined by anyone thinking adversarially about it.
The failure mode nobody is watching for
Most manufacturing security discussion centers on availability. Ransomware, downtime, dollars per hour of stopped production. I understand why. It's quantifiable, executives grasp it in one sentence, and it's happened to enough recognizable companies that it feels concrete rather than theoretical.
The thing about an availability attack, though, is that it tells you it's happening. The line halts, screens lock, and someone is on the phone within a few minutes. It's a bad day, but you know you're having it.
The scenario that holds my attention is the one where nothing appears to break. Suppose someone can shift what your inspection model treats as acceptable. Production continues normally. The metrics look healthy, because the number on the dashboard is the number the model is generating, and as far as the model is concerned everything is fine. Defective parts move into assemblies, assemblies ship, and the mechanism that would eventually surface the problem runs on warranty timelines instead of security timelines.
I've seen this described as a slow-motion breach, which I think gets it slightly wrong. The bad decisions aren't slow at all. They happen at machine speed, thousands of times over, exactly as designed. What's slow is the discovery. And when the discovery finally arrives, you're not managing a security incident. You're managing a recall, reconstructing a year or more of production records to work out how far it spread.
That asymmetry is the reason I think integrity deserves more attention than it gets in this sector. Downtime you find out about immediately and recover from on your own schedule. Corrupted decisions you find out about whenever your customers happen to tell you.
Three ways AI inspection systems get manipulated
There's a version of this article that turns into a taxonomy of threat actors and techniques, and I don't think that version helps anyone actually do something differently on Monday. So here are the three areas I'd look at first, in plain terms.
The retraining loop is an input. Plenty of production vision models are retrained on production data, which means the pipeline feeding that retraining is itself an attack surface. Anyone with access to the labeling process, the image store, or the feedback mechanism that marks a decision correct or incorrect has a route to moving where the model draws its boundary. The realistic version of this isn't a single dramatic change, because that would surface quickly. It's incremental. The model never appears broken. It just learns something slightly wrong and then executes on that consistently, at speed, without complaint.
Physical inputs can be crafted to fool the model. There's a substantial research literature demonstrating that vision models can be defeated by alterations a human inspector wouldn't consciously notice, including physical-world ones: surface markings, coatings, lighting conditions. Read as an academic result it's interesting. Translated onto a line where the model holds final authority over pass and fail, it becomes a practical method for getting non-conforming product through, and I don't think that translation has been made widely enough.
Nobody can tell you where the model came from. This is the least exotic of the three and, in my experience, by far the most common. Most deployed models weren't developed in-house. They came from a machine builder or a systems integrator, frequently built on pretrained components sourced from public repositories. When I ask what data a model was trained on, who has modified it since commissioning, or whether the version currently running is the version that was validated, I usually get an honest shrug. That's supply chain risk in a fairly pure form. It's structurally the same problem as running unsigned firmware, and the industry would not tolerate unsigned firmware.
Why this is a quality problem
The framing I'd offer to a plant leadership team is that this belongs to quality at least as much as it belongs to IT. I'll acknowledge I'm still sharpening how to make that case, but the core of it is straightforward.
Manufacturing already has this discipline, and frankly does it better than most industries I work in. No one puts a gauge into service without calibration. No one accepts a supplier's material certification without incoming inspection. No one alters a validated process without documentation and requalification. That rigor exists because everybody understands what a small, undetected upstream error turns into once it's had time to propagate.
Then a model gets installed that determines what ships, and effectively none of that apparatus applies to it. It hasn't been calibrated against adversarial conditions. Its inputs aren't being validated for manipulation. Its behavior over time isn't monitored for drift that might not be accidental.
I read that as a classification error rather than a lapse in care. These systems were filed mentally as production equipment, not as something with an adversary, so the quality machinery that would have caught this was never aimed at them.
There's also a practical reason to frame it this way. The people who own defect rates already have the instincts, the budget authority, and the cultural standing to act on this. Telling a quality director that something upstream of their metrics is unverified tends to move faster than telling them about threat actors.
A practical starting point
If any of this resonates, the first step isn't buying a tool. It's building a list.
Write down every system in the facility that makes an autonomous decision affecting product, process, or safety. Most organizations can't produce that list from memory, and assembling it tends to be clarifying on its own, because people surface things they'd stopped thinking of as decision-makers.
For each item, establish provenance and change control. Where did it come from, what's underneath it, who has the ability to modify it, and what evidence exists that today's version matches what was validated at commissioning.
Then test them adversarially rather than only for accuracy. Performance against a holdout set tells you how a model behaves when conditions are normal. It says very little about how it behaves when someone is actively working to defeat it, and those are different questions with different answers.
Finally, instrument the decision distribution, not just uptime. A model whose accept rate shifts quietly is communicating something. In most plants nobody is receiving that signal, because the channel was never built.
The short version
The manufacturing attack I'd worry about isn't the one that stops your line. It's the one that lets the line keep running while it produces something you'd never knowingly have shipped.
You won't find it in a SIEM. You'll find it in warranty data eighteen months later, assuming you find it at all.
The indicator that could have told you sooner is a number your quality team already circulates every week. It just isn't being read as a security number yet, and I'd argue it should be.
Frequently asked questions
Can AI inspection systems really be manipulated?
Yes. Documented paths include poisoning the retraining data so the model slowly shifts its accept boundary, crafting physical inputs that fool the model while passing a human glance, and tampering with unverified third-party models before they reach your line. We cover the wider testing discipline in our guide to AI penetration testing.
What is the difference between accuracy testing and adversarial testing?
Accuracy testing measures how a model performs against a holdout set under normal conditions. Adversarial testing measures how it behaves when someone is deliberately working to defeat it: manipulated inputs, poisoned data, or a swapped model. A system can score well on the first and fail badly on the second, and only the second resembles an attack.
How do I know if my plant's AI systems are at risk?
Start with an inventory of every system that makes an autonomous decision affecting product, process, or safety. For each one, ask three questions: can anyone prove where the model came from, who can change it, and would anyone notice if its decision pattern shifted. If any answer is no, that system is untested attack surface.
SubRosa performs adversarial testing against AI and machine learning systems, including the vision and decision systems running in production environments. If you're working through what this looks like in your own facility, we're happy to talk it through.