/ THE SHORT ANSWER
- 01Separate missed defects from false rejects.
- 02Build evaluation data around production variation.
- 03Preserve a controlled release and fallback workflow.
/ dotSuper point of view
dotSuper analysis: inspection AI should be judged against the consequences of specific errors and the capacity to handle uncertainty.
Start with the error that changes the release decision
It may miss a defect, reject a good part or produce an uncertain result.
Those outcomes create different work and different consequences, so one accuracy number cannot explain whether the system fits the process.
Define the defect classes with the quality team before collecting examples.
Include the evidence used to label a part and who can resolve disagreement.
If experienced inspectors disagree, that uncertainty belongs in the evaluation rather than disappearing into a supposedly clean training label.
Set the intended role of the system.
Screening parts for human review is different from releasing finished goods automatically.
The proposed workflow should state where the model informs a decision, who remains responsible and what happens when the input falls outside the evaluated conditions.
A hypothetical batch shows why accuracy is incomplete
The model flags 180 of those defects and misses twenty.
It also flags 100 good parts incorrectly.
These are invented figures for explaining the decision, not a study or vendor result.
The model makes 9,880 correct classifications: 180 detected defects plus 9,700 correctly accepted good parts.
Accuracy is therefore 98.8 percent.
Yet it misses ten percent of the defects, because twenty divided by 200 equals 0.10.
The quality team must also review 280 flagged parts, including 100 false rejects.
Whether that is useful depends on the defect consequences and review capacity.
A convincing headline accuracy value has not answered either question, and the example's error rates are not recommended acceptance thresholds.
Evaluate the operating conditions around the image
[1] The transferable question is how performance changes with the evidence and conditions.
The factory checklist below is our proposed application of that question.
Keep the test set distinct from examples used to develop the system.
Record how parts were selected.
A set dominated by familiar, easy images may exaggerate usefulness even when nobody intentionally manipulates the evaluation.
| Condition | Include | Decision supported |
|---|---|---|
| Defect variety | Relevant severity and appearance | Which defects are detectable |
| Production variation | Material, supplier and batch changes | Where performance may shift |
| Image conditions | Lighting, position and contamination | When inputs are outside scope |
| Human labels | Reviewed disagreements | Whether the reference is dependable |
| Uncertainty | Unclear and unfamiliar examples | When human review is required |
| Fallback | Unavailable camera or model | How release remains controlled |
Build the exception workflow before automatic release
Avoid a queue containing only an unexplained red box.
The reviewer needs enough evidence to make a meaningful decision.
Measure how long review takes and whether the queue fits the production rhythm.
A system that detects more potential issues can still overwhelm the team if it generates too many ambiguous cases.
Compare the proposed process with the existing inspection method using the same outcome definitions.
Retain a fallback appropriate to the quality process.
If the camera becomes unavailable or a new material appears, the system should not silently keep releasing parts under old assumptions.
Define who can suspend the model's role and how the work returns to an approved inspection route.
Keep legal classification separate from model performance
[2] That does not determine the classification of a particular factory inspection installation.
Assess the intended purpose, integration and applicable product requirements with the responsible specialists.
A standalone quality aid and a safety-related function can raise different questions.
Neither a vendor's accuracy claim nor the word industrial settles those questions.
Do not wait for a broad regulatory conclusion to document the operating evidence.
Keep the evaluated conditions, known limitations and release authority clear.
These records support the business decision and make later specialist assessment easier, without presenting an internal evaluation as a formal conformity assessment or certification.
Approve a bounded role and monitor meaningful change
Record what the system may do and when it must defer.
Expansion to another material or line should require evidence relevant to that change.
Monitor missed defects discovered downstream, false rejects and uncertain cases separately.
Investigate changes in those patterns with production and quality staff.
A stable aggregate score may hide a shift in the particular defect class that matters most to the customer.
The first useful deliverable is an evaluation plan linked to a release workflow.
It should explain which errors the business is trying to reduce, how they will be counted and who handles the remaining uncertainty.
That gives the team a defensible basis for choosing a model, limiting its role or deciding that a simpler inspection improvement would be more effective.
What this page cannot conclude
- 01The arithmetic is hypothetical and provides no estimate of actual model performance or acceptable defect rates.
- 02PTB's cited AI work includes medical applications; the manufacturing evaluation method here is dotSuper analysis, not PTB certification guidance.
- 03This article was researched and drafted with AI assistance. Sources and limitations are provided for scrutiny; it is not an independent professional review or a compliance certification.
Sources
- 01Digitalisation of quality infrastructurePhysikalisch-Technische Bundesanstalt (PTB) · accessed Sep 15, 2026
- 02Timeline for the Implementation of the EU AI ActEuropean Commission AI Act Service Desk · accessed Sep 15, 2026
This article was researched and drafted with AI assistance. Sources and limitations are provided for scrutiny; it is not an independent professional review or a compliance certification.
Our editorial standard · Found an error? Send a correction with its source.
/ CITE OR SHARE THIS GUIDE
Make the evidence easy to verify.
When you reference this guide, link to its canonical URL. That gives readers one stable place for the evidence, limitations and future updates.
dotSuper Research Desk. (September 15, 2026). Judge Factory Vision AI by the Defects It Misses. dotSuper. https://dotsuper.net/feeds/applied-systems/germany-ai-visual-inspection-defect-costs