Measure Escaped Defects Before Buying Visual Inspection AI

02A production quality manager needs to know whether it recognises the defects that matter under the factory's actual lighting, finish variation and handling conditions.

03Write the acceptance question in product terms.

04Which defects make a part unacceptable, which require review and which are cosmetic variation?

05Identify the authorised quality reference.

06If experienced inspectors disagree, resolve that disagreement before treating their labels as reliable evaluation evidence.

07Safe Work Australia's plant model code discusses worker consultation when introducing or changing plant.[

081] For an Australian factory, review the workstation change with the relevant people and check local applicability.

09A good defect classifier does not independently establish that altered equipment or work arrangements are safe.

02A missed defect may reach the customer; a false reject can consume inspection capacity or scrap acceptable material.

03Reporting both as one error percentage hides the decision the business actually faces.

04Set separate measures for each defect category where consequences differ.

05A cosmetic mark should not dominate the evaluation simply because it appears frequently.

06Include enough relevant examples to examine important defects and state when the sample is too small to support a conclusion.

07The AI-in-OT guidance addresses ongoing evaluation and oversight in operational environments.[

082] Our quality application is to assess the full inspection workflow over changing conditions.

09A model result is only one step between a camera image and an authorised decision about a physical part.

02The system flags 40 of those defects and misses ten.

03It also flags 30 acceptable parts, leaving 920 acceptable parts correctly passed.

04Overall, 960 parts are correctly classified: 40 correct defect flags plus 920 correct passes.

05Dividing 960 by 1,000 gives 96% accuracy.

06Yet the system misses ten of 50 defects, or 20% of the defective examples, which may be unacceptable for the intended application.

07The 30 unnecessary flags also require a process.

08If each takes two minutes to review, they consume 60 minutes in this invented batch.

09These figures are illustrative only.

10They show why a decision needs defect-specific consequences and review capacity alongside a headline accuracy result.

02It is not a universal test standard.

03Select cases based on the product and document what the evaluation cannot cover, including rare defects with too few examples.

04Keep the evaluation data separate from material used to configure the system where appropriate.

05Record who assigned the reference labels and how disagreements were resolved.

06Otherwise, the apparent result may largely reflect familiarity with examples or inconsistent human judgement.

02Define how the part is identified, segregated where appropriate and reviewed by an authorised person.

03If the software flags an image but the physical part cannot be reliably located, the information does not create a controlled quality process.

04Keep release authority explicit.

05A system may support an inspector without deciding final acceptance.

06Any proposed increase in automatic disposition should be reviewed separately, with attention to traceability, error consequences and the ability to recognise a failed or unavailable inspection system.

07There is a tradeoff between sensitivity and review burden.

08Adjusting a threshold can reduce some missed defects while increasing unnecessary flags.

09Evaluate that tradeoff with the people who manage quality consequences and production capacity, rather than allowing a vendor to optimise only the most flattering metric.

02Assign ownership for recognising those changes.

03The system should record enough configuration context to explain why results differ between evaluation and later operation.

04A common failure is treating the initial result as permanent approval.

05Another is assuming that manual inspection is an error-free benchmark.

06Describe the existing process honestly and compare the proposed workflow using the same definitions, including disagreement and missing evidence.

07Begin with a controlled evaluation on one product and a defined defect set.

08Agree the acceptance measures before seeing the results.

09The next decision should be whether the evidence supports a specific role in inspection, with a clear review process and limits on what the trial has established.