Build an Inspection Dataset That Holds Up Across APAC Factories

10/11/2026

A model can score well on thousands of inspection records and still leave a basic question unanswered: how many defective units will it release? A published dataset from a Siemens electronics line contained 440,274 automated optical inspection (AOI) measurement records over 132 days. It covered boards referred for manual inspection. Because AOI-passed boards were outside the sample, the dataset can support false-call analysis within that review queue, while measuring defect escapes across the whole line requires additional evidence. [1]

To build a credible inspection dataset, define defects with plant quality teams, sample both accepted and flagged production, and reserve whole lots, later production periods, and a receiving factory for the tests they are meant to answer. That gives APAC manufacturers separate evidence for commissioning at the pilot site and transferring the system to another plant. [1][2][3]

1. Define defects and the reference label

Start with the decision the system must support: release a unit, hold it for review, or route it to rework. For each defect, document the affected feature, applicable product revision, severity, and evidence required for a decision. TMA’s visual inspection guidance begins with defect taxonomy, line conditions, and acceptance criteria. Turn those requirements into a label guide with accepted, rejected, and borderline examples. [2]

Set the reference method with quality engineering. On a flexible printed circuit board (FPCB), for instance, a reviewer might use an approved microscope procedure to confirm suspected surface damage. Record the finding, the evidence used, and the governing specification revision. Keep the AOI call and final reference label in separate fields. Use an indeterminate status when the available evidence cannot settle the decision. [1][2][3]

Define the unit of evaluation before collecting data. A panel can generate multiple circuit images and many crops of individual features. If the plant releases individual circuits, evaluate at circuit level and specify how feature findings determine that disposition. This keeps the test denominator aligned with the production decision. [1][3]

2. Sample meaningful variation

Build a sampling matrix for factory, line, product revision, lot, shift, and imaging setup. Include conforming units from normal production, confirmed defects, and difficult examples such as glare, slight misalignment, or borderline marks. Record why each unit entered the dataset. A random audit of accepted units and a targeted collection of known defects answer different questions. [1][2][3]

Preserve two evaluation views. Use a production sample with a documented selection method to estimate day-to-day review workload. Use a separate challenge set enriched with rare or severe defects to test coverage. Report the challenge set separately and use the production sample for estimates that depend on real defect prevalence. [1][3]

Lessons Learned from the Field

TMA developed FPCB defect detection for a Korean manufacturer using images from its existing AOI systems and integrating the results into the manufacturing workflow. The engineering choice was to work with evidence from the installed inspection equipment. TMA’s public account documents that integration. Buyers should request the project’s sampling and validation details during technical diligence. [4]

Use this case to evaluate a low-cost offshore development centre scoped mainly to annotation or model training. Give bidders an AOI export with several views of one board and a rescan after rework. Ask them to trace each image back to the physical unit and explain which images must stay together in evaluation. Then ask how they would obtain reviewed examples from the AOI pass stream. This tests ownership of the data source, part identity, and quality decision as well as model delivery. [1][2][4]

3. Control annotation quality

Have site quality engineers review a shared calibration batch before annotation begins. Use it to resolve differences in local terminology and document examples that distinguish cosmetic variation from a rejectable defect. Assign independent review to critical classes and borderline cases. Send disagreements to a named adjudicator and retain the original decisions for audit. [1][2][3]

Track disagreement and relabelling by site, defect class, and reviewer. Investigate clusters before approving the dataset. If one plant consistently labels a mark differently, check whether the difference comes from the drawing, capture setup, or interpretation of the guide. Version the annotation instructions, reference examples, and approved labels together. [2][3]

4. Separate data for credible evaluation

Split data by the source of shared information. Keep every crop, rescan, and augmented copy of a physical unit in the same partition. Keep lots together when shared material or process settings could make their samples unusually similar. Set aside later lots to test performance over time and a whole factory to test transfer. AOI research has shown how random splitting and thresholds chosen after evaluation can overstate deployability when production data changes over time. [1][5]

Evaluation choiceDesign that can inflate resultsDesign that answers a production question
Split unitRandom images from the same boardGroup by board or panel, and by lot where appropriate
TimeMix earlier and later productionReserve later lots for commissioning evidence
FactoryUse every site during developmentHold out a receiving site for transfer testing
Sample populationCount only AOI referralsAudit passes and report challenge sets separately
ThresholdAdjust after reviewing final scoresTune on validation data and lock before testing

Technical deployment pipeline

  • Register the evidence. Link source images to unit, lot, site, capture settings, and inspection attempt. [1][2]
  • Qualify labels. Resolve disputed cases and freeze the annotation version. [2][3]
  • Build split manifests. Record group membership and check for duplicates and shared units across partitions. [3][5]
  • Lock the evaluation. Freeze the model, decision threshold, and acceptance limits before opening the holdout. [3][5]
  • Run in shadow mode. Compare recommendations with the existing disposition process on the target line. [2][3]
  • Approve by site. Review misses, false rejects, and operator review workload by defect class and plant. Show counts and uncertainty, especially for rare defects. [2][3][5]

Report defect recall as detected defects divided by confirmed defects, and false-reject rate as rejected conforming units divided by confirmed conforming units. Show the counts behind each rate. If a receiving site supplies data for tuning, reserve fresh production from that site for its final acceptance test. [3][5]

5. Maintain data rights and feedback

Assign permitted uses for images, drawings, and labels across training, validation, support, and transfer to other sites. Agree on who can access or export each dataset and how long it is retained. TMA lists ISO/IEC 27001 in its quality framework and reports GDPR checks within software quality assurance. Verify certification scope and have privacy owners assess any records that identify personnel. [6][7][8]

After launch, capture confirmed misses, unnecessary rejects, operator overrides, and imaging changes. Review those examples before adding them to the next dataset version. Maintain a regression set and collect fresh acceptance data after changes to optics, recipes, or product revisions. TMA’s published inspection architecture includes dataset versioning, model monitoring, and rollback. [2][6]

TMA reports 29 years of engineering experience, 4,000 engineers serving clients in 30 countries, and mission-critical programmes involving hundreds of engineers. Its customer R&D centres in telecom provide a precedent for long-term partnerships. For inspection programmes, use that delivery model to assign named owners for imaging, quality data, validation, and site releases. [7][9]

FAQ

How many inspection images do we need?

Size the dataset around defect coverage, operating variation, and the confidence required for acceptance. Track independent units and lots alongside image counts, especially when one unit produces multiple images. [1][3]

Can we evaluate a model using only AOI rejects?

Yes, for decisions within the flagged review queue. To assess line-wide escapes, include an independently reviewed sample of AOI-passed production. [1]

Can synthetic defects be used?

Use them for training experiments or separately reported stress tests. Base production acceptance on representative factory evidence and disclose how synthetic examples were used. [3]

What makes a factory holdout independent?

Exclude that factory’s data from model selection and threshold tuning. If local calibration is required, reserve fresh lots for the final acceptance test. [3][5]

How should buyers evaluate an inspection dataset vendor?

Request a comparable factory reference, label audit, reproducible split manifest, and witnessed holdout run. Confirm who owns capture quality, adjudication, and post-launch dataset maintenance. [2][3][4]

Scope the evidence before funding the rollout

Bring a sample AOI export, your defect catalogue, and production records from the pilot and receiving sites to a TMA Innovation dataset review. Use the session to agree on reference labels, define the factory holdout, and set release criteria before model development begins. [2][3][4]

References

[1] Pfab et al. — Data of automated optical inspection of surface-mounted technology electronic production, Data in Brief, 2024.

[2] TMA Solutions — AI Visual Inspection for Manufacturing QA/QC.

[3] NIST — AI Risk Management Framework, Valid and Reliable.

[4] TMA Solutions — Custom AI development and Korean FPCB inspection example.

[5] IEEE COMPSAC — Towards Improved Research Methodologies for Industrial AI: A Case Study of False Call Reduction, 2025.

[6] NIST — AI RMF Playbook, Measure.

[7] TMA Solutions — Company Overview and Engineering Capabilities.

[8] TMA Solutions — CSR Report 2024, Security and Compliance.

[9] TMA Solutions — Telecommunications Engineering and R&D Centres.

TMA Innovation
Author: TMA Innovation
Table Of Content
1. Define defects and the reference label
2. Sample meaningful variation
Lessons Learned from the Field
3. Control annotation quality
4. Separate data for credible evaluation
Technical deployment pipeline
5. Maintain data rights and feedback
FAQ
How many inspection images do we need?
Can we evaluate a model using only AOI rejects?
Can synthetic defects be used?
What makes a factory holdout independent?
How should buyers evaluate an inspection dataset vendor?
Scope the evidence before funding the rollout
References
Start your projectRight now!
Contact Us
Share: