VISYNEXTalk to an engineer
Measurement

How do I check synthetic labels? First, test the check

How to check synthetic defect labels automatically

Test the check before you trust it: plant a fault it must miss and one it must catch. A box-on-the-change check, run on 70 synthetic defect images, passed the first control on all 70 and the second on only 46. So the number it had produced was withdrawn.

What Visynex isVisynex makes synthetic defect images for training machine vision inspection models. Each image is made from one photograph of a good part, and every defect image ships only if a separate detection pass finds the defect ordered.

What the check asked

Every synthetic defect image carries a bounding box. We wanted a cheap, automatic way to flag images where the box might be on the wrong spot.

The check compares each image with the reference photograph of the good part, aligned to the same frame. It takes the 0.5% of pixels that changed most and asks what share of them falls inside the box. A high share means the box sits on the change. A low share means it doesn’t.

surface scratch on a laser-cut steel bracket, synthetic defect image with its detection box
surface scratch, synthetic, from one photograph

The two controls

On every one of the 70 delivered images of a laser-cut stainless steel bracket (our own test part), we painted a dark disc: 45 px radius, 80 grey levels darker, lightly blurred. It went at whichever of five fixed points was farthest from the image’s real defect. The rule was fixed before anything ran: pass both controls, or the number is not used.

  • The fault it must miss: a box of the same size in the corner farthest from the disc. A working check scores it near 0. Ours never went above 0.1 on any of the 70, and the highest was 0.0. Passed.
  • The fault it must catch: a box on the disc. A working check scores it near 1. We required at least 0.8 on each of the 70. Ours reached 0.8 on 40 of the 70. Failed.
  • Then a fault in the check itself: it aligned each image on whichever image came first in the batch. With each image aligned on itself, it reached 0.8 on 46 of 70. Still failed.
must miss: a boxaway from the discworking check: lowmust catch: a boxon the discworking check: high
A schematic of the method, not a result.

Why it failed

The disc was not too faint: its median depth after clipping was 67 grey levels, and the faintest was 61. Alignment was part of it: with each image aligned on itself, 8 of the 30 failures passed, while 2 images that had passed now failed.

The rest was competition. On the 30 failures (under the original alignment), a median of about 1,100 pixels inside the real defect’s box changed more than the disc did. About 1,200 more elsewhere also changed more: small differences from re-rendering and alignment. The check keeps only the top 5,120 pixels, so a few thousand competitors are enough to push the disc out.

The number it had produced did not survive. Of the 26 images it flagged as “box off the change” (under 0.1), it could not find the planted disc either on 12.

the planted discthe image's real defectre-render differencesonly the most-changedpixels are kept, sothe others crowdthe disc out
A schematic of the idea, not a result.

What transfers to your own data

  • A check that has only passed a negative control is not evidence. Ours did that perfectly.
  • Check that the check’s own setup (here, its alignment) does not depend on what else is in the batch.
  • Plant the exact fault the check exists to catch, at a realistic strength, and watch it fail.
  • Fix the pass rule before you look.
  • “The box is on the change” is not “the box is on the defect”. Anything else that changed between the reference and the image competes for the same pixels.
fix the pass rule firstplant a faultit must missplant a faultit must catchrun the checkpassed both:use its numberfailed either:don't use it
A schematic of the method, not a result.

Limits

  • One part, one check, 70 images.
  • The check was ours, and what it lost to is specific to how our images differ from the reference. A check built differently might pass.
  • The method, two controls and a rule fixed first, is the part that transfers.

Questions

How do I check synthetic defect labels automatically?

Before trusting any automatic check, show it can say both no and yes: plant a fault it must miss and a fault it must catch, fix the pass rule before you run it, and use its numbers only if it passes both.

Isn’t a check that never flags a wrong box good enough?

No. Ours never flagged the planted wrong box (0 of 70), and still missed the planted right one on 24 of 70. Passing only the negative control is not evidence.

Does “the box is on the change” mean “the box is on the defect”?

No. Anything else that changed between the reference photograph and the image competes for the same pixels, including the real defect and small differences from re-rendering.

For the engineer

Try it on your own part

Sign in with your work email, upload one photograph of a good part, and get up to 10 images of it back, labelled in YOLO and COCO. Each takes about a minute. Free, once per email address.

Try it free
For whoever signs off

Talk to an engineer

Thirty minutes with someone who built the pipeline. No upload, no account.

Book a call