defect-library-check

Check the defect library and the label table of an automated visual inspection project before any model is trained or any station is validated. Give it a defect library written as YAML or JSON and a CSV table with one row for each image, and it reports the mistakes that are cheap to find in two files and expensive to find after the numbers are in: a class with no definition, no limit, no severity or no sample, two classes with one limit, a library with no place for a defect nobody listed, labels that are not in the library, classes with no images or too few, a share of good images that does not look like a line, images that are not on disk, and the leaks that make validation numbers look better than they are: one part in two splits, one image in two splits, a split with no defect, no test split. It fails the run on the level you set, so the two files can be checked on every change.

The tool reads the text of two files and never opens an image: it does not run a station, and it does not judge whether a definition or a label is right. A clean result means only that none of the problems of its checks were found in the files. It is not a validation, not a certification and not legal or regulatory advice, and it does not claim compliance. Every check is a best practice of this tool: no standard sets any of them, and the reports say so. EU GMP Annex 1 (manufacture of sterile medicinal products), section 8, asks manufacturers to keep a defect library of the known defect classes and to validate an automated inspection method as equal to or better than manual inspection. This tool checks only the library and the label table that such work starts from. It cannot tell whether the annex applies to you or whether you meet it. It reads the files you give it and nothing else: it contacts nothing over the network, runs nothing, needs no key, reads no clock and writes nothing.

The mistakes in these files tend to be the same: a class copied and its limit left unchanged, a class nobody ever labelled, a label typed with another spelling, a validation set made of more pictures of the parts the model has already seen, a test set with no defect in it. They get past a read of the table in a spreadsheet and are cheap to find in a file. This tool reads them on every build and points at the line.

• The library. Duplicate class names or ids, a class with no written definition, no accept or reject limit (by size, by count or in words), no severity (critical, major or minor) or no sample reference, two classes with the same limit, and no class for "other or unknown" so that unseen defects have somewhere to go.
• The labels against the library. A label that is not a class, a class with no images, a class with fewer images than --min-per-split in a split, no good images, a share of good images outside --good-share-min and --good-share-max, duplicate image paths, and, with --images-dir, images that are not there (existence only; no image is opened).
• The split. The same part id in more than one split, the same image in two splits, a split with no defective image, no test split.
• The quality of the labels. A class where every image is flagged borderline, an image with two different labels, rows with no labeller when other rows have one.
• A summary. The number of images for each class and split, in every report.
• Paths that stay in their folder. An image path that is absolute, has a drive letter, is an address, climbs out with .., hides a colon or a control character, or goes through a link is reported and never asked about.
• Checks with ids. Twenty-six checks, each marked as a check of a file, of the library, of the labels, of the split or of quality, each with what it looks at, what to change, why it is a check at all and what it cannot see. Each message says what was found and where, as file:line.
• Readers of its own. A small, documented subset of YAML, JSON and a strict CSV reader, all with the line and column of every key or cell. They refuse what they do not read (anchors, tags, a key written twice, a stray quote) instead of guessing.
• Bounded work. Hundreds of thousands of rows, brackets nested a hundred thousand deep and a table of one image path repeated on every row all end in good time, and no function calls itself.
• Reports as text, markdown, JSON, SARIF and JUnit XML (one test case for each check), with an exit code for a pipeline: 0 no finding at or above the --fail-on level, 1 a finding at or above it, 2 usage or input error.
• Plain JavaScript, no dependencies, no clock and no network.

npm test

tests 318
pass 311
fail 0
skipped 7

These are the summary lines that npm test prints at the end on Windows. The skipped tests are the ones that need a symbolic link or a device (a link to a file, a link to a folder, and /dev/null named as a file) or a system that compares names with case; they are guarded with process.platform, run on Linux and macOS where they apply, and none of them asserts a line. npm run test:coverage runs the same tests with coverage; their time limits are four times wider when coverage is on.

Quick start

Requires Node 22 or newer. There is nothing to install.

git clone https://github.com/INNERLUXES/defect-library-check.git
cd defect-library-check
node bin/defect-library-check.js examples/bad/library.yaml --labels examples/bad/labels.csv

Or without a clone, in your own project:

npx --yes github:INNERLUXES/defect-library-check inspection/library.yaml --labels inspection/labels.csv --min-per-split 10

To check that the images the table names are there, below one folder:

node bin/defect-library-check.js inspection/library.yaml --labels inspection/labels.csv --images-dir inspection/images

To check the library on its own, before there is a table:

node bin/defect-library-check.js examples/bad/library.json

To see why a check exists, what it cannot see and what to change:

node bin/defect-library-check.js --explain split-part-leak
node bin/defect-library-check.js --list-checks

To show each check as a test in a build system:

node bin/defect-library-check.js inspection/library.yaml --labels inspection/labels.csv --format junit > junit.xml

examples/ holds made-up libraries and tables for made-up products. The good ones, in examples/good, are a glass vial library in YAML with a label table of 189 images and a blister pack library in JSON with its table; each class has a definition, a limit of its own, a severity and a sample, each library has a class for other or unknown defects, each class has enough images in each split, no part sits in two splits and every split holds defects, and they pass every check at error, warning and info level. The bad ones, in examples/bad, are a vial library with a mistake of most library checks (a class written twice, a class with no definition, a misspelt key, a class with no limit, no severity or no sample, two classes with one limit, an unknown severity, no class for other or unknown) with a table that holds a label that is not a class, classes with no images or too few, a path that climbs out of its folder, a duplicate image, an image with two labels, a class that is all borderline and a labeller missing; a blister library with mistakes; a table with a part and an image in two splits (leakage.csv); and a library and a table that cannot be read.

Usage

defect-library-check <library file> [--labels <label table>] [options]
defect-library-check --list-checks
defect-library-check --explain <id>

--labels <file>            the label table (CSV): image path, part id, label, split, and, if you have them, borderline, labeller
--images-dir <folder>      check that each image named in the table exists below this folder (existence only; no image is opened)
--min-per-split <number>   the fewest images a class may have in a split, from 0 to 1000000 (default: 5)
--good-share-min <number>  the lowest share of good images in percent, from 0 to 100 (default: 20)
--good-share-max <number>  the highest share of good images in percent, from 0 to 100 (default: 99)
--good-label <text>        the label that means a good part, read without case (default: good)
--format <name>            text, markdown, json, sarif or junit (default: text)
--json                     the same as --format json
--disable <ids>            comma-separated ids of checks to leave out
--min-severity <level>     the lowest level of a finding that is shown: info, warning, error (default: info)
--fail-on <level>          the lowest level of a finding that fails the run: info, warning, error (default: error)
--list-checks              list the checks with their level and kind, and exit
--explain <id>             explain one check: what it looks at, what to change, what it cannot see, and exit
--help                     show the usage
--version                  show the version

docs/inputs.md is the format of the library file, the YAML subset the tool reads, the columns of the label table and what the images folder is used for, with a short library that passes every check. The path in the report is relative to the folder you run the command in, with forward slashes on every system, followed by the line.

A sample run

The vial library and table in examples/bad, as the tool reports them (the head, two of the findings, and the end; examples/output holds the whole report in every format):

defect-library-check: examples/bad/library.yaml + examples/bad/labels.csv
library file: examples/bad/library.yaml (yaml, 40 lines, 1,287 bytes)
label table: examples/bad/labels.csv (csv, 43 lines, 2,130 bytes)
library: Glass vial inspection library (with mistakes): 7 classes; 0 for other or unknown; severity: critical 3, major 1, minor 1, none 2
labels: 41 rows, 39 images, 38 parts; 13 good (33.3 percent), 26 defective
splits: train 24, validation 7, test 8
findings: 27 (8 error, 19 warning)

  ERROR    class-duplicate          examples/bad/library.yaml:9
           The class "Crack" is written twice: first at line 3, again at line 9.
           fix: Give each class one name and one id. If two entries describe the same defect, merge them; if they differ, name the difference.

  ERROR    label-unknown-class      examples/bad/labels.csv:17
           The label "scrach" is on 1 image and is not a class of the library. The nearest class is "scratch".
           fix: Correct the label to a class of the library, add the class to the library with a definition and a limit, or take the rows out.

This report reads the library file and the label table and nothing else. It never reads an image. A clean result means that none of the problems of its checks were found in them; it is not a statement that the library is complete, that the labels are right, that a method is valid or that any rule is met.

RESULT: FAIL - 8 findings at or above error (8 error, 19 warning)

The vial library in examples/good with its table:

defect-library-check: examples/good/library.yaml + examples/good/labels.csv
library file: examples/good/library.yaml (yaml, 41 lines, 1,888 bytes)
label table: examples/good/labels.csv (csv, 190 lines, 10,672 bytes)
library: Glass vial inspection library: 6 classes; 1 for other or unknown; severity: critical 2, major 3, minor 1, none 0
labels: 189 rows, 189 images, 170 parts; 93 good (49.2 percent), 96 defective
splits: train 99, validation 45, test 45
findings: 0

This report reads the library file and the label table and nothing else. It never reads an image. A clean result means that none of the problems of its checks were found in them; it is not a statement that the library is complete, that the labels are right, that a method is valid or that any rule is met.

RESULT: PASS - no finding at or above error

How the files are read

• The library file is read as JSON when its first character is { or [, and as the YAML subset otherwise. The reader keeps the line and column of every key, so a finding points at the line.
• The label table is read as CSV with a comma or a tab. The header names the columns, in any order. A quote that is never closed, text after a closing quote, a missing required column and a semicolon as the delimiter are refused with the line and column.
• A syntax error is a file-unreadable finding with its line and column, and no other check of that file runs. The run fails; a file that cannot be read never passes.
• A row is read as far as it can be: a value that is not what the format says is a finding, and the row is left out of the other checks.
• Each image is counted once, by the first row that names its path.
• A file that is larger than 4 MiB, a link, a folder, a device, an address and text that is not UTF-8 or UTF-16 stop the run with exit code 2 and no report.

What it checks

Check	Level	What it looks at
file-unreadable	error	a file cannot be followed by the reader (a syntax error in the YAML subset or the JSON, a library with no classes, a label table with a quote that is never closed, a header without the columns it needs, or a file past a limit); no other check ran on that file
field-invalid	error	a value is not what the format says: a class with no name, a severity that is not critical, major or minor, a value that is not text, a row with the wrong number of cells, an empty image path or label, or a split that is not train, validation or test
field-unknown	warning	a key of the library or a column of the label table that the tool does not read, often a misspelt name such as defenition or splt; what is under it is not checked
class-duplicate	error	two classes with the same name (compared without case and without extra spaces), or two classes with the same id
class-no-definition	error	a class with no written definition: nothing says what the defect looks like or how it differs from the classes next to it
class-no-limit	warning	a class with no accept or reject limit (no "size_limit", "count_limit" or "limit"): nothing says how big, how long or how many of the defect make a part fail; a class named other or unknown, or marked catch_all: true, is left out
class-limit-duplicate	warning	two classes whose limit text is the same (compared without case and without extra spaces)
library-no-other-class	warning	no class for "other or unknown" (a class whose name holds other, unknown or unclassified, or that carries catch_all: true), so a defect nobody listed has nowhere to go
class-no-severity	warning	a class with no severity (critical, major or minor)
class-no-sample	warning	a class with no sample reference ("sample" or "samples"): nothing points at an image, a physical sample or a drawing that shows the defect
label-unknown-class	error	a label in the table that is not the name or the id of a class of the library, and is not the good label (--good-label); the finding shows the nearest class and says when the only difference is case
class-no-images	warning	a class of the library that no image of the label table carries
class-too-few-images	warning	a class with fewer images than --min-per-split (default 5) in a split that the table uses, counting each image once
good-missing	error	the table has no image with the good label (--good-label, default good)
good-share-out-of-range	warning	the share of good images among all images is below --good-share-min (default 20 percent) or above --good-share-max (default 99 percent)
image-missing	error	with --images-dir, a path of the table that names no file below that folder (it does not exist, or is a folder); only the existence of the name is asked, and no image is opened
image-path-unsafe	error	an image path that is absolute, starts with a drive letter, has the form of an address, climbs out of the images folder with "..", holds a control character, has a colon, a reserved Windows name or a trailing dot or space in a segment, or (with --images-dir) is or passes through a symbolic link or cannot be asked about; the tool never asks the system about such a path
image-duplicate	warning	the same image path on more than one row (compared after cleaning up slashes and "." segments)
part-id-missing	warning	rows with an empty part id, so the split checks cannot tell which part the image shows
split-part-leak	error	the same part id in more than one split: images of one physical part are in train and in test, or in validation and in test
split-image-leak	error	the same image path in more than one split
split-no-defects	error	a split that the table uses and that holds no defective image: every image of it is good
split-test-missing	error	the table has no image in the test split
class-all-borderline	warning	a class whose every image is flagged borderline (needs a borderline column)
label-disagreement	warning	the same image on more than one row with different labels, as when two labellers disagree
labeller-missing	warning	some rows have a labeller and some do not (needs a labeller column)

Every check is a best practice of this tool; --explain <id> says why it exists, what to change and what it cannot see.

What is not checked

• Whether the definitions and the limits are right. The tool checks that they are written down, and that no two classes share one.
• Whether the labels are right. It sees the written label and, when two rows disagree, says so. It does not know which is true.
• The images. The tool never opens one. It cannot see blur, a wrong picture, a copy under another name or a mark on the lens.
• Whether the library is complete for the product, or whether the samples named in it exist.
• Rates. It counts images and works out one share. It does not work out an escape rate or a false reject rate, and it does not compare a method with people.
• Whether a rule applies to you or you meet it. The documents name EU GMP Annex 1 (manufacture of sterile medicinal products), section 8, as the reason a defect library is kept, and the tool cannot tell whether the annex applies to your product, what your quality system must hold or whether your study is enough. It is not legal or regulatory advice.
• The export formats of any labelling tool. Convert the library and the table to the formats of docs/inputs.md first.

docs/limits.md has the full list.

Formats

• text for a terminal, markdown for a job summary or a pull request comment.
• json is the report object the other formats are built from.
• sarif is SARIF 2.1.0, for code scanning: the checks are the rules and each finding with a place in a file is a result with its line and column.
• junit has one test case for each check, a failure for a check with findings, a skip with the reason for a check that did not run and an error when a file could not be read. Every time is 0.

Exit codes

Code	Meaning
0	No finding at or above the --fail-on level (default error).
1	At least one finding at or above it.
2	A usage error, or a file that cannot be read at all (missing, a link, a folder, too large, not text); no report.

--min-severity hides findings from the listing and never changes the exit code.

Limits

A file past 4 MiB, a library of more than 200,000 lines, 40 levels of nesting or 5,000 classes, and a table of more than 250,000 rows, 50 columns, 2,000,000 cells or a cell of 20,000 characters is refused whole. Up to 1,000 findings of one check are listed; the rest are counted. See docs/limits.md.

Architecture

bin/defect-library-check.js      the command
src/cli.js                       arguments and the exit code
src/check.js                     options, a run, the report object
src/yaml.js, src/json.js         the two readers of the library, with line and column
src/csv.js                       the reader of the label table, with line and column
src/nodes.js                     the tree the library readers build
src/library.js                   the model of a library: classes, limits, severities, samples
src/labels.js                    the model of a label table, and the cleaning of image paths
src/checks.js                    the checks of the library, the labels, the split and the quality
src/images.js                    asking whether a name exists below the images folder
src/rules.js                     the 26 checks: id, level, kind, text
src/report.js                    text, markdown, JSON, SARIF, JUnit
src/files.js, text.js, size.js   reading a file, cleaning text, numbers
tools/check.mjs                  the repository checks that run next to the tests
tools/example.mjs                runs the examples and writes examples/output

Where the ideas come from

• EU GMP Annex 1 (manufacture of sterile medicinal products), section 8, asks for a defect library and for an automated inspection method that is equal to or better than manual inspection. The checks look at the two files that such work starts from, and make no claim about the work itself.
• Splitting by part, not by image, and keeping a test set that is used once, are the common advice of anyone who has been misled by a validation number that was too good. The checks look for the leaks that advice is about.
• The advice on definitions, limits and samples is the common advice of inspection training: a class is only as good as the words and the sample that two inspectors share. The numbers the tool uses (five images in a split, a good share from 20 to 99 percent) are its own choice and can be changed.

docs/context.md says what checking these files is good for and where it misleads, docs/method.md has the method, docs/decisions has the decisions, docs/threat-model.md and docs/secure-defaults.md have the security notes.

License

MIT, see LICENSE.