Training set GUI
The training set GUI (aka pixels_selection_GUI) is the interface we created for training set construction. From this interface, you can load stacks, label them, create label classes, add preprocessing operations and more. It will produce a what we call a training set structure detailing which z-pixels belong to which class as well as all training parameters. That structure can then be processed by the LoadAndExtract function to create a dataset that the classifiers can then be trained on.
Here we give a detailed description of how to use the interface.
The main window is dedicated to the labelling of elements of interest in the z-stacks. The left part of the interface is dedicated to displaying stacks and labels. The right sidebar is dedicated to the following elements:
First element from the top is stacks loading and switching.
When clicking on the "Add Stack / Set / Bundle" button, a file selection GUI will appear. You can switch selection modes with the "Files of Type" pop-up menu at the bottom to load different types of data:
- Stacks & Sets (.mat files): Either load stacks that have been properly formatted, or load training sets that have been saved to disk.
- Bundles (.zip files): Unzip and load bundles.
- TIFF images (.tif files): Read TIFF image files, format the stack and store them into a MAT file (saved to disk under the same path and name, but with a .mat extension), and then load the stack. If you select several multi-page tiffs, each one will be treated as a separate stack. If you select several single-page tiffs, the whole selection will be treated as a single stack and all the files will be concatenated into a MAT stack file that will be saved under the name of the first file in the selection (e.g. 'path/to/stack_z00001.mat' if the first file was 'path/to/stack_z00001.tif').
You can switch between loaded stacks with the pop-up menu in the upper-right corner. Note that if you load stacks but do not label them, they will simply be ignored during the training steps. However, they will be reloaded every time you load up the saved training set, and will be added to bundles, which might lead to long processing times or disk usage problems. You can not remove a stack once it has been loaded.
The second element is for moving between frames in the stack.
This one is pretty straightforward: The slider and the text box allow you to pick a frame in the stack to display. Depending on the memory available on your computer, switching between frames can be pretty sluggish. You might want to save your training set, restart matlab and re-load it.
The third element is for classes creation, removal and selection. What we call "classes" is the label that is going to be attributed to regions of interest for training.
When clicking the "add class" button, a window will pop up asking for a name for the new class and what color to use for displaying that class. The color can be specified as any of matlab's single-letter color shortcuts or an hexadecimal color code (must start with a '#' sign).
The "show tree" button shows the classes hierarchy tree.
The list box is used to select which class is currently being labelled. Classes can be modified by double-clicking them.
You can toggle the classes visibility using the "visible" checkbox at the bottom of the listbox. Visibility for all classes is toggled. If you need to turn visibility on or off for only one class, see the classes attributes paragraph below.
The 4th element is dedicated to preprocessing operations: Image processing operations that will be applied, frame by frame, to the stacks before training and prediction. By default there is a single "Intensity" operation (i.e. no operation is applied to the frames, the intensity of the z-pixels is used). There must be at least one operation in the list box.
The results of the operation selected in the list box will be displayed on the left. Note that this is probably not the best way to analyze the results of the different operations, especially if you are using particularly complex custom functions. I would advise that you evaluate those independently before using them.
The "add" button calls a GUI to append a new operation to the list.
The "remove" button removes the currently selected operation.
The "load from set" button opens a file selection window: You can select a training set that was saved to disk, and the operations from that set will be loaded. (Useful if you have found a series of operations that work well for your application.)
The 5th element is for labelling regions in the stacks for training. Once you have added at least one stack and one class, you can start adding and removing labels.
The "append pixels" button switches on the regions selection tool. You can click on the frame displayed to the left to create a new region that will be labelled as the currently-selected class. To close the region you can (1) right-click (2) click on the first node of the region. You can then double click inside the ROI to confirm the selection. For more info on this selection mode, see the dedicated matlab page.
The "remove pixels" button behaves similarly to the "append pixels" one, except that it removes the label of the currently-selected class from the region drawn by the user.
Matlab's built-in region selection tool is not super user-friendly, but it gets the job done. For faster set construction, check out morphology and boolean operations. You can also use other softwares that are more "drawing friendly" like illustrator, inkscape, photoshop etc... to produce "mask" tiff files and import them.
A region can have several labels. While this would dramatically reduce the classification accuracy if the labels correspond to competing classes (or as we call them, "siblings"). However, this is to be expected in hierarchical sets, especially between parents and children. For more info on this see hierarchy below.
Click this button when you are done constructing your training set. It will process the training set and output a training set structure that can be used to train the classifiers.
When double clicking on a class name in the classes listbox of the main GUI window, an "Edit class" GUI pops up. This windoews allows for general class editing as well as regions operations to be applied to the whole class.
Most of operations have an "apply to all stacks" option that will pop up. It will determine whether the operation will be applied to the class in all stacks, or only the current stack.
Both the name and the color of the class can be modified. The color can be specified as any of matlab's single-letter color shortcuts or an hexadecimal color code (must start with a '#' sign).
The visibility of that class in the display area can also be toggled with the "visible" checkbox.
An important aspect of training set construction is hierarchy. By ticking the "is child of class" checkbox and selecting a parent class, the user can construct the hierarchy tree.
The hierarchy dictates which classes the classifiers are going to be trained against, and what data is going to be transferred to which classifier during prediction. Here we call siblings all classes that have the same parent class. A classifier is trained for each siblings group. Then during prediction, z-pixels data is fed to the classifier for the first-level siblings (all classes that do not have a parent class). Then, depending on what each z-pixel is predicted to be, the z-pixel data is passed down to the relevant classifiers for the level 2 sibling groups. The process continues until all z-pixels have reached the ends, or leaves, of the tree. In the tree example given in the image above, a specific z-pixel can be classified as part of classes 1, 5, and 11 at the same time, but not classes 1 and 2, or 3 and 4 at the same time. This type of hierarchical structure can greatly speed up classification in cases where a large portion of the image is easy to classify as "not cells" for example, and then the rest of the z-pixels are sub-classified into different cell types or regions.
If the user does not construct a hierarchy tree, all classes will be considered first level siblings and only one all-encompassing classifier will be trained.
Use the "show tree" button in the classes section of the main window to show what the current hierarchy looks like.
- Erode: Morphological erosion. This operation will basically "shrink" the class by however many pixels the user specifies in the pop-up box. See imerode for more details. (structuring element is a disk)
- Dilate: Morphological dilation. This operation will dilate the class by however many pixels the user specifies in the pop-up box. See imdilate for more details. (structuring element is a disk)
- Dilate: This operation will keep only contour pixels of the class by however many pixels the user specifies in the pop-up box. (basically an erosion and then a boolean difference)
- Union: This will merge the current class with another one specified by the user in the pop up.
- Difference: Any region of the current class intersecting with another class specified by the user will be removed.
- Intersection: Only regions of the current class intersecting with another class specified by the user will be kept.
- Inversion: invert the regions, all non-labelled parts of the stack will be labelled as current class, all labelled parts are unlabelled.
- Duplicate: Create a duplicate of current class.
- Import mask: import a tiff file that will specify what the regions are. You can use this feature for semi-automatically generating training sets if you have access to fluorescence images with the training stacks, drawing the regions with other, more user-friendly softwares, or re-use prediction down the line as a training input.
- Export mask: Export current class regions as a tiff file.
- Delete class: delete class.
When clicking the "add" button in the Pre-processing region of the main window, the window above will pop up. The user can choose between a number of basic preprocessing operations, or write their own custom function. We have not observed a dramatic improvement in either accuracy or time performance when using those preprocessing operations over just using the intensity of the z-pixels, but the space of possible operations to try is too big for us to be able to try even a fraction of possible combinations.
The first 7 choices are common basic preprocessing operations:
- Intensity: No operation, only the pixels' intensity is extracted.
- Gradient: Image gradient. The methods are described in the matlab page for the imgradient function.
- Average: Mean filtering. hsize is the size of the filter, specified as a numeric scalar or vector. Use a vector to specify the number of rows and columns. If you specify a scalar, h is a square matrix. See fspecial for more info.
- Disk: Disk average. A disk-shaped averaging filter. "Radius" is the radius of the disk-shaped filter, specified as numeric scalar. See fspecial for more info.
- Laplacian: Discrete Laplace operator applied to the image. Alpha is the shape of the Laplacian, specified as a scalar in the range [0.0 1.0]. See fspecial for more info.
- Gaussian: Gaussian blurring of the image. Sigma is the standard deviation of the Gaussian distribution, specified as a scalar or 2-element vector of positive values. If you specify a scalar, the Gaussian kernel is square. See imgaussfilt for more info
- Laplacian of Gaussian: Laplacian of Gaussian / Mexican hat wavelet / Marr-Hildreth operator. hsize is the size of the filter, specified as a numeric scalar or vector. Use a vector to specify the number of rows and columns. If you specify a scalar, h is a square matrix. Sigma is the standard deviation of the Gaussian distribution, specified as a scalar. See fspecial for more info.
In this field, the user can specify anything that can be used as a Matlab function handle: Either (1) directly a function handle with the '@' sign followed by the name of a function (e.g. @functionname), (2) an anonymous function (e.g. @(input_frame) functionname(input_frame,parameter1, parameter2) or @(input_frame,frame_number) input_frame.*frame_number), or (3) the path to a function file (e.g. path/to/file.m). The latter can also be entered by using the '...' button to the right to browse files on your disk.
When providing a custom function, keep in mind the following:
- The function must have exactly either one or two inputs: the whole frame, or the whole frame and the frame number.
- It must also have only one output: an image array of the same size as the input frame.
Here are the two possible function headers if you are writing your own function file:
function output_image = myfunction(input_image) OR
function output_image = myfunction(input_image, frame_nb)Note: The frame number is not the absolute index of the frame in the stack, but the number of the frame being fed to the pre-processing routine after z-stack subsampling has been performed: It means that, if you were starting off with a stack containing 100 frames, but are subsampling it down to ~10 frames (let's say frames [5, 15, 25, ..., 85, 95] for simplicity): The frame numbers that the custom function would receive would be 1, 2, 3, ..., 9, 10, and not 5, 15, 25 etc...
Finally, a word of warning with the "custom function file" option: If you specify a custom function file, make sure that the data extraction will be performed on the same computer, and that you will not move the function file around, or the operation will likely fail during the "run through stack" function execution. We will be adressing this issue for the release of version 1.1.
For example, a function missing from the list of basic operations above is median filtering. Median filtering can be very helpful for de-noising images. It will be added to the GUI in coming updates, but in the meantime you can simply use the matlab function medfilt2 by entering @medfilt2 in the custom function field, or @(x) medfilt2(x,[5 5]) if you want to use a 5-by-5 filter size. Or, you could write the following function file:
function output_image = mymedianfilter(input_image, frame_nb)
if frame_nb == 5
output_image = medfilt2(input_image,[4 4]);
else
output_image = zeros(size(input_image));
endAnd then enter path/to/mymedianfilter.m in the custom function box. With this file, the median filter would only be applied for frame number 5. The other frames would be all set to 0, and they would be flushed out of the training data after the PCA.
When clicking on the "training parameters" button on the bottom right corner of the GUI, one of the two windows above will pop up, depending on whether you are on the SVM or the Random Forest branch. The left side of the window is dedicated to generic training parameters. The right side is dedicated to class-specific parameters, i.e. parameters that can differ for each class.
Note that you can load parameters from another set (even an SVM set if you are on the RF branch and vice-versa) by clicking the "load from set" button at the bottom right corner.
The maximum memory that Matlab can dedicate to training. This only works for SVMs actually, and it must be regarded as a rough estimate of how much memory is going to be used.
A subsampling of the Z-stack through the Z axis. The user can use only a sub-sample of the frames in the stack for training. This is particularly useful for data augmentation and focus shifting when using large stacks for training and then switching to small stacks for training.
- Use all frames: No subsampling.
- linear: Linear subsampling. The user specifies the number of frames to use, and that number of frames will be used with a linear spacing through the stack such that the middle frame and the first and last frame are being used.
- log: Logarithmic subsampling. The user specifies the number of frames to use, and that number of frames will be used with a logarithmic spacing around the middle frame in the stack such that the first and last frame in the stack are included.
- custom: The user specifies a series of frame indexes to use.
We recommend acquiring large Z-stacks for training, >100 frames per stack, and then training the classifiers with aggressively subsampled data (down to 10-20 frames with logarithmic subsampling), in conjunction with focus shifting. This has the advantage of "artificially" inflating the size of the training dataset, but also makes the whole process robust to variations in focus. The user can then acquire subsampled Z-stacks for prediction once the training is done (so only 10-20 frames per stack).
- # Components after PCA: Number of components to keep after the PCA step. Typically in the 5-20 range. After you are done constructing your training set and start running the LoadAndExtract function, the matlab console will display data loading messages, and then it will display a message like:
Variance accounted for by the first 10 components: 87.9%. If you are not satisfied with the result (typically the higher the better, but you do not want to run into dimensionality issues), you can load the training set back up. Or, you can run an evaluation script on this particular value. - subsampling for estimation (%): If you run into memory issues during the PCA step, you can tell the LoadAndExtract function to use only a percentage of the whole dataset to run the PCA.
It is possible to specify a focus shifting "radius". Focus shifting consists in artificially augmenting the dataset by artificially shifting the Z-stack by a certain amount of frames in Z, and appending the shifted data to the dataset for training. This greatly increases the robustness to variations in microscope focusing.
If you specify a focus shifting radius of 3 for example, you are going to use 7 different shifting values in your training set: -3, -2, -1, 0, +1, +2, +3.
We have been using radii of 1 to 6, but higher values might also work.
Allows you to parallelize training: For SVMs, this means that each class's SVM will be trained in parallel. For random forests it means that each siblings group will be trained in parallel. You can select any of the matlab processing clusters that are set up on your system, which means you can easily offload the training to another server, cluster, or cloud-computing service. How to set up a cluster
Some of the parameters can be set specifically for each class. A generic "default" class is created to specify parameters that will be applied to all class unless they have been specifically set by the user.
Note that for the classifier parameters (SVMs/RFs), since one classifier is being trained per siblings group (see hierarchy), most of the parameters are not class-specific but actually classifier-specific or siblings group-specific, and modifying a value for one of the siblings will modify it for all siblings.
If you run into memory issues, or if the training time is too long, or for a number of other reasons, you might want to reduce the sampling rate of your training set: This will randomly sub-select pixels for each class. Please note that (1) because this subsampling is random it can lead to slight variations in performance between two identical training runs, (2) this subsampling happens before any further subsampling for evaluation purposes or for PCA, and (3) the actual number of z-pixels in your training dataset will be multiplied by focus shifting (i.e. for a focus shifting radius of 3, you will get 7 times the number of z-pixels). When doing random subsampling and focus shifting, different z-pixels will be subsampled for each iteration of the focus shift.
SVMs have a lot of parameters and I won't get into details on what each of them does, you can get that information from the Matlab SVM fitting function page: fitcsvm
In short, the SVM convergence parameters deal with training convergence. Typically I would mostly play with the Delta Gradient Tolerance and KKT tolerance, with values for the DG tol ranging from 0.001 to 0.1 in extreme cases, and KKT tol with values between 0.001 and 0.05. Basically the higher those values are the faster the training will converge, but if they are too high you well get an accuracy hit. On the other hand if you set them too low you will also overfit and get bad prediction accuracy results. There is no definitive set of values that will work, and you might have to run a couple evalutation scripts. As a rule of thumb though, if your training does not converge after hours of training, if your training set is not insanely big, you are probably overfitting. We typically train in 20 minutes to 2 hours.
SVM hyper parameters deal with kernel parameters and data normalization. We typically standardize the data, use the auto estimation procedure for the Kernel scale, use a Gaussian kernel.
Hyper-parameters optimization can be run using Matlab's built-in cross-validation procedure, but we actually recommend using evaluation scripts instead now. Those UI elements will probably be removed in a future update.
Random forests require a lot less parameters.
- Number of trees: Number of classification trees to 'bag' together. We typically use 50.
- In Bag Fraction: Fraction of input data to sample with replacement from the input data for growing each new tree.
- Sample with replacement: 'on' to sample with replacement or 'off' to sample without replacement. If you want to sample without replacement, you need to set 'InBagFraction' to a value less than one.
- Min leaf size: Minimum number of observations per tree leaf.
There are of course more parameters than this, but we found that those were the most important ones to improve accuray and performance. More info can be found on Matlab's random forest function page: TreeBagger
You can zoom in and out of the image with the + and - tools (:mag:), move around with the hand/grab tool, and display information on a specific z-pixel with the data cursor tool. It displays image intensity levels, x and y coordinates, pixel index and the classes it belongs to.
The save button (:floppy_disk:) allows you to save the current training set, with its training parameters, preprocessing operations, hierarchy etc... to disk. You can then load it back by using the "add Stack / Set / Bundle" button to the right. You can choose between saving a training set or a bundle by using the "file type" pop up menu at the bottom.
Please note that when you save a training set to disk, you will lose the undo/redo history, meaning that you can not revert changes on a training set that you reload.
Training sets are saved to disk as MAT files. Their structure is described here. If you move a training set between different computers, you will probably end up with file path errors. We tried to make it as easy as possible to correct it, and it is just a matter of specifying the new file path for the z-stacks, but it is a long and slow process. You may also need to check your Matlab parallel cluster settings.
Bundles are basically training sets that are zipped together with their Z-stack files, such that they can be moved between computers easily without having to worry about file paths. You may still need to check your Matlab parallel cluster settings, either by selecting a new cluster in the training parameters interface of our software, or by creating a parallel cluster of the same name on the new machine. If you are using custom pre-processing function files, you may also need to reset those.
You can undo/redo the last 10 z-pixel operations with the undo/redo arrows. The way this function is implemented is very dirty and you might run into memory problems if you use it a lot. You can just save the dataset to disk and restart the interface if it is a problem.