Assemble your training set, train on it, see what the network is learning
Starting from an on-the-shelf and a raw training set is every AI engineer’s first step. Before customization the engineer needs to select a promising architecture and see how it performs on the dataset. However, training sets can be incomplete and biased which can alter the performance assessment done. Usually the engineer only sees the performance score of the network on the dataset, for example using a confusion matrix (see Fig. 1).
At first an engineer is blind to any performance cause, as the artificial neural network does not explain much what is impacting its performance. Each guess the engineer is doing needs to be tested and the network trained again, this iterative process can take a very long time in some cases.
Improve the architecture to ensure better computer vision performance
Once the training set is stable enough you can now try to optimize the neural network architecture to improve the overall picture analysis performance over the test dataset. Normally the optimization is done manually using intuition and experience. Each optimization is a trial-and-error approach and much time can be invested without any certainty of any significant improvement.
Here is how Saimple can help your engineer find a more performant architecture:
After training the neuronal network it appears that the computer vision’s system performances are not always good on certain images of the class Helicopter. The engineers suspect that in the internal process of the network, some layers fail to intercept the feature of the landing gears. Usually, it has to change iteratively the layer, modifying its size, and number until finding an adequate solution. Some use automated processes to find this solution, but each time the training and the evaluation are necessary, which can still be long and tedious. With Saimple your engineer can visualize at each layer how the process is handling the input information (Fig. 1).
By doing this the engineer can discover that the features of the landing gears he is interested in are lost at the second hidden layer. Therefore any attempt to preserve these features should be done either on this layer or at least near this one.
Using this process your engineer discovers quickly where to change its network to improve its performance.
Save time and effort testing and validating your system against the future condition of use with formal methods
The intended surveillance system is designed to operate outdoors in any condition of brightness and weather. However, testing a dataset is usually not enough to validate the performance on every condition and even less to validate it on every combination of conditions. The real life scenario might not be covered by the testing dataset. As a result the performance of the system may vary greatly with no explanation once in production.
To avoid such situations any engineer should increase its database coverage, either manually or through data augmentation. But this is very time consuming.
With Saimple it is possible to test the robustness of your neural network type (for convolutional neural network or recurrent neural network for instance) against a variety of perturbation automatically. In this example let’s assume that the camera can suffer from blurry inputs (Fig 1.) or the presence droplets of water on the protective glass (Fig 2.).
Usually an AI engineer would perturb the data training set by adding noise to every single image (different settings of the blurring kernel, or different amount of droplets). This would require an exponential number of tests, but it won’t be able to prove whether the system is performant or not beyond this test. For that your AI engineer needs Saimple to cover all the space around the images he has with all the possible noise he can apply (Fig 3.).
But not all noises are mathematical, for example droplets on images can be challenging to model. For those Saimple allow the use of masks to model specific noises applied onto an image, for example droplets (Fig 4). The tool will help prove that the classifier can handle the application of a mask. After setting up a mask the engineer can see how much the classifier resists the perturbation model by the mask (Fig 5).
Knowing when the network classification can fail gives the engineer the knowledge how much noise can the system handle (Fig 5). It is critical to know as soon as possible if a system will resist to avoid costly deployment phases that could fail
Using Saimple the engineer was able to check during the design phase if its system will handle the volume of noise the camera can add. He was also able to test before putting in production the limit of its system to environmental perturbations.
Assuming that the classification system was trained on 64-bits architecture it takes a lot of memory space. To allow integration into the camera the system has to use only 16-bits operations. Changing just before going into production the precision causes a drop in the performance of the classifier of 3.9% (Fig. 1).
Using Saimple earlier on this situation would have been avoided since the robustness of this neural network was not guaranteed upon a change that goes as down as to 16-bits operations. However using Saimple the engineer can find how mixed precision can be used to have both a reduction of the size of the neural network and the conservation of performance (Fig. 2). Thus avoid costly and time-consuming retrofit actions which are needed otherwise.
Saimple allows your engineers to build quickly documentation and profit reproducible benchmarks of robustness and explainability. With the built-in features to export any result of Saimple you can assemble quickly any chart or graphics you need for your commercial or regulatory documentation.
Testing every aspect of AI is crucial, with Saimple you can automate your testing of the robustness and the explainability within your current continuous integration framework.
Related Use Cases
Validating the reliability of AI algorithms for detecting cardiac arrhythmias in ECGs
Improve AI system design for quality control industry
Measure robustness of data augmentation
Discover more Saimple applications to help you build reliable and trustworthy AI
Data augmentation
Data is the fuel of any neural network training process. Without enough of it your AI performance might not be there. Doing data augmentation is one solution for that. But it is a tradeoff between the time you invest in it and the quality of the AI you obtain. Numalis can help you find the adequate tradeoff.
