Data splitting in machine learning (Source: https://towardsdatascience.com/data-splitting-technique-to-fit-any-machine-learning-model-c0d7f3f1c790)
To prevent overfitting and to correctly evaluate your model, divide your data into train, validation, and test batches.
What is Overfitting in Computer Vision?
When training a computer vision model, you present your model examples of images from which it may learn. To assist your model to convergence, you may use a loss function to tell it how near or distant it is to reaching the correct prediction. A loss function describes the "badness" of a model. The better the model, the smaller the value of the loss function.
Based on the loss function, the model creates a prediction function that maps the pixels in the image to an output.
The risk in training is that your model will overfit to the training set. In other words, the model may develop an extremely specialised function that works well on your training data but does not generalise to images it has never seen before.
If you hyper-specify your model to the training set, your loss function on the training data will continue to decrease, but your loss function on the held-out validation set will ultimately grow. This is depicted here with two curves visualising the loss function values as training progresses:
An example of overfitting during training.
This indicates that your model is not learning well, but rather memorising the training material. This implies that your model will struggle with new images that it has never seen before.
To counteract overfitting, the train, validation, and testing datasets should be designed.
What is the Training Dataset?
The training set differs from the validation set in that the training set is the biggest corpus of your dataset that you reserve for training your model. Because the model has previously had a chance to look at and memorise the right output, inference on these photos will be regarded with a grain of salt after training.
As a starting point, we propose assigning 70% of your dataset to training.
What is the Validation Dataset?
The validation set is a subset of your dataset that you will use during training to see how well your model performs on images that were not used in training.
During training, it is usual to report validation metrics such as validation mAP or validation loss after each training period. These measures let you determine whether your model has achieved the highest performance possible on your validation set. You can opt to discontinue training at this moment, which is known as "early stopping."
You may iterate on your dataset, image augmentations, and model design as you work on your model to improve its performance on the validation set.
We propose reserving 20% of your dataset for validation.
What is the Test Dataset?
After completing all of the training experiments, you should have a good idea of how your model will perform on the validation set. However, keep in mind that the validation set measurements may have impacted you during the model's development, and as a result, you may have overfit the new model to the validation set.
Because the validation set is actively used in model construction, it is critical to have an entirely different data base - the test set. At the very end of your project, you may run evaluation metrics on the test set to get a feel of how well your model will perform in production.
We suggest dedicating 10% of your dataset to test set.
How Train, Validation and Test Relate to Preprocessing and Augmentation
Naturally, the train, validate, and test notion informs how you should process your data as you prepare for training and deployment of your computer vision model.
Image transformations are employed in preprocessing procedures to standardise your dataset across all three divides. Examples include static cropping and grey scaling your images. To train, validate, and test, all preprocessing methods are used.
Image augmentations are used to expand your training set by adding minor changes to your training images. These only occur on the training set and should not be utilised in evaluation methods. You should utilise the ground truth pictures from the validation and test sets for assessment.
Typical Train Pitfalls, Validation, and Test Split
Train/Test Bleed
When some of your testing images are extremely similar to your training images, this is known as train test leak. For example, if you have duplicate images in your dataset, you should ensure that they do not enter distinct train, validation, and test splits, since their existence would skew your evaluation metrics.
Excessive focus on the Training Set
The model improves with additional data. This motto may persuade you to use the majority of your dataset for training and only leave 10% or so for validation and testing. Skimping on validation and test sets, on the other hand, may obscure your assessment metrics with a small subsample, leading you to select a poor model.
Too much focus on Validation and Test Set Metrics
Finally, the validation and test set metrics are only as good as the data that underpins them, and therefore may not be truly reflective of how well your model will perform in production. However, you should utilise them as a guidepost, increasing the performance and resilience of your models.
Learn more about SKY ENGINE AI offering
To get more information on synthetic data, tools, methods, technology check out the following resources:
- Telecommunication equipment AI-driven detection using drones
- A talk on Ray tracing for Deep Learning to generate synthetic data at GTC 2020 by Jakub Pietrzak, CTO SKY ENGINE AI (Free registration required)
- Example synthetic data videos generated in SKY ENGINE AI platform to accelerate AI models training
- Presentation on using synthetic data for building 5G network performance optimization platform
- Working example with code for synthetic data generation and AI models training for team-based sport analytics in SKY ENGINE AI platform
