If you are a software engineer you know that your time is valuable; you prioritise your tasks in the seemingly never-ending list of books to read, an optimisation to carry out, an iteration or a transition to complete.
Ceiling Analysis is a way to systematically find the weakest component of your system, and therefore optimising that weakest component would best serve your time to bring the greatest improvement to the overall system.
To perform ceiling analysis in the context of deep learning, I firstly need a system in place, and have my testset data run through my system providing a baseline accuracy to improve upon.
Ceiling analysis is the process of manually overriding each component in your system to provide 100% accurate predictions with that component. Thereafter, you can observe the overall improvement of your deep learning system component by component.
Let's firstly invent a scenario and then explain how to use ceiling analysis.
The Scenario
Lets say we have built a deep learning system that fetches Instagram images from my feed that aims to categorise the images into 3 groups:
- Group 1: Image with 1 male face, and with no text in the image.
- Group 2: Image with 1 female face, and with no text in the image.
- Group 3: All other images.
The pipeline would look something like this:
Instagram API Request -> Image Storage -> Text detection -> Face Detection -> Gender Detection
My pipeline is relatively linear, but my stack has a mixture of layers consiting of 3 neural networks:
- A CNN (Convolutional Neural Network) that determines whether there is text in my image
- A CNN that determines whether there is a face, and only 1 face, in my image
- A CNN that determines whether that face is male or female
I run my testset data through my system, 10,000 processed Instagram images, and get the following accuracy:
Baseline Accuracy: 68%
Improvements can be made; anything below a 95% accuracy is not considered useful in such a system, therefore ceiling analysis is in urgent need to determine what component would yield the fastest improvements.
What to improve?
In ceiling analysis, we manually overwrite a component to provide 100% accuracy. We do this chronologically until all our components are manually overridden, and observe the changes in accuracy, one component at a time. By the end of this process, my algorithms and overall system will be predicting 100% accuracy.
The process looks like this:
- Change your first component to provide 100% accurate results
- Run your system and observe the accuracy %
- Change the sequential component to provide 100% accurate results
- Run your system and observe the accuracy %
- Repeat for your remaining components
My results would look something like this:
Baseline Accuracy: 68%
Perfect Text Detection: 69%
Perfect Face Detection: 78%
Perfect Gender Detection: 100%
As we can see, perfect text detection only yielded a 1% improvement in our overall accuracy. This suggests that perhaps any time invested in improving our text recognition algorithm will likely not improve our overall system that much.
Text recognition is not the issue here.
Moving onto the face detection, again, looks quite strong, but could be optimised as there is a 9% difference given perfect face detection results. Working on this component may be a good bet to improve the overall accuracy. Before making the final call, let's observe the final component.
It appears that gender detection is struggling the most, yielding a 22% improvement in accuracy with perfect results. Now, this is by far the weakest link in my overall system.
Conclusion
Ceiling analysis suggests that working on the gender detection component of my system will yield the best overall improvements. The data is clear, so I would now be confident to assign my team to work on gender detection.
Ceiling analysis acts as a guarantee that you are focussing on the correct component of your system of which improvements are likely to yield substantial improvements in accuracy.
As a project manager, you do not want to allocate resources and time to tasks that will not yield results; be 100% sure that what you assign your team to is worthwhile.
Always refer to the data. Ignore your hunch or gut feeling.
Ceiling analysis is not only limited to deep learning. As a web developer, it can also be used to analyse the speed for your content delivery for example.
For a complex web application that delivers content from a range of sources, utilises complex queries and aggregation from your databases, performs requests to a number of external sources and locations etc, ceiling analysis is a useful tool to determine how to speed up your app.
Is ceiling analysis in your software toolkit?
Try to use this tool in other areas of your life - determine the best course of action before embarking on a task.
You may save a lot of time.