Why is now an excellent time to utilise machine learning?
There used to be a time when taking a photo was highly technical work.
The photographer had to know how to process the film, set up the camera, get the exposure right, tune the focus, configure the flash and process the film and prints afterwards. Every step required expert knowledge and lots of experience. If one didn’t want to master all these steps, the only option was to hire a professional photographer.
Those days are long gone. Today, anyone can take a photo with an iPhone that could easily outshine the work of some experts. All you need to do is to find the right scene, “point and shoot”. The process has been automated.
You still need the eye, expertise and equipment of a professional to shoot a near-perfect photo in some cases. But most of the time, we could just do it ourselves because the camera has been automated, and the cost for performance has dramatically reduced; anyone can “point and shoot” and get a pretty good result.
The same thing is happening in the world of machine learning.
I was involved in a machine learning project a few years ago. The goal was to predict which clients were most likely to default on a loan using various data sources based on half a million records. It took us two weeks to clean and pre-process the data with 200 features, going through them one by one.
After that, we performed feature engineering which requires data and business knowledge. Selecting the models requires knowledge of potential performing algorithms. Tuning the parameters of the algorithms requires you to know which will produce good results. To train the model and compare the results, you needed to know which metrics to use. The whole process was very manual and took approximately two months. I won’t bother you with the coding we had to do but look at the snapshot of one feature analysis I did.
Figure 1: Feature analysis
I wanted to automate all of this manual work. I tried my best to research different tools available to automate as many things in this process as possible, and there are quite a few options.
For feature processing and engineering, there was “Featuretools” to automatically engineer all possible new features and choose only the important ones. How to select the best predictive model from all the evaluated models? You could write a script to select the best performing model based on evaluation score automatically. Model parameter tuning? There’s “Hyperop” to automatically tune them, which is used by a lot of Google research papers as well. One by one, I assembled these “parts” to automate the process.
The whole process was automated and generalised. When I handed the script to my project team members, they only need to know how to start it, and they would get close to best predictions with their datasets, just like the “point and shoot” camera.
As a graduate data scientist, I can develop the process by assembling parts. It means the available technologies have already reached the point where a “point and shoot” solution is possible, at a much larger scale, and for more generalised purposes, to automate these kinds of data science projects.
Think about the implications here. Everyone in the organisation is free to use machine learning capabilities to help predict and solve general business problems when they encounter them, the same way we can take a great photo whenever we see a great scene. Users might get so used to it that could become habitual to use machine learning to solve problems, just like we are used to taking photos whenever we want.
So what is the most important question.
Let’s do some rough maths. It used to take two months to run a data science project like the example above that I did a few years ago, and now it would likely take just a day. Even if we only run a project like this every week, it’s still almost eight times more than what we would previously do. Also, since everyone could run their own projects without the need of a group of highly paid data scientists, the number of projects could be much greater.
Use the ROI of your last data science project as a reference and go figure! All you need is to have the eye to find that scene or problem, and then “point and shoot”.
There needs to be someone to build that platform to start with, however, there might still be more complex problems that will require the assistance of a data scientist. ABM Systems recently partnered with DataRobot to provide clients with a data science solution that enables this.
If you would like to speak to me about how to utilise Machine Learning (MI), please get in touch with me directly via email email@example.com or fill in the form below.