Why Human-In-The-Loop Computing Is The Future Of Machine Learning

November 16th, 2015

Awake Goy

Why Human-In-The-Loop Computing Is The Future Of Machine Learning

November 16th, 2015

Now that machine learning is becoming more and more mainstream, some design patterns are starting to emerge. As the CEO of CrowdFlower, Iâ€™ve worked with many companies building machine learning algorithms and Iâ€™ve noticed a best practice in nearly every successful deployment of machine learning on tough business problems. That practice is called â€œhuman-in-the-loopâ€� computing. Hereâ€™s how it works:

First, a machine learning model takes a first pass on the data, or every video, image or document that needs labeling. That model also assigns a confidence score, or how sure the algorithm is that itâ€™s making the right judgment. If the confidence score is below a certain value, it sends the data to a human annotator to make a judgment. That new human judgment is used both for the business process and is fed back into the machine learning algorithm to make it smarter. In other words, when the machine isnâ€™t sure what the answer is, it relies on a human, then adds that human judgment to its model.

This simple pattern is at the heart of many well known, real-world use-cases of machine learning. And it solves one of the biggest issues with machine learning, namely: itâ€™s often very easy to get an algorithm to 80 percent accuracy but near impossible to get an algorithm to 99 percent. The best machine learning lets humans handle that 20 percent since 80 percent accuracy is simply not good enough for most real-world applications.

<!–