Find the oddballs before you foul out: eliminate business latency with outlier detection
Delays can be devastating, particularly in today’s businesses where a web of vendors, customers, partners, competitors – and the web that is the Internet itself – can cause a small localized mistake to erupt into a worldwide crisis in hours. Finding those small mistakes, the slippage between the many gears which makes your company work, among millions of metrics in a large organization is a herculean task which requires the alacrity of The Flash.
Business latency is a fundamental business problem because market competition is simply driven by firsts: the first to successfully fill a market niche, the first to solve an industry-wide problem, the first to become compliant with new industry standards and regulations, the first to build buzz and market share, the first to scoop up scarce, specialized talent, and the first to adapt to the continuous changes and challenges of the industry.
Being second in any of those categories can be fatal, just ask MySpace and Sears. In order for companies to sprint their fastest, they must be continuously on top of their vital business metrics, which can signal they’re about to stumble.
Quick decisions require quick outlier detection
The beginnings of a business disaster can first manifest as outliers in data sets, hidden in one among a million metrics. This is also true of new business opportunities that reveal themselves as unexpected positive deviations. To catch both types of opportunities – chances to do more business and to head off potentially costly disasters – businesses which can, monitor every metric they can gather.
Casting such a wide net sounds completely logical: since what isn’t measured can’t be improved, measuring everything can lead to systemic improvement, right?
Well, only if what’s being measured is also being properly analyzed and monitored in real-time, and this is because systemic failure can occur in hours. Thus, there is an opportunity to catch these problems in their earliest stages. By finding the one problem quickly, you can bend the curve before a cascading failure occurs. Even in the fast-paced world of business, things go bad quickly, but not instantly, which leaves open a narrow window of time for outlier detection systems to make a real difference.
The goal of using outlier detection systems is to identify, characterize and fix the bad outliers before they result in large amounts of lost revenue. Outlier detection systems can also help your business capitalize on good outliers by giving you the information you need to strike while the iron’s hot.
You won’t always be saved by the bell (curve)
There are lots of mathematical and statistical tests for outliers, but almost all of them assume a Gaussian distribution. Real key performance indicator (KPI) time series seldom exhibit that textbook bell curve. Therefore, a much more flexible, robust approach must be used which avoids false positives by maintaining high accuracy.
In modern outlier detection systems, advanced machine learning is used at many levels: initial normal metric learning, outlier detection, significance ranking, and anomaly grouping (concise reporting). Both supervised and unsupervised, univariate and multivariate algorithms are used in a system which can accurately find outliers in metrics showing any kind of distribution.
When you employ a robust outlier detection system which isn’t chained to the bell curve, any metric from segment of any business in any industry can be monitored: web page load times, disk I/O latency, request timeouts, app uninstalls, etc. And outliers in those metrics can be found in real time. With that robustness comes flexibility. This means that now, you can respond to any incident impacting your organization. Universal outlier detection enables quick reaction of any type of business incident, so you won’t be hemorrhaging revenue for hours before you’re aware of the problem.
The ability to quickly and accurately find any type of anomaly in any metric brings the benefits of machine learning at scale to your business, so that you can resolve complicated, urgent problems.
The machines are coming…to help us
AI can produce transformative results in your company when combined with your data, just like it has in other areas. A poignant example involves the intersection of mental and public health: suicide prediction.
In an early, but promising set of trials, AI was able to predict with 92% accuracy whether or not someone was going to attempt suicide within a week. This particular AI was trained on hospital data from over 5,000 patients. Very welcome news, since our current risk assessments for suicide are almost worthless. Fast and accurate prediction of a person’s likelihood for attempting suicide could enable mental professionals to intervene with treatment for the underlying factors, saving thousands of lives a year and sparing loved ones the pain and loss from a death which in at least some cases could be prevented.
This example alone gives a powerful counter-example to widespread fear of AI rising up against its human creators. Far from being the brains behind a Terminator, machine learning could save many human lives.
Suicide prevention isn’t the only area which could benefit from specialized AI. An AI-powered spacecraft control software commanded NASA’s Earth Observing 1 (EO-1) satellite to take images of a new volcanic eruption before earth-based scientists even sent in their request to NASA. That removal of lag time enabled them to gather data throughout the whole event, and prevent a rare opportunity for gathering scientific data to be missed. Aside from the purely scientific value, advances in volcanology could also give us the insights we need to save the lives and property of those living near active volcanoes.
This same powerful technology can spot the eruptions occurring in your business right as they happen, giving you the more of an increasing rare commodity: time.
