Mining & Modeling: A Match Made in IT Heaven

    March 18, 2016

    By Per Bauer

    Data mining and simulation appear at first to be completely different disciplines, but if you study their overlap, you’ll see how crucial their cooperation is to IT maintenance.

    Could a hurricane result from the flapping of a butterfly’s wings? What about ten thousand butterflies? If you could discover a causal relationship between butterflies and hurricanes, you would have the key to technology that could predict future storm events.

    While the Butterfly Effect has become a cliché, it illustrates the fact that, in large IT systems, seemingly small, unseen factors can have massive, rippling, and non-linear impacts.

    More importantly, it demonstrates the harmonious relationship between data mining and simulation: by analyzing troves of historical IT data to uncover meaningful patterns (data mining), companies can use those relationships to construct models and predict future events (simulation).

    The two techniques are often described as distinct, if not competing strategies, but the truth is that they’re a match made in heaven. This is to say that if IT professionals can confidently find the butterflies that cause hurricanes, they can create a perpetually balmy forecast.

    Wedding Bells in the Server Room

    A wonderful TechTarget article discusses this interrelation in some detail, using nuclear cooling coil failure rates and city storm drain systems to illustrate its point. In the former example, a consulting company used data mining to analyze the historical data of a nuclear waste vitrification plant, determining which variables contribute most to cooling coil failures.

    Using those variables, they could create models to accurately predict when coils would malfunction, during which events, as well as determine how many failures the plant could withstand before shutting down. Replace coils with servers, and it starts to sound a lot like capacity management.

    Yet, “data mining” is somewhat of a misnomer, as TechTarget notes. The process doesn’t involve “digging” for data as much as it does analyzing datasets with sophisticated algorithms in order to discover patterns. Using historical patterns, IT professionals can make predictions about how systems are likely to perform under present conditions. Simulations, on the other hand, extrapolate relationships from those patterns, making predictions about how new variables will likely affect a system.

    In this way, data mining is a fundamental prerequisite for simulations. Conversely, simulations provide essential feedback on the accuracy of patterns found during the data mining process, helping us to determine if we’ve stumbled on an important trend or simply an anomalous fluke. And while IT professionals have begun to point out this natural matrimony, TeamQuest has been segueing these techniques together for some time.

    While Fresh, This Romance Isn’t New

    These strategies are the fundamental drivers behind our complementary Vityl Monitor and Predictor tools, both the advanced descendants from a long line of data mining and simulation software. Where Monitor gathers and analyzes universal datasets from an entire enterprise suite, Predictor extrapolates from that data, using automated modeling to predict likely future performance and predictive analytics to prepare for less likely “what-if” scenarios.

    The benefit of having merged these two technologies over a long period of time is that we have been able to closely corroborate our predictions with tangible results. After decades of tweaking proprietary algorithms and machine learning techniques (the real key to establishing meaningful patterns), our predictions are typically 95% accurate or greater.

    Indeed, there’s also a butterfly effect that can occur within data itself: small errors of analysis can rapidly balloon (especially in complex systems), throwing the predictions of data mining and simulations entirely off base. It’s not only that mining and modeling are a match made in heaven -- they must be used in concert to ensure accuracy. When a harmony of data is achieved, IT experts can guarantee a long and sunny honeymoon.

    (Main image credit: stokpic/Pexels)

    Category: data-management