The team tasked with evaluating Industrial IoT Predictive Maintenance solutions often lacks direct Machine Learning discipline expertise.
As industrial plants digitalize their production assets, IIoT Predictive Maintenance is now an integral part of the Smart Factory roadmap. A common scenario is that the decision-making process to procure IIoT Predictive Maintenance technology can be led by representatives of plant management, Operations & Maintenance (O&M), Operational Technology and Information Technology. In many cases, the industrial is unable to analyze vendors’ capabilities and must rely on third party consultants for guidance.
This article provides five questions to ask any IIoT Predictive Maintenance vendors you are considering. These questions can be used in a RFI/RFP or informally during solutions briefings.
#1 Which Machine Learning Methodology is the IIoT Predictive Maintenance based upon? Why is this Methodology used? Are there any more advanced Methodologies?
The selection of a Machine Learning methodology is not a theoretical exercise. For the industrial plant, it is important to keep in mind that Methodology impacts the performance and cost of the IIoT Predictive Maintenance solution.
Let’s compare Unsupervised and Supervised Machine Learning.
With Supervised Machine Learning, the learning algorithm needs to be “trained” on the industrial machinery using data labels or classifications. When the Supervised Machine Learning algorithm recognizes new data, it then associates it with the data labels that is has been trained on. To use a simple analogy: we can teach an algorithm to recognize the difference between a banana and pear. When a new fruit is introduced, the algorithm will classify it as either banana and pear based on the labels it has trained on.
The Unsupervised Machine Learning algorithm is not provided data labels for training. Instead, the algorithm generates the label by analyzing vast amounts of data. In the example above, the algorithm analyzes vast quantities of fruit and can then recognize and label a banana or pair even though it was not given a definition or example.
Keep in mind that training the algorithm required significant input from those professionals who understand how the machinery operates, and intensive modeling and programming work. Plant technicians and programmers will spend lots of time together. Putting aside the data science, from a practical perspective, Unsupervised Machine Learning cuts down on the time and expense associated with training the algorithm.
#2 Is the Machine Learning Algorithm selection process manual or automated?
It is important not to gloss over the topic of Machine Learning algorithm selection. If your vendor is providing Automated Machine Learning, then the selection of the algorithm to be used for a given situation is automatically determined by the solution itself without the need for human input.
When the selection process is manual, a Big Data professional needs to make the decision about which algorithm to use from a library of hundreds of even thousands of algorithms.
Industrial plants cannot hire enough Big Data scientists and engineers to deploy solutions locally. According to estimates by IBM, demand in the US for Big Data scientists and engineers is likely to grow by 39% between 2017 and 2020. Equally troubling for the industrial sector is that higher paying industries including Financial Services, Insurance, IT and Professional Services are competing for the same professionals.
How is this relevant to Machine Learning algorithm selection? Industrial plants cannot rely on recruiting sufficient numbers of Big Data professionals to manage onsite IIoT predictive maintenance. At stake is the ability to scale a solution across an entire plant. If, for strategic reasons, management sets policy that only internal Big Data resources will take responsibility for tasks such as algorithm selection, it should recognize that there will be a high premium associated with this decision.
#3 What level of technical expertise is required to use the Industrial IoT Predictive Maintenance solution?
In the Emory University Future of IIoT Research Study sponsored by Presenso, Reliability and Maintenance personnel were asked to identify the largest blockers of Industrial IoT Predictive Maintenance. Skills shortage (data scientists), was cited as the strongest factor negatively impacting the deployment of Industrial IoT Predictive Analytics. Over 80 respondents indicated that skills shortage had a strong (44%) or moderate (37%) impact. Only 8% suggested that it had no or limited impact on deployment.
From interviews with O&M professionals, there is a concern that Maintenance and Reliability staff will be required to gain knowledge of discipline for which they are untrained. Whether or not these concerns are warranted, they are not isolated.
What does this mean for the industrial plant? Selecting a solution that requires existing plant personnel to gain new skills is risky. Technicians and engineers can be expected to access monitoring systems and dashboards with their existing level of skills. In other words, solutions need to adapt to the current industrial plant environment, not the other way around.
#4 Can you give me a customer reference from my vertical that I can call?
This does not need explanation. A vendor that cannot provide a customer reference from their vertical may not be ready for prime time. Perhaps there is a good reason that that a company starting out does not yet have references, but consider the risks involved.
#5 How does the vendor’s anomaly detection results compare with benchmark data?
When evaluating a vendor, you want to compare their performance relative to others. One effective way to do so is to compare vendor test results against published benchmarks. Anomaly detection is a domain that is researched and used in a variety of applications. A partial list includes computer servers, cybersecurity, medical fields, consumer behavior and now, machine monitoring. It serves as the basis for all industrial failure analysis and prediction solutions.
For instance, Presenso has benchmarked its anomaly detection capabilities against three algorithms for anomaly detection developed by Yahoo, Netflix and Twitter, respectively. These companies released their anomaly detection algorithms to the public as open-sources.
In the first test compared with Yahoo, Netflix, and Twitter, the results were decisive: Presenso outperformed the other algorithms in all measures – the accuracy of the results, calculation time and memory consumption. Using a cost-function that inputs the above 3 measures and calculates a single measure, Presenso score was 88% whereas the next best runner-up scored only 51%.
In the second test, Presenso was used to predict Remaining Useful Life (RUL) of jet engines using a data set that was released as part of a contest organized by NASA. The criteria measured by the organizers were: Recall, Precision, F-measure, Mean Absolute Percentage Error (MAPE), and Root Mean Square Error (RMSE). Presenso results of correct RUL predictions surpassed all those submitted by the winners of this contest, including Microsoft MAPE. Presenso outperformed by 7.7 times and RMSE by 5.6 times better than the next best competitor. Precision, recall, and F-measure were 6%-16.2% better.[/box]
Recommendation and Conclusion
This list we provide in this article is a starting point and questions should be developed by the team assigned to decision making process.
There are two steps that are critical to the purchase decision.
First, business and functional requirements should be formalized, and each vendor should be evaluated relative to objective criteria. For more information on how to write a requirements document for an Industrial IoT software solution.
The second step is to conduct Proof of Concept (PoC) before reaching a purchase decision. For a step-by-step guide for conducting a PoC, please refer to this document.