This is a customer post co-authored by ICL and AWS staff.
ICL is an Israel-based multinational manufacturing and mining company that manufactures products based on unique minerals and meets the basic needs of humanity, primarily in three markets: agriculture, food and engineering materials. Their mining facilities use industrial equipment that must be monitored because machinery failures can lead to lost revenue or even environmental damage. Due to the extremely harsh conditions (low and high temperatures, vibrations, salt water, dust), connecting sensors to these mining machines for remote monitoring is difficult. Therefore, most machines are continuously monitored manually or visually by field workers. These workers frequently check camera pictures to monitor the condition of a machine. While this approach has worked in the past, it does not scale and comes at a relatively high cost.
To overcome this business challenge, ICL decided to develop in-house capabilities to use machine learning (ML) for computer vision (CV) to automatically monitor mining machinery. As a traditional mining company, the availability of internal resources with data science, CV or ML skills was limited.
In this post, we discuss the following:
- How ICL developed in-house capabilities to build and maintain CV solutions that enable automated monitoring of mining equipment to improve efficiency and reduce waste
- A deep dive into a solution for mining control devices developed with the support of the AWS Prototyping program
Using the approach described in this post, ICL was able to develop a framework for AWS using Amazon SageMaker to create other use cases based on extracted vision from about 30 cameras, scalable to thousands of such cameras in their production facilities .
Build internal capabilities through AWS Prototyping
Building and maintaining ML solutions for business-critical workloads requires a sufficiently skilled workforce. Outsourcing such activities is often not possible because internal expertise on business processes must be combined with the creation of technical solutions. Therefore, ICL approached AWS for support in their journey to create a CV solution to track their mining equipment and acquire the necessary skills.
AWS Prototyping is an investment program where AWS embeds experts in customer development teams to create mission-critical use cases. During such an engagement, the customer development team is enabled on the underlying AWS technologies while building the use case over the course of 3-6 weeks and receiving hands-on assistance. Apart from a corresponding use case, all customers need is 3-7 developers who can devote more than 80% of their working time to building the aforementioned use case. During this time, AWS experts are fully assigned to the customer’s team and work with them remotely or on-site.
ICL Computer Vision Case
For prototyping, ICL chose the use case of monitoring mining control mechanisms. A screener is a large industrial mining machine where minerals dissolved in water are processed. The water flows in several lanes from the top of the machine to the bottom. The inflow is monitored for each of the lanes separately. When the inflow runs out of the lane, it is called an overflow, indicating that the machine is overloaded. The overflow inflow is minerals that are not processed by the criterion and are lost. This should be avoided by adjusting the inflow. Without an ML solution, the overflow must be monitored by humans and it potentially takes time for the overflow to be observed and addressed.
The images below show the inputs and outputs of the CV templates. The raw camera image (left) is processed using a semantic segmentation model (middle) to detect the different lanes. The model (right) then estimates cover (white) and overflow (red).
Although the prototyping focused on a single type of machine, the general approach to using cameras and automatically processing their images when using CV is applicable to a wider range of mining equipment. This allows ICL to extend the expertise gained during prototyping to other locations, camera types and engines, and also to maintain ML models without requiring third-party support.
During the engagement, AWS experts and ICL’s development team meet daily and develop the solution step by step. ICL data scientists would either work independently on their assigned tasks or receive hands-on pair programming support from AWS ML experts. This approach ensures that ICL’s data scientists have not only gained experience systematically developing ML models using SageMaker, but also integrating these models into applications as well as automating the entire lifecycle of such models, including automated retraining or model tracking. After 4 weeks of this collaboration, ICL was able to move this model into production without needing further support within 8 weeks and has since built models for other use cases. The technical approach to this engagement is described in the next section.
Track mining spotlights using resume models with SageMaker
SageMaker is a fully managed platform that addresses the entire lifecycle of an ML model: it provides services and capabilities that support teams working on ML models from labeling their data in Amazon SageMaker Ground Truth to training and optimizing the model, as well as hosting ML models for production use. Prior to the engagement, ICL had installed the cameras and taken photos as shown in the previous images (left image) and stored them in an Amazon Simple Storage Service (Amazon S3) bucket. Before the models can be trained, it is necessary to generate training data. The joint ICL-AWS team tackled this in three steps:
- Label the data using a semantic segmentation labeling task in SageMaker Ground Truth, as shown in the image below.
- Preprocess the image labels using image augmentation techniques to increase the number of data samples.
- Separate the labels into training, test, and validation sets so that the performance and accuracy of the model can be adequately measured during the training process.
To achieve production scale for ML workloads, automating these steps is critical to maintaining the quality of training inputs. Therefore, whenever new images are labeled using SageMaker Ground Truth, the preprocessing and segmentation steps are performed automatically and the resulting datasets are stored in Amazon S3, as shown in the model training workflow in the following diagram. Similarly, the model development workflow uses inputs from SageMaker to automatically update endpoints whenever an updated model is available.
ICL uses several approaches to apply ML models to production. Some include their current AI platform called KNIME, which allows them to quickly deploy models developed in the development environment to production, industrializing them into products. Several combinations of using KNIME and AWS services were analyzed. the previous architecture was the most suitable for the ICL environment.
The built-in SageMaker semantic segmentation algorithm is used to train models for screen grid area segmentation. By choosing this built-in algorithm over a self-built container, ICL does not have to deal with the undifferentiated heavy lifting of maintaining a Convolutional Neural Network (CNN) while being able to use such a CNN for its use case. After experimenting with different configurations and parameters, ICL used a Fully Convolutional Network (FCN) algorithm with pyramid stage analysis network (PSPNet) to train the model. This allowed ICL to complete the building of the model within 1 week of the prototype being committed.
After a model is trained, it must be deployed so that it can be used to track the screener. According to model training, this process is fully automated and orchestrated using AWS Step Functions and AWS Lambda. After the model is successfully deployed to the SageMaker endpoint, the incoming images from the cameras are resized to fit the model’s input format and then fed to the endpoint for predictions using Lambda functions. The result of the semantic segmentation prediction as well as the overflow detection are then stored in Amazon DynamoDB and Amazon S3 for downstream analysis. If an overflow is detected, Amazon Simple Notification Service (Amazon SNS) or Lambda functions can be used to automatically reduce the overflow and check the corresponding lanes in the affected screener. The diagram below illustrates this architecture.
conclusion
This post described how ICL, an Israeli mining company, developed its own computer vision approach for automated monitoring of mining equipment using cameras. We first showed how to tackle such an organizational challenge focused on enablement, and then gave a detailed look at how to build the model using AWS. While the challenge of tracking may be unique to ICL, the general approach of building a prototype together with AWS experts can be applied to similar challenges, particularly for organizations that lack the necessary AWS knowledge.
If you’d like to learn how to create a production-scale prototype of your use case, contact your AWS account team to discuss a prototyping engagement.
About the Authors
Marcus Bestehorn leads the customer engineering and prototype teams in Germany, Austria, Switzerland and Israel for AWS. He has a PhD in computer science and specializes in building complex machine learning and IoT solutions.
David Abecasis leads the data science team at ICL Group with a passion for educating others about data analytics and machine learning while helping to solve business challenges. He has an MSc in Data Science and an MBA. He has been fortunate to research spatial and time series data in the field of precision agriculture.
Ion Cleopas is Sr. Machine Learning Prototyping Architect with MSc in Data Science and Big Data. It helps AWS customers build innovative AI/ML solutions by empowering their technical teams on AWS technologies by co-developing prototypes for challenging machine learning use cases, paving the way to production.
Myron Perel is Principal Manager of Machine Learning Business Development with Amazon Web Services. Miron advises Generative AI companies to build their next generation models.