Hi and welcome. My name is Marilyn, and I'm a systems engineer at JMP. Here to talk to you today a little bit about accelerating mineral identification with JMP Pro, AI powered insights for the mining industry. Before we get started, let me just give you a little bit of background and motivation for this presentation.

In the mining field, there are a lot of challenges in identifying or classifying minerals. Identifying minerals in the field can be challenging and can require specialized knowledge and can be susceptible to mistakes. Traditional methods depend heavily on manual feature extraction from the minerals to be identified using microscopic examination, for example, which can be both labor-intensive, time-consuming and prone to errors.

Additionally, as mining operations move into more remote areas or to lower grade deposits, there's a growing need for on-site real time mineral classification. My proposed solution to these challenges through this poster is leveraging the Torch add-in for JMP Pro to build a model that can identify or classify the minerals in real time.

For my example today, I want to let you know that JMP Pro is required to run the Torch add-in that will use the images as the Xs or inputs to create a classification model to predict future minerals. In this example, I used a dataset with 953 images of seven different minerals, and I used this to train a model to, predict minerals for mineral classification. This data set is a version of a data set from Kaggle.

Now, in real life, probably mining engineers would want to go ahead and crush the ore to the size that they would typically use in their operation, and then take images, take pictures of the crushed ore. While they do that, they would also want to think about where the camera is going to be placed in the lighting with the objective of making the images that they take to train their model to look as closely as possible to the images that they'll be taking in real time for classification in the future.

Today what I'd like to do is walk you through the steps. You'll see how easy it is to start with some image images of, minerals that have already been identified and to quickly build a predictive model, that can be used for classifying the minerals in real time in a mining operation. First step is to install the Torch add-in, and this is available for JMP Pro in the JMP Marketplace. Let me show you what that looks like.

The JMP marketplace, has a lot of extensions that can be downloaded to extend the functionality within JMP. This particular add-in is right here. It's called Torch Deep Learning for JMP Pro. To download this add-in, we could just click the download button. Once the download is finished, we can click and drag that download on top of an open JMP window to install the add-in.

There's information on this page as well about system requirements and supported JMP versions, as well as help, and documentation for using the add-in. Once the add-in's installed, you'll have a new menu at the top titled add-ins in JMP. Here you can see my Torch deep learning add-in. I have an image here in my JMP journal that shows the options within this, add-in.

There's an option to make K fold validation columns. We're going to take a look at that today. We're going to fit our Torch model. This add-in also comes with several sample data sets. There are some image utilities that help with manipulating, and changing the images around. We'll take a look at some of those options today as well. When this add-in is downloaded, it comes with, certain models and additional models can be downloaded for this add-in. Then the help and documentation also lives within this menu.

That's step 1. I have the add-in installed. My next step is going to be to import the images into JMP and to clean the data. First, we're going to import mineral images for training the model into JMP. To do that, we're going to use JMP's import multiple files feature. Now, before I go there, I just want show you how these mineral images are arranged in this folder. To do that, I'm just going to bring this file over. You can see that this folder is titled mineral images, and within it, there are a lot of subfolders. Each of these subfolders is titled according to the mineral images contained within.

For example, if I go into the biotype folder, you'll see here that I have a lot of images of that particular mineral. That's the way that this dataset is organized, or these images are organized on my machine.

Now we're going to use JMP's file, import multiple file features to bring these images into JMP so that we can train our model. We're going to do this. I'll show you the steps within the journal, and then I'll show you what to do for each step. To start, we're going to use file, import multiple files. This is already set to the location of that folder with the images that I have. What I want to do is go ahead and include the subfolders. I can see here that this particular folder has only more folders in it. We'll include the subfolders.

Now I can see down here that I've got my images. We can see there's a lot of JPEGs in here and there's different file formats. I've got some PNGs as well. That's perfectly okay. Other than that, in the resulting data table, I want JMP to add a column that has the file name. The file name is going to reference the folder that has the mineral title in it. What I'm going to get is a column that has the name of the mineral and another column in my data table that has the corresponding image. I want JMP to go ahead and stack all of these files, all of these images into a single JMP data table.

Now go ahead and click import and JMP's going to go out to that folder and bring in all of those images for me along with a new column that has the name of the mineral in it. Here we go. Here's the resulting data table. Really easy to import my 953 images of these minerals. Then we'll take a look here. I don't need these two columns to the right-hand side, so I'm just going to delete them, so we can take a closer look at what we have.

Each row here has the image of one of the minerals. In the second column here, I have the name of the mineral followed by the title of each of the images and that JPEG. I'm going to have to do a little bit of data cleanup. That's going to be our next step.

Now we've imported multiple files. The next step is going to be to clean the data. The first step in cleaning the data is going to be with cleaning up that labels column. We're going to use recode and the advanced options to extract the segment. I just want the segment with the name of the mineral. To do that here in my data table, I can right-click, and I'm going to go to recode.

I want to store these results in the same or existing column, and I'm going to use the hot spot or red triangle menu to access the advanced options. I'd like to extract part of a text segment. Here I can use this control. What I'm looking to do, again, this is what I have in the data table, and here I can see the result.

This looks pretty good to me. The result side here shows that I'm just going to have the name of the mineral, and I can strip out the name of each individual file I'll give it an okay and a recode. Here we go. We have the names of our minerals. If we expand the header graphs, we can see these are the same mineral names that we saw in the folder that I showed you earlier.

All right, so that was our second step. Let's take a look at our next step because we do have a little bit more data cleaning to do. We are going to take a look at, the image utilities within this add-in. In this particular data set, the images of the minerals that I have, have some extra items in their fingers and hands, and that could cause some problems with fitting the model.

I am going to modify these images. We're just going to crop them to keep the center 50% of the image and cut out anything that's on the outside of these pictures. To do that, you can see here the image utilities menu. This is part of the add-in for Torch. I come up here to my Torch deep learning. Here are my image utilities.

Okay, so we can get image sizes that just stores in a separate column. The size of the images, we can resize the images if we like, make them bigger or smaller. I'm going to go ahead and crop them, like I said. That's this here just shows you a little picture of what that's going to look like. Let's go back to that data table and let's fix up these images using our add-in Torch deep learning, image utilities, and I'd like to crop the images.

There's only one column with images in this data table, and I can change the settings here, but this is exactly what I want. These settings are following this example. With these settings, JMP is already telling us we're going to get the central 50% of the images, which is exactly what we want. I'll go ahead and hit run. JMP is going to manipulate these images.

Then for comparison, I'm going to show you a little bit of before and after for each of these. Now there's a little glitch here, in what we're seeing, so I'm just going to go ahead and save this file to my downloads' folder. I'm just going to go ahead and reopen it, and you'll see here in a moment why.

Here we go. I can just open it for my recent items. Here we go. Here's what I want you to look at. Take a look in row number 4. In row number 4, right here, you can see that there's some fingers in this image, for example. To the right-hand side, after we cleaned that and cropped the image, you can see that that's sort of zoomed in, and we cut out those extra things in the picture that we don't want to, include in the images that we're using to build our model.

We're going to use that new column picture crop as our X and our Y for our model is going to be the mineral, the correct mineral. Let's take a look at our next step. We've modified the images, imported and cleaned the data, and the next step is going to be to create some K fold validation columns.

To do this, we're also going to use this Torch add-in. Let me show you what that looks like. In my data table, add-ins, Torch deep learning, we want to make some K fold columns. In this case, we're going to take a look at the response column, the Y, which is the correct mineral, and we're going to also use that column for stratification. What we're doing here, is we're going to create three columns that group this data.

In each of those three columns, there's going to be five groups. We're going to use these K fold columns for validation as we build our model. The idea is going to be, we're going to use just, part of the data to build the model and other parts of the data to test the predictions and then update and improve the model. That's the idea, just using bits of the data at a time. Through this menu, we are creating those stratification columns that allow us to do that.

I'll go ahead and hit go. JMP is doing something really fancy using some advanced DOE techniques to create these grouping columns for validating, and improving the model for us. There we go. Here we can see there's three new columns, fold A, B, and C. To use all of these columns and all of these groupings for validation, it's actually going to take a pretty long time, maybe a couple of hours to run on my laptop. Even though if we were fitting our final model, we'd probably want to use all of these columns and let JMP run a long time to fit the model.

Today, I won't make you wait so long. We're just going to use, single holdout validation, to make things a little bit faster for purposes of this demonstration. To do that, I'm going to use column utilities and make indicator columns. For our validation purposes today, we're just going to use this column that says number 1. We might use, say, this part of the data, 763 rows to fit a model, and then maybe the remaining 190 rows to test the predictions and update the model. We'll just be using simple validation for our example.

But that's how we can create K fold validation columns through this add-in. Then our next step is to go ahead and fit that model and iterate. What I mean by that is we're going to fit the model and then just tweak it a little bit. We have a lot of controls for these Torch models. Little things that we can change to see if we can make an improvement in the prediction. First, we're going to specify the model. We're going to tell JMP what we want to predict.

Let's go back to the data table and do that. Add-ins, Torch deep learning, Torch deep learning. Here, I'm going to tell JMP that the Y or response I want to predict is the mineral, and I want to predict it based on that 50% crop of the images that we had. We're just going to use some simple validation for this example, and then we'll give that an okay.

We've got more options to specify here. I'll speak to these very briefly, and we do have a lot of other documentation about these options. Here we can control how this model is being fit. For example, under this image model drop down, these are models that have been pretrained on thousands and thousands of images. We're going to use a pre-trained model and then add our own images to customize the model for our needs. I'm going to choose Google Net. I've had some luck with this particular dataset in that image model. If you're not sure, of course, you can just use the defaults in JMP.

I'm also going to tweak here the number of epochs. That's really the number of passes through the data. I'm just going to reduce this number a little bit to speed things up. Of course, we can also control the learning rate or how quickly the model adjusts to errors from a previous model. Then we've got these additional advanced options here to the right. Let me go back to my journal to speak to those just really briefly.

We have a lot of advanced model controls, and I'll say that these are broken down into three buckets. First, there is model specifications. What should the model be that we're fitting? We have some controls for training. How should we fit this model? How should it be trained? Then we also have some options for image modeling. For manipulating and doing things to the images.

Let me show you what that looks like, and we'll go ahead and let fit this model for us. If I expand the options to the right-hand side, here you can see there left. It can seem a little overwhelming, but we do have a nice bit of hover help. If you just circle your mouse there, jump gives you a little tip on what each of these options is for. The layers in a deep neural network are essentially similar to the number of nodes in a neural net. I'm just going to go with eight for our example.

We'll take a look at this next bucket here, and this is the training bucket. Here I'm going to modify the batch size. The batch size is how many rows of data we're processing at a time. Because my computer is not going to be very fast otherwise, I only want to take a look at 64 rows of data at a time. Then lastly, we have these image options to the right-hand side. Here, image size helps us control the size of the image. I'm going to say 228 for that. That seems to have worked out okay for me in this particular dataset.

That's all I'm going to change for now. But the idea of fitting a model using Torch deep learning is, you come in here, you fit a model, and then you iterate. Maybe you make small tweaks and changes to try to improve the model's predictive ability or performance. I'm just going to go ahead and give this an okay. JMP's going to go ahead and crunch through that data, and we'll see what kind of model we get.

As JMP does this, we're going to get some updates in this little window that show us the performance so far, both on the training data. That's the data that is used to train the model and then validation. We're really going to focus in on the validation numbers because that's going to give us a better idea of how well this model is going to predict when we give it some new images.

Here I'm looking at the accuracy for both the training and the validation, and it's looking pretty good so far. We're over 90% on the training data and accuracy is somewhere between 70 and 80% for the validation data. Now this is a model that we trained really quickly. If we use different options, we had more epochs, for example, we could probably get another, better model, without making too many changes to what we already have.

But let's take a look here because this is our next step, which is to compare models. Now we've built just one model. We'll take a look here at what it looks like with one model. As we build more models, JMP is going to add more information here in model comparison, so we can compare between different models. For this initial model, model 1, the response was mineral. This is the information that we entered into the window to fit our model.

We have information about R-square, but I'm not going to look at that because we're not predicting a number. We're classifying images. We're more interested in the accuracy measures and in particular, the validation accuracy. This is already looking pretty good. We're almost at 90% accuracy with this model that we trained really quickly.

Here, we can get a graphical representation of how that model evolved, while it was training. You can see that there's two lines here, one for the training data and 1 for the validation data. In general, what we can see for this trend is that the loss or error in the model is decreasing as we go through different epochs. As we make more passes through the data to extract more information from those images, the error is being reduced and our predictions are being improved.

Seeing as how these lines are still declining here, we'd probably want to increase the number of epochs and let this model have a few more passes at the data, and that might help us improve this model. These are some of the things that we can take a look at to assess how well the model performs. Here we have these actual by predicted plots where on the left side we see the real mineral and the predicted on the bottom probability. Essentially on the right-hand side of these graphs, these are well predicted on the right side. On the left side, these are points that are not predicted so well.

Everything here is interactive, and it's dynamic. If we go back to the data table, for example, and we set our images' column as a label, as we hover over these points, we can see which image this looks like it was quartz and seeing as how this image is pretty far to the left on this graph, this was probably incorrectly classified. It was quartz. We could also potentially just highlight all of the images that were not correctly classified. Looks like that's about 40 rows right here.

We could go back and take a look at those images and see if there's any potential issues with the images, or maybe we need to gather more images for those particular minerals to try to improve those predictions.

Down here, a little more information, we'll take a look at the confusion matrices for the validation data. What we'd like to see on both of these diagonals, whether we're looking at the counts or the rates over here, we want this diagonal to have big numbers and hopefully zeros above or below.

For biotite, the actual mineral is listed on the left-hand side and across these are the predicted minerals. About 75% accuracy on biotite, boronite 92%. We can see how well we're doing with each of these different models. Here it looks like 77.8% for muscovite. It looks like muscovite is the mineral that we're doing the worst job at predicting or classifying with this model.

Again, in my case, we just have one example for the comparison. I want to show you back in this journal what it would look like if you were comparing several models. As you as additional models are fit, as you tweak the options, JMP is going to keep adding those additional models in the model comparison. Then you can compare the validation accuracy for multiple models, and you can also visualize the performance of these models and compare them.

Again, just try to make an improvement and find the best possible model before moving ahead and deploying. We talked a little bit about comparing models and evaluating these results. Let's talk about a little bit more about evaluating the results.

There's some really nice interactive plots, that you can see in this output. I want to point out that for each model that you have, there's additional options here for things that you can save back to the data table. Here I could go ahead and save the embeddings, for example. Let me click there and I'll go back to my data table and show you.

You can see here there are 8 columns for the embeddings. What are the embeddings? If you recall in the model controls, I set eight as the number of nodes for my neural network. That's what this is. These nodes are the features that were extracted from the mineral images. I could, if I wanted to use these embeddings as the Xs or inputs for predictive models using other modeling platforms within JMP. I could just use Torch to extract the information and use that to predict or to build a model using a different platform if I'd like.

Another great option, at least for me, because I like to visualize how well my model is predicting in this way, you can save the predictions into the data table. Let me go back there and show you what that looks like. This is the most likely or predicted mineral that we stored here in the data table. Using Graph Builder, it's really easy to graph the actual mineral on the Y. I'll also use that as my color variable. I'm going to put the predicted mineral on the X-axis.

What does this represent? Well, big boxes are the minerals that were correctly classified and all of these points outside of that were incorrectly classified. This was quartz. It was classified as boronite. This was the mineral is muscovite, and it was incorrectly classified as biotype. It makes it easy to see which images were incorrectly classified and how many of them were incorrectly classified.

One more option that's worth mentioning. From this red triangle, we can also generate the Python code for making real time predictions. If this was our final model, we're happy with these predictions, and we want to implement this in real time, this is where we can save the Python code. Now I'm not going to do that, but I do want to go back to my journal and talk about what we could do once we export, that Python code.

We'd want to use that model to predict. In a mine, here's what we might do. Once we have the trained model, and we exported that Python code, we could integrate that into an on-site type of system. What that would look like, we'd need a camera to take pictures of the minerals or the ore that are coming out of the mine. We'd want some kind of edge device where we're going to add our Python code so that that Python code can take in the new mineral images that are being taken by the camera and just quickly classify them as this mineral or that mineral. That's the idea. We could make real time predictions.

What could we do with that? Well, we could use that information to inform decisions like adjusting milling parameters. We might automatically sort the ore from the waste based on those predictions. Or we might want to make some real time estimations about how much of each mineral we're going to extract from the ore. This could be really useful if the information is coming in real time.

Just to wrap up, a few takeaways here. I hope that you have seen through this demonstration today that Torch Deep Learning and JMP Pro offer a nice no code solution for mineral classification. Models can be trained quickly using labeled images and JMP has all of the tools to help us import images and clean up the data and pre-processing the data before creating the model. Real time classification is possible with pretty simple hardware, and this approach supports faster, more accurate decisions in mining operations. With that, I thank you very much for watching.