Hi, my name's Steve Maxwell.
I am an engineer.
I work in the semiconductor industry.
Today, I'm going to give a presentation on how I use JMP
in the building of the image analysis pipeline.
A little bit of background on the problem.
Two of our module groups at our facility
ran into an issue with a feature
where there was erosion on the sidewall of the feature.
The SEM system that we used to measure
the size of this feature wasn't detecting this feature,
and it didn't have the capability to characterize this.
It's an older piece of equipment, but it's also paid for so we liked it.
The approach that I employed to help them out
was to build a separate analysis pipeline to analyze images,
to classify images that's having damage or not ,
and it produce a metric that could be sent to SPC system for tracking purposes.
I use JMP primarily to analyze data,
to build a model system deploying and manufacturing,
as well as, more recently,
integrating some of the actual image analysis into JSL.
I'll show some examples of that.
This is a sample of what it is that we're looking at.
This is a good image the left- hand side,
and this is a bad image on the right-hand side.
We can see here the like variation and erosion.
That's what we were looking for in these images.
We could see from here that the measurement,
which is done on the inside of this
wouldn't really reset to these features that lands at the same spot every time.
We're not able to tell by just using our normal methods.
This is a little bit more detail about what we were seeing.
This is an example of what some of that data looked like
if we could see from here that though the process was having issues,
we were not detecting it with the measurements that we were using.
We launched into an approach to address this issue.
I use the standard ETL format for most data science problems.
Extract, transform, load.
I got to break it into four categories,
configure, collect, analysis and report.
Configure and collection, standard methods.
Analysis, that's the key to this.
That's how I'm reprocessing the images
and then extracting metrics for those images that we can use to build a model
to determine whether or not, we could detect the issue at hand.
The approach for that analysis is to convert the images from RGB,
which is come off on the system, the measurement system,
and convert those to gray scale.
Then the images are cropped, remove any measurement details,
everything like that out of them.
Then the next step is to apply median to these images.
The reason why do that is we're going to blur the images
and remove some of the noise out of the image.
This is a key step for image analysis and image processing.
The approach was used as an experimental approach different kernel sizes,
different kernel shapes, but in the end, is settled on
using a disk- shaped kernel with a pixel radius of like 5, 7 , 10, and 15 pixels.
For those that are unfamiliar with that terminology,
kernel is basically just an array
that's used to deteriorate the image when you're reprocessing it.
After that was done,
extraction of the metrics from the reprocessed images was developed,
used several different standard metrics,
structural similarity index, mean square error, Shannon entropy,
then a few different ones like gaussian version of SSIM,
as well as looking at the means of the standard deviations of the
SSIM image gradients from the gaussian weighted images.
From there, used dimensional reduction techniques
to extract which of those metrics was most relevant,
to try and build a model, a predictive model
and then convert that into some of that metric
that could be easily interpreted on the manufacturing
while running production.
One of the follow-ups on this is when doing the metric extractions,
the reference image exploits the image that
used a kernel with a pixel radius at five,
and then the 7, 10, and 15 kernel radius is prepared to that image.
This shows a little bit more detail about
what the analysis pipeline approach looks like.
The offline approach, we start with the image acquisition.
Obviously, this is done during the manufacturing process.
After that, initially, this work was done Python,
which was to convert the grayscale crop, de noise, the images,
and then feature generation metric extraction, SSIM,
mean square, entropy, calculations, etc.
JMP JSL has a Python wrapper that you can use for integration.
What's great is you can actually move a lot of this code
in to the JMP environment.
I'll go into a little bit more detail about what that is
and how I use it to help speed this process along.
The final component of this was to develop a quality metric.
This is very JMP-intensive step as well.
Then once a metric is determined,
you move that over into an online process
in which, the manufacturing data is analyzed,
and then that information is pushed to a SPC system.
A little bit more about Python/ JSL Integration,
the Integrated Python analysis code into JSL using Python/ JSL wrapper.
The data transfer occurs by converting the JMP table into Pandas Data frame.
The data frames are passed in an analysis code.
The results are returned from Python to JMP as a data table.
What's great about this is it enables using
common image analysis libraries as skimage, scipy, PIL, OpenCV , etc,
to perform the work while keeping the data within the JMP framework.
This, in my opinion, was key because I'm not moving data
from one system or one platform to another platform and back and forth.
I was able to keep it all in one spot.
This is a little bit more detail about
what the analysis pipeline is doing when it's reprocessing the images.
This is what I was able to move from just running slowly in Python
to integrating it into JSL using, let's say, Python libraries.
You can see, here's a bad, here's a good.
We cropped the image to only show
only take basically this component of it, convert it to gray,
and then applying medium filters to blur the images with different kernel sizes
as you can see as we increase the kernel sizes,
they become a little bit blur.
What I want to show takeaway here is for a good image, you can see
the kernel with disk size 5 versus, 7 versus, 10 versus 15,
look pretty similar to one another .
Whereas disk 5, disk 7, disk 10, disk 15, on a bad image,
you can see that by time you get to disk 15,
this image, or kernel size 15,
this image is starting to look similar to this.
By removing the noise from this image,
and then denoising the images incrementally
using different kernel sizes.
Th is what enables us to successfully extract metrics
that we can use to differentiate between the good versus the bad.
Sorry, hold on here. Let me go back up.
At this point, I'm going to launch demo that shows
how I'm moving this data into JMP
and then what it looks like as it basically reprocessing this stuff.
How I do that is the first thing I need to do is I need to build a data table.
It's a great feature in JMP,
you go to file, import multiple files here.
Then what I've got is I've got basically a sample set of just five images
that I'm going to combine all t ogether into one data table.
What it does is it actually import the images as well as the image file names.
You go here, click that file column name, click import,
and
what we get,
is this.
We've got an image of our picture that we want to analyze.
We've got a file name tag next to it.
We've got the data imported image.
Now, the next step is to run our analysis code
against each of these images and start extracting metrics.
I've got a...
modified version of that code here.
The way this code works is
there's a python initialization that occurs JSL.
And then once that's done,
the data table that you want to analyze
is past eight to Python as a data frame.
At that point under the Python submit,
basically, you convert over to Python code.
And then you're using standard approaches,
standard coding procedures for Python
and analyze the image, as you can see here,
we've our library imports, etc.
How high this code was, basically to find a bunch of functions.
Then those functions will return an output
that would basically get added back into the data frame
that is pushed back out to JMP into the data table.
That's done by using this lambda function,
th is lambda function applies this fu nc tion all rows,
this specified column in the data frame or the JMP table.
When we run this...
What we're going to do is take that table I just built
and we're going to extract,
but what we're going to do is reprocess the images and extract the metrics.
What we see here is the images that are showing earlier.
We've taken our picture, we've converted it to gray.
Then we started learning it.
We've got 5 pixel radius kernel 7, 10 and 15.
Then from there, we're extracting all of our different metrics.
SS IM for 7 compared to 5 SSIM.
SSIM is structural similarity index,
7 compared to 5, 10 compared to 5, 15 compared to 5.
That's basically what we're doing, is calculating structural similarity index
between this image and this image,
this image and this image
and this image and this image.
If we go back to our presentation,
from here, I used several different approaches for interrogating the data.
First was multivariate
lots correlations to work with here.
The red indicates bad images, blue indicates good images.
An outlier analysis, just looking at the Mahalanobis distances,
Jackknife distances.
That's just to see whether or not we're heavily skewed,
you can see that we've got more outliers
in the bad population compared to the good.
But I think if you separate these two out,
this comes down a little bit.
I'm showing this just to show you can go in here,
and you can start trimming some of these out if you want to,
if you're not seeing good fits.
But for the purpose of this exercise,
I didn't have to remove or hide any data in the analysis.
I was able to use all as it is.
But how I did that was here.
This is the main data table.
This is all the data that I looked at, here all the images.
I would just go to analyze, multivariate methods, run the multivariate.
What we want to do is just go with that,
actually, sorry, in the second here.
When classifying,
I did use conversion on the classification from a character base t o turn.
There's a great feature in here.
We can go to column, utility, and then make indicator columns.
You can go here and click append column name.
What it does is a little say, "Okay, that is one, zero."
For this row, this is bad, not good.
Then, down here to go vice versa.
You can use that information
to feed into some of the different modelling platforms
that we're going to look out here in a few minutes.
Back to the multi variate.
Going here, going to run all these.
Let's run this platform.
That's interesting.
Let's go here, my bad.
W e can see our correlations.
Then to get the outlier analysis,
go here, click,
sorry, go into the multi variate and you go down to outlier analysis.
You run Maha lanobis Distances, you run Jackknife Distances .
There's different approach is based on.
I've had to get successful for this.
This is first pass.
Looks like we've got something to work with here.
The next step is to break these down a little bit more
to look at how each of the different metrics is responding
to different kernel sizes.
What we're really looking for here is
how to get...
We start to see larger changes in the kernel sizes.
Are the lines still overlapping with one another?
We're starting to see separation.
Looking at 15 on bad , versus 15 are good,
15 are bad, 15 are good for structural similarity in MSE
and we can start to see that, we're seeing separation here.
I think that is a good indicator
that we should be able to extract something from this.
That was done using the graph builder demo.
I've got data table built
specifically for this
graph builder, great feature here.
I'm dumping in kernel size.
I'm looking the value the network,
these into groups based on good versus bad classification codes.
Then my image metric is going to use page.
Here I just go create a linear fit
that look at advanced for prediction , R².
It will just pop the equation is there.
You can see the goodness of the fit.
I do like looking at cheer points.
For this, when I used centered grid and move this along.
It's just easy around the eyes for presentation purposes.
You can scroll through here,
and it's got for each of these really nice looking graphs to preview.
Based on the multi variate and this linear fit,
I'm feeling pretty good about the data
that we extracted from these images.
The next step is, we've got a lot of metrics that we've extracted from these.
Do all of it matter, do some of it matter?
The next couple slides, we go over dimensional reduction.
Specifically, I used a PCA and Partial B squares for this PCA.
With this is classification, bad versus good.
This goes back to what I was demoing earlier
and the data table.
When you're building a PCA, you want to have, obviously,
your inputs and your outputs recast as vectors in that analysis.
The reason why you want that is because
you're looking for non-o rthogonality with regards to your inputs and your outputs.
The more orthagonal, they are to one another,
the less likely they're playing a role in what's going on.
Recasting
a character- based classification into a number makes that easy to work with.
We could see here, we've got some here that look like they could be interesting.
Some of these are interesting.
I highlighted these to run this analysis
and simply took my data table that I built,
go here, analyze, predictive modeling.
Sorry, multivariate, principal components.
What we're going to do is cast good, bad classification,
and all of our different metrics into the analysis.
See here, these look interesting.
These were particularly interesting.
What's nice is when you highlight these,
and you go back and you look at your data table,
it'll highlight which problems you selected.
Oftentimes PTs can get really busy,
so it's handy to be able to go references, and say,
"Okay, so these are the ones that I'm actually interested in."
Let's get back to here.
After this, in a PLS, this is more for
targeting how many factors we think are playing a role
in the data that we've got,
as well as helping us identify additional metrics
that might be interesting.
For this particular data set,
it looks like it settles in and about
probably minimum of five factors that we would expect
that comes from this prob greater than
van der Voet T² metric based on the job manual,
they referenced a academic research that indicates that
anything goes about 0.1 is typically where you want to start at.
From there, we run the PLS factor identification.
This is your VIP versus Coefficient plots,
and from there, you can highlight the different
metrics that seem to be driving this, the ones that are most relevant.
The way that works looking at these charts is you want to go out to the extremes.
These are interesting.
As it gets closer, closer to the zero coefficient,
it's less interesting.
I'm not going to run the PLS platform
because this will actually take a while to run.
I don't think anyone wants to just watch my computer run
while doing this.
Based on our dimension reduction,
I ended up with these factors being the most relevant
with regards to what it is that we want to feed into the neural network.
Some experimentation, I found that it was mainly the PLS results,
and then the standard 10 metric from the PCA as well
when that was added in,
it really helped pull the model together.
Training versus Validation.
Obviously, I'm using a neural network to build the model for the classifier.
The reason why I'm using that platform specifically for this,
it has K-Fold validation booked in on its standard in the JMP platform,
which is great.
I've found that K-F olds helps to compensate
when you're teetering on class imbalance.
That's a really common problem when you're dealing with,
studying manufacturing processes.
Y ou're not going to have a lot of the defective data to work with.
You've really got to make the best of what you've got.
I find that sometimes K-Folds helps with that.
The output from the neural network was pretty good, actually.
The way I do this is I'll run this model, like about five times,
and then I'll compare all five of them to one another.
I'll take basically the median, this classification rate,
and then go down here and look at our false positives and false negative rates.
With regards to,
the false positive rates.
I look at the ROC Curve.
This is a secret operator characteristics curves.
You want to look at these,
they want to be described as high and tight,
the closer these lines get to this middle line,
the closer your getting to the coin pass and your the prediction.
Obviously, that is not what we're seeing here.
Then the prediction profiler has some great features
because you can basically reorder these based on
which are playing the biggest role in the prediction,
as well as visualizing, like what each individual metric is doing
with regards to playing a role in the prediction itself.
In this case,
we're looking at transitions of the dominant dimensions here.
Here's a little bit weird.
I'm not sure what's going on with that.
A s you get back here, where the less dominant dimensions,
you can start to see they become more pushed.
I'll run the demo for the neural network.
We go here.
Analyze.
A ctually, I've got this one built already.
Let me...
We launch this analysis.
We've included our factors, our responses are classification.
Neural network doesn't care
if you're using a numerical or character- based response.
That will work with either one.
Here, for validation method,
you've got a K-Fold, using five folds.
The gold standard for most like AI-type analysis is ten.
I found this five was fine
so no reason to push it.
I use three hidden nodes.
Hit go and then here's the output.
Like I said, I'll run this probably
five times, take the reading value you can see here.
Our classification is pretty good.
We'll see a little bit variation on the validation set.
But it's all within reason for what it is that I'm looking for
with regards to the profiler and the ROC curve.
You get down here, which ROC curves you can see here,
as I described earlier, this height characteristics on the curves.
Validation, may be a little bit less, but we can work with it.
Then with regards to the profiler.
The profiler outputs looks like this.
It doesn't arrange them and which one is most relevant.
You go here
and you go to assess variable importance, independent uniform inputs.
Go back up here, we can see a summary report.
Here, you go down to variable importance, menu.
From here, you can reorder factors by main effect importance.
For this model, it reordered this way,
Colorize.
You can see here we've got the darker color is most relevant
and then it moves to the right.
Let's see.
Converting the model to an SPC metric, to make the data easier to interpret,
we need to take the result from the model converted into metric
that can be posted to SPC system and plotted on SPC chart.
The activation component of the neural network model is a soft map function
that's a standard in JMP.
This calculates the probability of an image being good or bad,
with the higher the two resulting and with the images classified as
and what we're going to do here is going to introduce a third metric unknown.
I'll show a little bit about how that is done.
One of the nice things about the neural network platform
is the SAS DATA Step output file is actually like a Pythonic model script.
Typically for the metric system, you want this in python,
that'll be really cool,
forgot how to do it all in JMP.
This code from the neural network here,
the actual model itself can be outputed in this format.
You just go to model and make SAS DATA S tep.
Here we go.
I think in JMP Pro, you could just output it by python directly,
but not everyone has access to JMP Pro.
It gets either 80 % of the way there,
not 100 %.
It just requires a few modifications in the actual code.
On the left hand side, we've got the SAS DATA S tep format.
On the right-hand side, it's the conversion of it to the python.
It's just importing it as importing the math library,
and then applying that so we can take the tangent to things.
Then you get that here.
What we want to do is threshold,
our S oftm ax probability calculation we're doing this,
rather than just saying, whichever of these,
is highest is your actual results.
If it's 0.5 1, bad, 0.49 , good, then it's going to be bad.
Is we go and we say, "Okay, if the Softmax output is greater than 0.75
then it's that classifier, because it'll be equal and opposite."
It'll be acceptable one.
If it's 0.75 bad then it's bad,
but if it's a number less than that, it won't classify it as good or bad.
It'll just classified as unknown.
We'll pass that into our system.
What that tells us to do is someone look at those images,
because the classifier can't decide whether or not it's good or bad.
The output, and this is the finished product,
is to convert, like I said,
the final metric into something that could be reported in our SPC system.
Basically, what we're doing here is just actually reporting
what our Softm ax probability is,
taking the inverse of that, and then applying the threshold
to determine whether or not things are known or unknown.
Based on that, this is our initial data
that I've showed at the beginning of the presentation
that shows this is the chart look like for the measurement of the defect.
Then this is what we get when we go in
and we reanalyze that data and apply this new metric,
which tells us whether or not it thinks that there's side wall feature,
you can see looking at the same data set
that yes, it very much so identifies that as either good or bad.
The green indicates that's a good.
The red indicates those were classified as bad.
Then you can see here on SPC Data,
they would instantly get flagged as bad as well.
These blue ones hanging out here
are the ones that are unknown that it is telling us so much,
"Go look at this."
That is my presentation.
I've posted the information to the user community site
if anyone wants access to the code
that's used for the image analysis that is there.
Thank you very much.
