Once more unto the breach! I was talking with my dad a few weeks back. Not to date him, but he’s been using JMP since version 1.0. I remember seeing the 3.5” discs sitting next to our Mac II-se when I was first learning to use a mouse. The same machine was on display at the 2019 Discovery Summit conference in Tucson, Arizona. Here’s a picture of it:     Anyway, my dad and I were discussing my first Five Things blog series. He mentioned that he wished someone would write a series about how to get started with scripting in JMP. At the time, I was already in the process of writing some other articles, so I tucked it away for a future project. Life, however, had other plans. A week later, I was asked to help with a customer presentation and, to my surprise, one of the topics they wanted me to cover was an introduction to scripting! The universe loves jokes, apparently. Anyway, here we go again! Following the format of my first Five Things series, here are the five(ish) things that I think everyone should know when they are starting to script. Up first, a conceptual thought. See a need, fill a need Before you start scripting, you need to consider the need. Odds are, if you want to script, there is some reason. Maybe you’re doing it to make yourself more efficient at work. Maybe you’re looking to standardize analytical methods for a group. Or maybe you are, in fact, one of those rare people that decided to pick it up for grins and giggles. None of these are bad reasons, although the last one may make things a little more difficult (I’ll explain why in a bit). The point is that you should have a reason in your head and a specific purpose for your first script. There are scripts, and then there are scripts There are, in my mind anyway, two different types of scripts: those that act on existing data, and those that create data. Scripts that work on existing data interact with a data table and existing platforms in JMP to create a new user defined output. These can be things as simple as wrapping a window around multiple reports or as complex as a custom-built report in Application Builder. The key commonality is that scripts like this are modifying existing content. The second group are scripts that create data. Scripts that create data can be simulation tools, custom graphics, tools for managing metadata in a JMP file, etc. This group is a little more nebulous, but the common theme is that the scripts extend JMP in ways beyond enriching existing data or analyses. Most of the time, these scripts are going to create new content from scratch. Scripts that modify existing content are generally where people start when they think about learning JMP scripting language (JSL). This is also where we are going to start our discussion. So, let’s take a step back and think about why you are interested in learning JSL. Give me a sentence, Vasily. One sentence only, please. Take a moment and think about your need. Which pain point are you hoping to address with JSL? Do you now see why scripting for grins and giggles (at least at first) can be tricky? You’re much more likely to fight up the learning curve if you’ve got a reason to reach the top. I’m not saying you can’t pick up JSL for the heck of it, just that the fight is harder without the extra motivation. The next step, and the final one that we’re going to cover in this episode, is what you do with that need. The simple answer is that you are going to do two things. First, you are going to write down that (possibly nebulous) idea that’s driving you to learn JSL into a single sentence – a problem statement. And, I do mean just one sentence. Make sure you are able to crystallize the idea into something tangible that you can build off of. It will also help with reining in scope creep. Here are some examples: I want to automate all the graph generation for my weekly report into one script. I want to automate the formatting work for a customer provided data table. I want to generate a set of control charts from a data file provided by my factory computer system.  Note that I’m not talking about specifics; I am simply stating the pain point that has me looking at JSL. And then I stopped writing. Make your punch list Got your problem statement? Good! Now we are going to write a series of specific deliverables for your script. It will be our punch list to help us locate the resources SAS provides for learning JSL. Do not list any graphical user interface (GUI) or user interface (UI) specifically. List the actions that the script needs to perform. Here are examples for the problem statements above: Problem Statement 1: I want to automate all the graph generation for my weekly report into one script. Punch List: Pull a data set from our production database containing the last three weeks of data. Make a graph of time vs temperature for my product. Run an analysis of time vs temperature using a polynomial model. Export the graphs and reports to a PowerPoint file. Problem Statement 2: I want to automate the formatting work for a customer provided data table. Punch List: Rename columns based on internal standards (using table from specs). Assign units to the needed columns (using table from specs). Convert Lat and Long to Geographic coordinates. Change defective units column from continuous to ordinal. Problem Statement 3: I want to generate a set of control charts from a data file provided by my factory computer system. Punch List: Import data from network location. Assign spec limits. Build control charts. Note here that I’m not boxing myself in too much. I’m just building a punch list of things I need the script to do. Since I’m assuming that you’ve never worked in JSL before, I’m also assuming that you don’t know how you’re going to attack these problems. By writing them down, you know what you need to look for in the help and resources. Which is where I will pick up in the next installment of this series. Closing thought for this week Since this series is going to span a few weeks, I decided to make it more of a work-along series than something you just read. So, take these last points I made as homework. If you want some accountability or feedback, put them in the comments below. I’ll chime in if I can help or let you know if you might be setting yourself up for some trouble down the road. For bonus points, open up a script window in JMP (File > New > Script) and put your problem statement and bullet points into the script window, one per line with a double forward slash (“//”) at the start of the line. You may see the text color change to green for those lines in the JSL Script Editor. We’ll cover what you just did in an upcoming episode. Oh, and if you’re wondering who Vasily is and why he should be giving you a singular sentence have a look here. ... View more

  In his final installment of Things I Wished I Knew When I Started Using JMP, Systems Engineer Mike Anderson reveals the secrets of JMP Masters. Mike starts the episode sharing the one common thread among the most proficient JMP users: They. Save. Their. Work! By “save their work,” Mike means they use the automatic code generation capabilities in JMP. This simple action lets users recreate their analyses in their current data set and allows them to rerun the analysis with new or updated data. Finally, Mike shares a little about special keys, in particular the broadcast key. Pressing these special keys allows you to do things like standardize graphs, close all the Outline Items (those title boxes with the gray arrows next to them), export multiple fit equations into a single column in the data table, or even turn all the summary statistics for all the subgroups in a report into a single data table. In short, you’ll be able to increase your efficiency in all kinds of unexpected ways.   ... View more

  According to JMP System Engineer Mike Anderson, “A chart or graph should have a clear purpose. If it doesn’t meet that purpose, it’s not a good chart.” To illustrate this example, Mike shows us a spider diagram of five sensory attributes for 14 types of beer to see if we can answer some simple questions. Ultimately, it’s pretty obvious it’s not easy to get that information out of the spider plot. More importantly, Mike asks: “How hard did you have to think?” A useful visualization, Mike says, must meet five criteria: It’s comprehensive, truthful, “not ugly,” obvious and effortless. So, can his example be presented in a better way? Mike shows us a number of possible candidates so you can judge for yourself.   ... View more

  You’ve probably used JMP’s Distribution Platform, but chances are you’re only scratching the surface of its usefulness. In this Episode of Things I Wish I Knew When I Started Using JMP, Mike Anderson gives us a tour of all the capabilities of this powerful platform, including how to customize the information you view, how to turn on various capabilities to perform various statistical tests on your data and get even more out of The Distribution Menu’s visualizations and more. Mike concludes his session by answering several questions for the viewing audience.   ... View more

  According to JMP Systems Engineer it’s the first thing you should do when handed a new dataset – graph your data. In this episode of Things I Wish I Knew When I Started Using JMP, System Engineer Mike Anderson shows you where to find and how to use the Analyze Menu in JMP. Learn how the Analyze Menu can help you understand the basics of your data and variables (Distribution and Fit Y by X); discover more complex questions including regression (Fit Model) and Machine Learning and Artificial Intelligence Methods (Predictive Modeling, Specialized Modeling, Screening, Multivariate Methods and Clustering); and better understand process, quality and people (Consumer Research).   Full Transcript (Automatically Generated) Morning, everyone. It's a beautiful Monday here in New York, the sun shining finally. And today we're going to be going over another of the things that I wish I had known when I started learning about  jmp. Also something that I like to tell a lot of people when they're first getting started using  jmp, I've been working with a lot of people that have been taking the demo downloads that we have lately. And this is one of the things I tell them on the first day. Because this can be a little bit confusing coming from different software's because the way  jmp considers the Analyze menu is organized entirely differently than the way you would expect coming over from somebody else's software.   As always, I'm going to be taking your questions. So if you've got any questions, find that q&a window and at the end of the session, I'll try and answer as many of them as I can. I got one cut that came in over the weekend from actually from the first episode of this series, where they wanted to know how I did something and I'll be going through that as well, too. To show them how we did that, and we're going to dust off some fun and go into image analysis to do it.   Alright, so first thing, the Analyze menu. When I first started learning the Analyze menu, it really didn't make sense to me until and it actually didn't make sense to me until I started teaching other people to use the software. And then as I started kind of refining my idea of what the the analytical workflow is, a lot of the ideas that we talked about in this series came from those early experiments and how to teach  jmp on my own. I started to understand how the Analyze menu is laid out and that there's a method to this apparent madness.   And it's there's a, what I want to share with you today is a general guideline about how to kind of guess where those things are that you're looking for. Kind of moving your guessing around in the menu from just wild guess to scientific, wild guess And we'll we'll leave the the middle of the layout for the sake of the recording. So let's get started. Let's look at let's, what I've got right here is a screen capture of the Analyze menu right here. So I can use my little laser pointer and point around. And let's take a second let's step back. When you're handed a brand new data set. What's the first thing you should do? Well, if you were here, for the first episode of this series, you would know the first thing you should do is graph your data. Well, if we make zero assumptions about our data, if we make zero assumptions about shape, about relationship about anything, the first type of graph that you probably would run into is something called a histogram.   And it makes no assumptions. It's great for understanding how data is shaped looking for outliers doing basic statistical tests. There's lots of things you can do with just a histogram. If we come under the Analyze menu, the first thing we run into is this platform called the distribution. And that graph that goes with that distribution. Remember from the second thing, we talked about the the  jmp workflow, get a graph, look at the graph, ask questions about what you see and then push the answer button. The graph that comes with the annal with distribution platform, is a histogram. So that's if we want to just look at one thing now if you look at your data, and then you moved a little bit further down the kind of the, the chain of logic that you might be following or that the inquiry that you might be looking at, you might start proposing relationships.   I think a relates to be I think there's a correlation between A and B. Those are all possible things. And those by various relationships, we're starting to make inferences about whether or not something is a response or a factor whether or not it's an X or Y. In that case, we would come down to fit y by x. And there because of some, some clever coding, a lot of different plot, a lot of different comparisons are available all of them comparing one variable to another, most of the time making an assumption that one of them is response, and one of them is a factor.   So it would seem at first glance, at first glance, that the Analyze menu is organized by the number of variables you're trying to compare. But then we get to tabulate and then we get to text Explorer, and that that logic kind of breaks down. So let's let's skip those two for just a minute. And let's move down to fit model and what we're doing Here as we're proposing more complex relationships, it's multivariate Yes, but we're composing, where we're proposing different levels of complexity in our models. And that's the trick. That's the key. The  jmp analyze menu, at least, from what I've seen, and the way that I, the way that I teach it is that it's organized as you go down the menu. You're asking more and more complex more and more specialized problems. Which is why down here at the bottom, we have people consumer research, because let's be honest, people are generally kind of nutty, in very difficult to predict things.   And because of that the data associated with people is invariably very complex and very hard to understand. So we've got some very specialized tools in that platform down there in that subgroup down there at the bottom, to help us navigate those complexities that come with the human condition. And so let's look at look at this a little bit down in the middle here. Let me bring up a little bit of an antic annotated version of this. So we talked about one variable, two variables, fit models regression. In here we have machine learning and AI type models. We also have multivariate platform, screening is kind of the odd man out, except it under the hood, a few of the screening platforms use machine learning technology so that that makes some sense as to why they're there. Then we get into prop quality and process. So that's a more specialized tool. And like I said, people down at the bottom, because it's incredibly complex data.   Now, let's scoot back up to this transfer to the tabulate and text Explorer. And this is a problem in the scheme, because those don't necessarily make sense for them to be right there. Unless you understand that tabulate one of the roles that tabulate has, is to transform data from one form to another. It's good for doing cross tabs, it's good for doing well not so much cross tabs, pivot tables and things like that. So its goal is to do summary to reshape data so that other parts of the software can take the inputs to transformative tool, text explores the same way. A little cue about the role of a platform in  jmp is when we put explorer at the back of something. One of the goals of the tool, one of the goals of the platform is to transform data from something in this case text which is really tricky to work with, to something that we can use for fit model that we can use for doing sentiment analysis that we can use for doing all these different things. So those two bridging tools, those two those To platforms there are bridging their goal is to transform from repair to fairly simply, fairly simple, simplistic type one, not simplistic, but fairly simple type analyses to more complex multivariate analyses that come along.   So let's take a quick let's take a second and do a pop quiz here. I'll leave that up delay to give everybody a cue, and you can follow along. Well, I guess I'm not gonna leave that. Let's start with a question. If you wanted to compare one variable to a target, where would you look in the Analyze menu? I'll give you a second to think about that. got an answer. Okay. one variable, that's the key. So we're going to look on where there is one variable, analyze distribution. What would we do right there? Well, let's just take some data. And let's take some data here that I've got this is just processed data that doesn't really doesn't matter. For the sake of the discussion, we'll just grab something. Click ok. We're in the distribution platform now awesome.   And if I follow my workflow, I'm gonna look at my data I'm gonna say I, you know, as normal enough horseshoes and hand grenades. We could, we could look at that and more in depth if we wanted to. But we come under here for this option that says test mean and I, I'm going to propose that my hypothesized mean is 30. And when I do that, I get down here at the bottom, my T test for the me. Okay. So that's part that's step one. Let's try the net. Let's try one more. Now, what if we wanted to get correlation coefficients? So we're comparing two variables. But we want to do it across all the continuous variables in a data set. Where would you go for that? Give me a second. Think about it again. Okay. If you said by variant, you could do that.   You could do that in by various. It's possible. For come under by under or excuse me, if you if you say fit y by x are saying I'm going to compare two variables. Yes, you're comparing two variables, but you're comparing two variables and then two variables, and then two variables, and then two variables, and then you've got all these common tutorials. combinations of two variables. So while you can do this, you know, let's just grab some, let's grab a bunch of continuous variables here. Again, we're not really worried about what's going on here. Let's just due to, you could come into by very, very easily, and come under the red triangle, and then come down to the density ellipse. And that will give you the correlation. I could do it there. That's true. But then you'd have to do it for F pairwise for every combination in this data set. If you've got 150 different combinations of 150 different continuous variables, you're going to be at that all day.   That's not what we want to do. So we come a little deeper, our problem is more complex. We're coming down further in the in the model, do we need to transform our data? Now we don't do we need a model. That doesn't sound quite right. modeling, modeling, screening, possibly, but multivariate methods, multivariate might be the right place to look for this. And again, there there may be if you're just starting out with  jmp, there may be some, some hunting and pecking some a little bit of guessing around. But the purpose of this little trick is to kind of give you an idea about where you need to guess. So if I dump all my continuous variables in here, Wallah, I get all of my correlations. And I can even if I want to be fancy, I can put them on this scatterplot matrix in here. Just like that, and they're colored so it's easier to look at what's what's Important. Okay. All right. And that's that's the trick about the that's the trick for today. That's the thing we want to cover for today. And if you're interested, this particular graphic, it's up on the blog article that comes with this. It's look at the blog article for this, for this content today is linked in the  jmp on the air segment for this.   So you can get to this article, you can get to that little graphic if you need some help at any time. All right. Now I want to come to and while I'm doing this question, if you have any questions, I've got the q&a open here. Go ahead and type them in the q&a. Instead of taking a two minute break. I'm going to cover a question that I got earlier in the week from someone that wanted to know how I did something in the first episode. But in the meantime, if you've got questions, feel free to type them in to that to the q&a and I'll keep an eye on that while I'm going. All right. The question I got was how Did I make this 3d graphic? And it wasn't the specifics about the 3d graphic itself? That's just scatterplot. 3d, that's easy to do. The question was, how did I get the data? out of my images? Which is a very interesting question. And the short answer is that on Windows, this is really easy to do. So let me pull over my windows enter in my password here before I pull it over.   Now what the problem is, so here's a here's  jmp on Windows. On Mac, the workflows a little bit harder because of a there's some differences in in the underlying code. But the the trick the question here is how do I get Get if I have a series of when I'm opening if I have a series of images or TIFF images and I want to bring those into  jmp, let me just open up all my show all my files here. And in this case I do I have my images set in here and what's called a multi file TIFF. So in this TIFF just like in a GIF, you know what, let's leave that let's leave the argument about GIF or GIF off the side for the moment. In this single file, there are multiple images, it's a movie more or less. This is really handy for for taking thicknesses of so if you're doing X ray tomography, you can set up the image stack in here and it will play through in certain graphic softwares or it's a great way just to keep everything together when you're doing image analysis. But the question is how do we get this data in? Well, in either case, john Ponty a few years back made a great tool called the image analyzer. And this on the Mac, or on the PC side, sorry, this, this makes this job really easy to do.   So I'm going to navigate, I'm going to go choose I'm going to grab my TIFF file here. And it's going to read all the bits attached to this image, spit it out as a data frame with 2 million rows, but also create some images so you can kind of see what's going on inside of this file. So on Windows easiest pi, go grab the image analyzer, it's super easy to do. on the Mac, you would have to do this the image analyzer add in is available but on the Mac It only pulls in the first image, it won't pull in a multi image stack. So you'll have to bring these in one at a time using that image analyzer and then concatenate them. Or if you're into the if you're in the process of doing image analysis, you probably know about a tool called image J. And this is what this is the way I did it. And what you can do is there's a macro out there, I can come in and open my data. Let's go open recent.   There's my image subset, same data set, I can come into my macros. And I have a macro right here that does the same thing runs through all the images, extracts all the bits and puts them into a CSV file that I can then bring into  jmp really easily and I'll post That batch, macro as well on on the community later on this afternoon. So that's how I built those things. Now I'm not seeing any questions here. So we're going to go ahead and we will, we'll wrap up. One thing I do want to say is, again, the point of the primary point of today's content is that's how you analyze or navigate the Analyze menu. Next week, we're going to spend a little bit more time on that first part of the Analyze menu on the distribution platform and go over some of the things you may not have known are capable of doing you're capable of doing distribution platform.   ... View more

The Engineering Mailbag Episode 4: Pythons and Pandas, Oh My! Every now and again we systems engineers run into interesting questions that would fall somewhat outside the typical range of JMP usage. These applications are generally clever, and often bring home how using data isn’t limited to solving business or technical problems. Other times, the questions are just unexpected, challenging problems. These “curve balls” (as I like to call them) come in many different forms: coding problems, interesting analyses, ways of visualizing data… you get the idea. I haven’t written from the mailbag in a while, but this particular question really seemed to be something that needed to be addressed. The tool I used to answer the question really isn’t discussed as much as I think it should be. So, this is an opportunity for me to help out a colleague from New York and show everyone something interesting that you might not be aware that JMP can do. The Question Mike[JMP], Often, I need to do groupbys where I need to grab say the max value everything grouped by some label where the Python Pandas equivalent would be something like: df.groupby(['Something', 'Something Else']).max() Another might be something like pandas.nlargest with a groupby, where I would want say the top 5 things out of a groupby. And after a couple minutes of fiddling with a column formula I give up and export the data to reimport it into JMP later. Tabulate sort of does this, but not always in the way I want. So a "How to JMP like Pandas" document would be super helpful for me, and might be helpful for JMP with how widespread Pandas has become… Thanks, Nick My Response Hi, Nick! There is a trick that you can use to get more flexibility out of formulas that mimics a lot of what [Pandas] can do. It involves telling JMP what columns you want to group by. You do this by right clicking on the column name and selecting New Formula Column > Group By.  After doing that, any formula columns you create by right clicking on the column name and using New Formula Column> … will be grouped by the grouping columns you set.  After that, it’s just a matter of filtering or subsetting using data filters or local data filters. Let me know if that helps! Best, M Pandas: Cute Mustelid or Data Analysis Tool All right, for the uninitiated, Pandas is not a cuddly, chromatically challenged mustelid. Pandas (usually abbreviated as “pd” in python scripts) is a “…Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series [source]." It basically fills the role that the data organization and manipulation parts of JMP (data tables, Row and Column menus, etc.) do in the Python programming environment. So, it’s pretty important to Python data analytics workflows. Now, if you are looking to run Python from within JMP, this ain’t the blog article for you. Go have a look at this white paper. If you are looking to break yourself free from doing data transformation in a command line, read on. DataFrames and Data Tables Still here? Awesome. Okay, the rest of this article is going to go through some of the common bits of Pandas and their analog in JMP. It’s not a comprehensive list by any means; it’s intended to point your nose in the right direction when thinking about the two environments. The first bit I want to cover is the “DataFrame.” In Pandas, this is where everything is stored. Like data tables in JMP, it is essentially a collection of data columns with a name (column name), etc. You act on columns and rows (transform things, summarize, etc.) but you’re always acting on either the DataFrame as a whole or some part of it. In JMP, the primary data container is the data table. You can act on columns and rows in the same ways you would in a Pandas DataFrame. In both, you are able to deal with missing values in the data table. In JMP, they are designated as either a cell with a “.” in it (for numerical data) or a blank cell (for character data). You can transform data, summarize it, etc. (more on those bits later). In both cases, the more you tell the software about the data the less leg work you have to do, which brings me to my next point. Column Properties and dtypes In the matter of column properties and data types (dtypes in Pandas), JMP wins hands down. Pandas inherits most of its data type understanding from another Python package called NumPy, which is cool. NumPy is pretty sturdy code, but it’s really limited in this regard. It handles about eight types of data (see the table below[source]) with somewhat limited support for different formatting (it gets pretty confused by currencies without help, for instance). Kind of Data Data Type Scalar Array String Aliases Documentation tz-aware datetime DatetimeTZDtype Timestamp arrays.DatetimeArray 'datetime64[ns, <tz>]' Time zone handling Categorical CategoricalDtype (none) Categorical 'category' Categorical data period (time spans) PeriodDtype Period arrays.PeriodArray 'period[<freq>]', 'Period[<freq>]' Time span representation sparse SparseDtype (none) arrays.SparseArray 'Sparse', 'Sparse[int]', 'Sparse[float]' Sparse data structures intervals IntervalDtype Interval arrays.IntervalArray 'interval', 'Interval', 'Interval[<numpy_dtype>]', 'Interval[datetime64[ns, <tz>]]', 'Interval[timedelta64[<freq>]]' IntervalIndex nullable integer Int64Dtype, … (none) arrays.IntegerArray 'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64' Nullable integer data type   Strings StringDtype str arrays.StringArray 'string' Working with text data Boolean (with NA) BooleanDtype bool arrays.BooleanArray 'boolean' Boolean data with missing values The column properties in JMP comprise about 43 pages of the Using JMP book. They are significantly more extensive and are good at recognizing data types as they are imported. Of particular interest to a new JMP user is the fact that column properties extend well beyond the data types that Pandas uses. It can handle a host of metadata that trigger actions across the software. Now, to be fair, you can attach attributes to a Pandas DataFrame, but since the DataFrame is just a container, you have to either code in the support for the attributes or hope someone put in the support in the package you are using. All that is already done for you in JMP. Tabulate, Column Formulas and GroupBys The truth is that the previous two items don’t really matter that much in the context of the question I was trying to answer. For our purposes, the data table/ DataFrame is just a box to put stuff in. The column properties/types can help with analytical considerations, but, again, that’s not important for the question we’re working with here. The bit of Pandas that we are interested in is called GroupBy. And it’s for slicing and dicing data in preparation for doing other analyses. A quick example using the JMP Big Class data table and getting the average height and weight of boys and girls by age might look like this:         # Import Pandas import pandas as pd # Bring in the data data = pd.read_csv("/Users/mike.anderson@jmp.com/Desktop/Big Class.csv") # Put the data into a DataFrame df = pd.DataFrame(data) # Run the group by, get the means, and show the results summ_df= df.groupby(['sex','age']).mean() print(summ_df)         With the output looking something like this:         height weight sex age F 12 58.600000 100.200000 13 59.000000 95.333333 14 62.600000 96.600000 15 63.000000 102.000000 16 62.500000 113.500000 17 62.000000 116.000000 M 12 57.333333 97.000000 13 61.250000 94.250000 14 65.285714 103.857143 15 65.200000 110.800000 16 68.000000 128.000000 17 69.000000 153.000000         Normally, JMP users would look to Summary or Tabulate as their primary tools for these data transformation operations. They might even use the Formula Editor to create a column. But, there is a secret weapon in the data table (and in the platform column list) that you can use to do just about everything you’d want to do in Pandas and more. That secret is the ability to right click on a column name and select New Formula Column. This works anywhere that there is a column list! From there you have many different options for doing various transforms, such as combining columns, doing cumulative sums in rows, etc.   That, in and of itself, while handy, is not particularly magical. The place where things get really interesting is when you turn on the “Group By” flag for a column in JMP. This will automatically make any formula columns you create aware of the levels in a grouping variable. Just think about that for a second -- you can create a formula column anywhere in the software that can be stratified based on a grouping column. Let’s look at an example of what this means from another question I received recently: An Example: TWO QUESTIONS FOR THE PRICE OF ONE!! About the same time the initial question came through, I got a question from another customer about how to do a numerical integral (area under a curve or AUC) for a large collection of samples. Moreover, they were willing to provide me with a sample data set they had published previously for a journal article in Environmental and Molecular Mutagenesis. It turns out that what they wanted to do was easily accomplished using Group By Flags and Function Columns. Here’s the procedure (you can follow along using the data from the JMP Public site 😞 First, let’s graph the data. The details of the study aren’t important for this discussion, though it is interesting reading. The short version is that it’s a large collection of chemicals with different concentrations (Conc. (uM)) and a series of genetic responses. For our exercise, they’re interested in calculating the AUC for each of the genetic responses. Here’s a look in Graph Builder with a column switcher: The first step in the process of calculating the AUC is setting up the grouping. We’re going to right click on the Chemical column, select “New Formula Column,” and then “Group By.” We’re going to do this again for the Biomarker column. Note that order can matter here. We’re grouping first by Chemical and then Biomarker within Chemical. After that, it’s just a matter of using the “New Formula Column” in the data table to build up the calculation. Going forward, I broke each step out into individual columns for the sake of clarity. And, if you’re really interested, I’m using the Trapezoidal Rule Riemann Sum for the math here. Ok, back to the calculation. The first thing we need is the distance between each point in the x-axis (the point-wise change in x). We’re going to select the Conc (um) column, right click on the title and select New Formula Column>Row>Difference. Now open the formula in the new column that JMP created. It should show something like this: Note that JMP is handling calculating between rows and has adjusted the formula to reset when it detects a new Chemical or a new Biomarker type. Next we need the average value of Value between each data point and the previous one. Right click on the column header and select New Formula Column>Row>Lag. Open the formula and modify it a little so that it’s calculating the difference between rows and not just giving the value of the previous row. That’s shown in this screenshot: Those two columns (I renamed them dx and dy in the data table) need to be multiplied together to get the area of a segment. Select the two columns, and right click on the column name of one of them. Select New Formula Column>Combine>Product. The new column gives us the approximate area under the portion of the curve between each data point and the next one. I renamed that column iAUC(for instantaneous AUC) in the data table. The last step is to sum up the values for each section. This is done by right clicking on the iAUC column and selecting New Formula Column>Row>Cumulative Sum. Because we selected the Group By columns at the beginning of the exercise, the tally is reset for each sample in the data set. From here we could use Tabulate or Summary to create a table with the max values for each sample (the approximate total area under the curve). Here’s an example for a common chemical: Final Thoughts And, that’s it. You can easily replicate behaviors in Pandas using structures that exist in JMP. Just remember to use the Group By variable and you’re off. No mustelids, chromatically challenged or otherwise, required. Editor's note: Have you read the other installments in our Engineering Mailbag series ? See how Mike helps a user who was using JMP  to decide whether to sign up for a snow removal service and develop unique visualizations for STEAM education programs. ... View more

  In this episode of  Things  I Wish I Knew When I Started Using JMP ,  Systems Engineer  Mike Anderson   introduces the 4-step JMP Workflow,  an easy way  to create and understand  JMP reports.  The process is simple, but  powerful .   Step 1 – Make a  report.  Step 2 – Look at the graph; are you seeing what you want to shar e  within  five  second s ?   Step 3 – Ask questions about what you see.  Step 4 – Push the  “ Answer  B utto n .”   Using data from  Futurama , one of his favorite shows,  Mike d emonstrate s  how easy, and effective, the 4-step JMP workflow can be. ... View more

  Learn some of the ways you can work with your text data in JMP and JMP Pro.   Full Transcript (Automatically Generated) Morning, or good afternoon, everyone. My name is Mike Anderson. I'm a systems engineer for New York. And today I'm going to be showing you a few of the things that I love about text Explorer. I got really passionate about this particular tool back when it was when it was first introduced. And it's it's been kind of a hobby for me to kind of get go around with it and play with the play with text explore a little bit.   The for the sake of argument here, I'm going to use some data that I've never used before. I was working on this over the weekend trying to pull this data set together. And this is being stuck at home like everybody else. I've been going through some of my old favorite TV shows one of those back in grad school was Futurama. So I've pulled all of the the plots and plot summaries from Wikipedia for all of the Futurama episodes, and that's what we're going to be analyzing today. Now before we go into this in too much detail, let me give you a little bit of an idea about what text explores for and it all lines up in that last name in jmp, there are two platforms that are called explorers. There is text exploring, there's functional Data Explorer. These platforms, while they can do an analysis among themselves, they're one of their primary features is to transform data from one usually tricky format to analyze into something that's easier to work with in jmp.   So that said, Well, we can do some very interesting things in the text Explorer. One of the things that I love about it is its ability to transform text data into something that I can do sentiment analysis with. And I'll there's some resources on that I'm not going to go down that road. That's that's a whole nother rabbit hole to go down. But one of the things that make that I love about text Explorer is that facility to take text which is challenging to analyze in a in a statistical package to something that we can analyze that we can model We can crunch in a little bit more in a more comfortable setting. So let's go ahead and let's get started with this. And I'm going to show you first the how to get the platform set up. So we'll go into analyze text Explorer.   And it's as simple as dropping in the free text that you've got in your data set. So text explorer goes into we go into there first, we can we have a bunch of options that we can use to filter down the data, we can get rid of really, really short words, we can work with phrases and things like that as well. There's a great tutorial by Nick Shelton on how to do all of the basics that's out on the web. So I'm gonna skip over that just for a second and get to kind of the highlight reel. of the things I love about text explorer in general. The first thing that I love is the word cloud. Okay, this is when people talk about text analysis. This is the first thing that pops to mind. It can be an potful it can be incredibly powerful. It's a great tool for visualizing this data and it makes it kind of easy for people to consume. In this case. The first thing we see out of the gate we see the main characters we see Philip j fry we see Ben Bender, we see Leila, we see these main characters that are in almost every episode, so it's not surprising that they are going to that they're going to show up a lot in the plot summaries. Now removing that information is really easy. I can right click on something and say add a stop word and that excludes that word from analysis. If I do that with the just the by removing just these three, just fry bender and Lila   I get up to something where I can start picking out different themes that are present within the dataset. Now, I've colored this based on the overall Internet Movie Database rating for The Internet Movie Database rating for the for the individual episodes. And you can see that in some cases, things that involve Farnsworth or involve involve robots, you can see that those tend to be a little bit lower on the rating scales than things that involve the professor's or for instance, time travel or something like that some of the more kind of out there sci fi type topics that are associated with this with this show. And that brings me to the next thing that I love about this, that I love about this platform. And this is a pro capability. But what I love is the ability to come into after I've gotten rid of the words that don't necessarily matter. Once I've got into got the dataset curated, I can come in and say do something called Latent Semantic Analysis. And that's a lot of words to say, we're going to analyze all of this, all of these terms that we've that we've looked at, and turn them into things that maybe we can look at it from trends are phrases and words that tend to show up together. And that's what we've got down here.   Now in this case we don't see a lot, that's good, you dramas a show was kind of all over the place, there was no real formulaic element to it in terms of the different plot elements that it had into it. So expecting to see a lot of interesting things in here wasn't something I was looking for. What I was looking for curious about was how the topic analysis, which is one of the options for under SVD would be able to handle this kind of data. And what it does in this case is it starts picking out individual episodes, not by picking them out based on their on on the row of information but just based on the words and phrases that tend to show up together. It starts reconstructing the synopsis for different for different people. So it's, and that's kind of a good idea for what for what topics are in this case, they're not necessarily topics, they're words and phrases that show to show up together very frequently.   And you can use that with your own personal Insight with your subject matter expertise to read into that and start figuring out things that are that are important within this dataset. And let me show you one where we've got some common themes that tend to show up a lot within a data set. This is the same type of analysis. This is the this is a actually one of the sample data sets in jmp, and it is looking at survey of pets.   And what we can see here is we this, this top plot now actually has some structure to it has some things that are interesting to it. And the way I explain this to people is each one of these kind of tendrils that goes off along along each side. You can see there's three of them here. And you can see three One, two, maybe three, maybe four over here for different three or four different topics, or three or four different common sets of documents that show up within the data set. And this helps you when you're looking at data where you're looking for commonality when you're looking for common themes that show up within a data set like survey analysis, or when you're analyzing reviews from products or when you're looking at, you're looking at your social media. Those kind of common threads and themes can provide insight for you and we can come in and just grab them really quickly. Here I'm using the lasso tool. And I can pull up the text and I can look in here and I can see what's going on. Aside from the strange. The cat, the itinerant cat entry here, there's a couple of them. This thread is talking about guard dogs. And we can get that just from the the different terms that tend to show up together within the data set. And that's that's the crux of the things that I like about the crux of the things that I like about this, this platform, the ability to visualize a very challenging data set the word cloud, the ability to take in, dig into that data set for its for different themes or trends you may not know are there. And the third is its ability, as I said, first to transform data from one form to another to let us do things like sentiment analysis and things like that. The way we do that, in this case is we come up to the red triangle, and we come down to where it says save document term matrix. And what that will do is it will take all of the terms that we've taken in our data set and create columns of it within the original data set for each of those terms, and those become the factors that we use when we're building a model. For for instance, sentiment analysis. And this is a bit of a high This is a bit of a high level over View there are some great tools for working with sediment and working with with text explorer out there on the web. There's two mastering jmp seminars already recorded. You can go and look at those today. Those are going to be those links are going to be in the community page for this episode of jmp on air. Julian, I'll take it back to you now. ... View more

    This add-in combines a number of visualizations related to consumer analytics, including Spider Plots, a multivariate plot on normalized values (riffing on @danschikore's RadViz add-in), a parallel plot, and pairwise correlation analysis.  The add-in installs as "Parker Analysis" (e.g. Peter Parker) under the add-in menu.   The setup is straight forward - just drop the variables of interest into the variables field and the sample/person/product ID in the Label field.  A summarized version of the Candy Bar Nutrition sample data set is included to demonstrate the table set up.     Since collected data can sometimes not extend the full width of an available scale (i.e. possible values ranged between 1 and 7, but observed values only ranging between 2 and 4) the platform has an option to use the Axis column property.  Select the checkbox in the lower left of the launcher and the tool will scan the variables for Axis settings and scale those parts of the Spider plot accordingly.       Comments, complaints, and thoughts for improvement are always welcome!   Updates: 19May2020: Switched from adhoc parallel plot to using Graph Builder's Parallel Plot and added a bar chart with a column switcher.     ... View more

It’s settling into the deep part of winter in Saratoga Springs, New York.  This is the time of year I sit with a hot cocoa, look out at the snow, and think. The new year is a contemplative time for many cultures – a time to assess our shortcomings, to make resolutions to improve or change. Saying that I’m introspective this time of year is putting it mildly. I don’t know what it is about winter, but it puts me into a deeply contemplative state of mind. I get a lot of ideas during this time of year for projects and writing. This little musing has been crystallizing all weekend – a dusty older voice in my head, like Jean Shepherd reminiscing about his time on Cleveland Street, except mine is talking about data, analytics, and Mongolian Hot Pot... Mongolian Hot Pot During the first week in January, I was in a planning meeting with the other members of the NY JMP Team in NYC. We decided to do a little team building activity around a trip to a Mongolian Hot Pot restaurant in China Town. Here’s are a picture: Your New York JMP Account Team: (from right to left) Brendan Leary, Tracey Bailey, & Mike Anderson   Now, before you think this is a travel log, hold up a sec. I do have a point. First, the restaurant is excellent – you should go if you’re in the area. Just Google it. Second, during the meal, we were discussing how they provided instructions for how to “do” Mongolian Hot Pot – the order in which different things were added to the soup to cook and how long they should stay in the broth. The heavenly elixir that is Mongolian Hot Pot. The stuff on the right is super spicy, the stuff on the left, not so much. It got me thinking about how many of my favorite food experiences all had a ritual component. Hot Pot, Ethiopian, Sushi, Thanksgiving Dinner, even making pizza for my family on Friday nights – we have a habit of creating rituals around consuming food. They give the experience added meaning and significance. Food rituals can bring people together and cement relationships. And, in my current contemplative state of mind, it got my wheels turning. Analytical Rituals Now, what does this have to do with JMP or analytics in general? Well, I was thinking later that night about how rituals crop up in other parts of the human experience – and I’m not necessarily talking about religious observance here. Over the years, I’ve seen many cases where analytics have become ritualized. And it turns out that this can be a big problem. Rituals all have a common attribute. Their goal is to teach or remind through actions. They also generally have some kind of framework (cultural, societal, etc.) to reinforce the meaning of the rituals. Analytical rituals start similarly – with an attempt to codify a best practice or something that has worked well for someone so that it won’t be forgotten. These analytical rituals have many names – Procedures, SOP’s, Macros, Templates, Success Stories, etc. And, truthfully, that’s fine, and probably a good thing. The ritualizing of analytical practice does what any ritual does. It adds significance to the practice. By ritualizing analytics, we elevate its importance in an organization – particularly when leadership adopt or endorse the rituals. As commonly is the case, analytical rituals generally become part of an organization's culture – become embedded in its DNA. And that’s where the problems start – when was the last time you had a good think about your DNA? The Problem with Analytical Rituals The problem that comes with analytical rituals is that they, like all rituals, can move over time from conscious observance to rote observance, that is, just going through the motions. And when that happens, they lose their meaning. This is particularly easy for analytical rituals because they don’t necessarily have the cultural or societal support structure to bind the meaning to the actions. Analytics, on the organizational scales we're seeing today, are fairly new and don't always come with the inherent tribal knowledge or caution that come with other sciences. And that's a problem. For analytical rituals, the loss of meaning is particularly dangerous. Since the meaning contains the reason for the analysis, the assumptions that are being made, and the appropriate use cases – the Why of the ritual – the loss of meaning can lead to misapplication of the analysis. Ultimately, analytical rituals stripped of their meaning are dead and dangerous. Over my career, I’ve seen many cases where it’s apparent that a valuable analytical ritual has lost its meaning and devotees are just going through the motions – and maybe not the correct motions at that. Let me give you some examples. The War on the P-Value A little while ago, I was reading an article about how some scientific journals were starting to discourage reporting of p-values or of the p < 0.05 criteria for significance. The argument was that people were blindly using the criteria, or in some cases tweaking results so that they could report a favorable result vis-à-vis the p-value. The actual situation is a bit more complex. At some point in the past, researchers were educated in statistics and began reporting p-values as a convenient shorthand of the significance of findings or strength of trends in their data. This practice was incorporated into an analytical ritual by the journals with a guideline of p < 0.05 being a benchmark for significance. This is a good thing – the p-value is a fine metric for reporting values, provided the context is maintained along with the reasons for choosing particular p-value criteria. Further, a clip level of p < 0.05 for hypothesis testing is widely used and is generally good enough for many applications – provided the practitioner understands that it’s a guideline that has to be interpreted critically, and not by rote. The problem here came when the meaning of the ritual, along with its context, was lost. Hollow rituals lead down bad roads – as has been seen in disciplines where these p-value guidelines have turned into p-value rules. The p-value rules have in turn promoted or enabled undesirable behavior. The Case of the Change Point Worksheet Speaking of bad roads, let's talk about the venerable (or infamous, if you like) analysis template. Many organizations, in an effort to simplify the work of their technical staff or compensate for a gap in institutional knowledge, develop analytical templates in spreadsheets or standard analyses in online analytical hubs. One might argue that this codification of analyses is a good thing, and I would agree, to a point. The organization has put resources towards making it easier to follow a sound analytical practice. As was the issue with the p-values, the problem comes when the meaning of the analytical ritual gets lost. One example in particular sticks out in my mind. I saw an organization that validated changes to its process based on a spreadsheet template. The template had a colored box and a box and whisker plot. If the box was green, the change was considered acceptable. When I saw this template, I asked the company when it was acceptable to use the template and when it wasn’t. They couldn’t give me an answer other than it was the tool used to validate all process changes. The ritual had lost its meaning. Digging deeper (and reverse-engineering the template), I was able to determine that the analysis was a simple ANOVA – which comes with a certain set of assumptions and conditions under which it can be used. Someone in the past had developed the template to standardize one specific type of analysis and, because of the loss of the analytical ritual’s meaning, it was being wildly misused. The Problem Solving Methodology I’m going to tread lightly here, but this next area of analytical rituals might tweak some noses. There are many analytical rituals that come packaged in “Problem Solving Methodologies” or “Continuous Improvement Methods.” I’m not going to name names – you can probably think of two or three on your own depending on your industry. Let me be clear: There is nothing wrong with these methodologies or the analytical rituals that they teach. They are good starting points to help people along their analytical journeys. The problem comes when the adherents to these methodologies do not have the underlying meaning for the analytical rituals, which is something that I see a bit more often than I would like. This may be caused by many things – because the adherent forgot, because they weren’t properly instructed, or even because the instructor was hobbled by organizational constraints. In any case, the result is the same: systems of analytical rituals devoid of the underlying meaning are recipes for disaster. They lead to inappropriate application of tools or misinterpretation of the results. These, in turn, can lead to mistrust in statistical analysis in general, which is a hard thing to fix. The Solution So, what is one to do about hollow analytical rituals? In theory, the fix is easy: Instill the meaning back into the ritual; remind ourselves of why we are doing a particular analysis. In practice, that might be a tall order. People questioning any ritual are met with resistance. For some reason, analytical rituals tend to hit a nerve for some and are particularly difficult to address. I don’t know why, though I have my theories. The truth is that without constant reinforcement of the concepts, any analytical ritual can eventually lose its meaning. Like with Shelley’s Ozymandias, time will always win if we are not vigilant. And, that brings me to the point for this little stream of thought: The only true way to fix a hollow analytical ritual is by fixing it for ourselves. A New Year’s Resolution In many cultures, the new year is a time for resolutions. It’s a time to look at our weaknesses and resolve to improve – not to be perfect, just better than we were last year. So, here is my proposal – a challenge or call to arms, if you like: Examine your analytical rituals. Find one that you think you might be a little weak on, and get better at understanding it. There are some excellent free resources available from JMP to help. The first is a website we provide that has information on common statistical topics. It's called the Statistics Knowledge Portal, and it's updated fairly regularly with new content. The other is a complete course of study in statistics provided by JMP and SAS Education called Statistical Thinking for Industrial Problem Solving (if that's a bit of a mouthful for you, we just call it STIPS). There's also the message board on the JMP User Community. Last, if you need help getting started, reach out. I don’t know everything (ask my wife), but I know a lot of people who are a lot smarter than I am and they love to help. We'll get you pointed in the right direction. Thanks for sticking through this little article (vanity piece, maybe?). I honestly wrote it down just so that dusty older voice would go away long enough for me to work on some other writing (one of my resolutions). I do like cocoa and contemplating, though...   Happy new year! Good luck and success with all your resolutions – analytical or otherwise.     ... View more

Earlier in this blog series, I mentioned that functional data analysis (FDA) could be used with experimental design. It’s an interesting idea to consider. What if you wanted to optimize the shape of a curve? Or, understand how to control the periodicity of a response? That’s what I want to discuss in this last entry in this series. To do it, we’re going to use “The Way Back Machine” and go back to 2010 to that year's JMP ENBIS Challenge.   The 2010 ENBIS Challenge What's Golden Curve optimization or analysis? Let me take you back to 2010. Ian Cox (@ian_jmp  Worldwide Technical Director) submitted a simulator for an LPCVD (Low-Pressure Chemical Vapor Deposition) process to the European Network for Business and Industrial Statistics (ENBIS) annual conference for their ENBIS Challenge. The challenge was to find a process recipe that would maximize the simulated profit and reduce defectively on the simulated parts. If you want to play with the simulator, it’s still up . Part of the challenge was that you had to consider the cost of development in the profitability calculations and only had a certain amount of total material to play withal (for production and development). I stumbled into the simulator a couple of years after the fact while I was looking for an example I could use for some training purposes. It turns out Ian did his homework when developing the simulator, and it does a nice job of replicating the output of the LPCVD process with realistic defect rates, etc. It’s still an informative tool for demonstrating why DOE is the only way to do process development. We’re going to use it here, although maybe somewhat outside its original scope. Golden Curves Ahead One of the main benefits of FDA is that it makes it possible to use curves (not individual data points) as factors in a model. One could also argue that we would want to look at how process factors (time, temperature, etc.) affect an observed curve. FDA can do that, too. The next logical step in that line of thinking is:  I know the curve I’m after  (either through first principles or process constraints) . I want to optimize my process conditions to fit the curve I’ve provided as closely as possible.  This idea is an extension of what we would do in JMP with the Profiler and the Maximize Desirability routine. In the world of FDA, that is called a “Golden Curve” optimization. This Golden Curve Analysis is what we are going to focus on today.   The Process First off, I’m not going to give a long-winded treatise of LPCVD. It’s not that important to this discussion. But, just so you’re not flying blind, here’s the elevator speech version. The process is used to put films and coatings on stuff. The films can be functional (corrosion protection) or decorative (like finishes on higher-end bathroom fixtures). In LPCVD, you put a substrate into a chamber (this could be a silicon wafer, a drill bit, or the faucet in your kitchen). The chamber is then evacuated (the low-pressure part) and heated up to some temperature. Then reactive gasses are introduced into the chamber at a controlled rate. The gasses react and attach themselves to the surfaces of the parts. Some times radio frequencies are used to break chemical bonds in the gases to make the process more efficient. ENBIS Simulator DOE Interface In the DOE simulator, we have control over a number of the process parameters and gas flow rates. The simulator gives data measured from 49 sites across the substrate arranged as a center point and three concentric rings of measurements. This measurement pattern is relatively standard in the industry. It’s not my favorite by a long shot, but it shows up a lot.  The Problem There are a couple of problems we have to deal with in the LPCVD simulator. First is the issue of getting the target thickness and film parameters correct. If you didn't care about the profile of the film, this could be done using a standard DOE and focusing on some summary statistic like the average or median film thickness. The second issue is that, because of the geometry of the chamber and the way the gases flow, the film is not uniform. It usually has a toroidal feature in the middle of the substrate. This second problem is why we need to use FDA to deal with the analysis.   The Analysis   Let me skip ahead to the bit where I'm analyzing the DOE results. Doing a graphical analysis of the DOE wafers, we can see that the non-uniformity of the wafer has radial symmetry. We can use this to our advantage and convert the (x,y) coordinates provided by the simulation to calculate the radius for each point using the Pythagorean Theorem. This transform reduces the dimensionality of the data set and makes FDA more straightforward.     Before I do the FDA in the Functional Data Explorer platform in JMP Pro, I need to add a set of data to the DOE analysis results. This data set will be the target curve. It contains the target parameters for a large number of radii between the measured data points. I'm providing a large number of data points to force the optimization algorithm to look for a solution that matches the target curve (in this case, a uniform value across the substrate) closely.  Now, I can do FDA. The setup in JMP Pro is below. I’m focusing on just the thickness in this example. Note that I’m using radius as my “X, Input” value. Recall in an earlier post in this series that the definition of time is somewhat fluid in FDA, so this is OK. Note that my DOE parameters are supplementary variables. Once I’ve hit OK in the launch dialog, I need to do one thing differently than I usually would with FDE. I need to click on the “Load” button in the Target Function part of the FDE interface. Doing this will bring up a dialog that lists the function IDs. I select the one associated with the target function (added when I created it) and click “OK.” Now, with the preliminaries over, we can create the functional form of the experimental data and pass it to a modeling platform. There’s a good tutorial on how do the mechanics of creating the functional form here. JMP Pro 15 handles all the mechanics of passing the functional forms and the supplementary variables to the Fit Model Platform behind the scenes. So, you only need to select Functional DOE analysis from under the red triangle menu for the model. This will give you access to the Generalized Regression personality in Fit Model and pull up the Profiler automatically from within FDE. (Note that all the reports and data are available here on JMP Public.)   When you’ve provided a target function, the Profiler has two additional rows of graphs. These show the difference between the function produced by the current settings and the target function and the integrated error between the target function and the current function. When you select Maximize Desirability from the Profiler red triangle menu, JMP attempts to minimize the integrated difference between the target and the displayed function. In our case, JMP searches for the setting that produces the flattest thickness curve in the process. From there, you’d need to read off the values in the Profiler and do a couple of verification runs to check the results. (Always verify your DOE predictions, people!!). Let's Practice What We Preach So, looking at the FDOE Profiler, we can see that the parameters that really drive this process are Deposition Time, Deposition Temperature,  and Deposition Pressure. We are going to presume that we need to run the process at 300℃, deposit for 824 seconds at 261 mTorr. However, there is a great adage involving assuming and what it turns oneself into ... so we're going to do some validation runs with those settings. Before I do the runs though, I'm going to do a little simulation using the Profiler's simulation capabilities to understand the probable range of the data. I'm going to do that to act as a stand-in for a calculated confidence interval. It's not perfect, but it will at least give me an idea of what the output data should look like for this situation if my model is correct. Here's the FDOE output with the simulator turned on and some best guess values for the process parameters.   And, here are the results of the validation runs with some historical data from the ENBIS challenge for comparison. Note that this control chart is looking at the average film thickness over the entire wafer. If you look at the Profiler above, you can see that the edge and outer region of the wafer have lower film thickness. They also have the majority of the data points in the wafer (look at the sampling plan at the top of the article). This will combine to skew the wafer mean values downward (one of the major reasons I'm not a fan of this sampling plan). But, that little quirk aside, the wafers are now all within spec and the variation lot to lot has also dropped.   Conclusion Golden Curve Analysis is probably my favorite application of FDA. Trying to optimize a process with a known spatial correlation while ignoring the effects of that correlation has always bugged me. However, it’s quite a challenge to do the analysis other ways -- particularly within the framework of a DOE. ... View more

If you're just now tuning in to the series, welcome! This is the third entry in what was supposed to be a simple little blog article to deal with some writer's block on another project. Things escalated quickly…  Here are the other entries in the series: A 5-minute introduction to functional data analysis Chemical reaction monitoring using functional data analysis And now on to the subject of this post.  An Interesting Question How do you use functional data analysis for images or wafer maps? Almost as soon as the Functional Data Explorer (FDE) came out, I started getting questions about using FDE for images or wafer maps. Conceptually, it made sense to me that this would be useful. And, I was pretty sure the math could handle it. However, Functional Data Explorer in JMP Pro wasn't set up to handle the data structure. So, I found a way around the problem. This article is an explanation of the thought process and mechanics of how I got image/wafer map data to work in FDE. The trick is to change your point of view. Let me show you what I mean.   Some Motivation Before we go too far down this path, let's take a second to consider why we would want to analyze wafer maps or images. People are visual creatures. We can easily see patterns in data that (whether they are there or not).   This ability is particularly useful in manufacturing and defect analysis. With wafer maps, it is easy to pick out patterns caused by different processes by eye (like the ones shown above). Having an algorithm that can do something similar that presents the information in a manner that is easy to use in subsequent analyses is priceless. Similarly with image analysis or other data that are presented in the form of heat maps, being able to translate visual information into an analytically friendly format opens many new avenues for analysis.   A Thought Exercise The biggest problem with getting FDA to work is deciding what the "function" is in the data set. And that's where most people start. In JMP Pro, they will usually sit, staring at the "ID, Function" field trying to figure out what to put in there before they do anything else -- and that is a horrible plan,  especially  when you're trying to do FDA on images, wafer maps, etc.   Let me give some context to where I'm going with this. The question that got me thinking about this problem was around something called a wafer map. Here's a picture of one: In the semiconductor industry, we tend to think of wafer data in one of two ways: a collection of individual cells (called die) or as images, where the wafer is the thing we analyze. From a process perspective, that first method doesn't make sense. Since changes to a process affect the wafer globally (it's tough to adjust process conditions to alter one specific die), each wafer is more like an image. And therein lies our problem. How do we encode the spatial information into a functional data analysis? Now, what if we stop worrying about defining the function for a second. What if we change our perspective a little and determine what the "time" variable is? If we do that, the path forward becomes more apparent by the process of elimination. It turns out each wafer is a snapshot in time. The same goes for pictures -- just like a movie -- each frame of a movie film represents an instant in time. If someone is doing FDA on image data, they are generally looking at the same field of view collected over time. With images, "time" can also be thought of as slices moving through a volume or changes in energies, exposure times, etc. So "time" is a stand-in for an independent variable (i.e., what you alter in your experiment, etc.).   In our semiconductor data set, the wafers all to have a value attached to them called the Wafer ID. These IDs are ordered chronologically. In the 3D scatterplot above, we can use the z-axis (up) as moving forward in time. So, if the wafers (or images) are the "time" variable, what does that leave for us to consider the "function?" The answer is the die or pixels. If we consider the (x,y) coordinate of each die or pixel as an ID of sorts, we can create a unique identifier that we can use as the function ID. Using that logic, we can plot the average response for different families of functions (die) and look at the average number of defects over a series of wafers (the "time" domain). We can see that the edge has a much larger problem with defects than the center of the wafer (to which every semiconductor engineer goes, "Yup, now tell me something I don't know."). Changing Perspective It turns out that considering a die or pixel location as the functional ID has some positive and negative consequences. On the plus side, we can do FDA on images and image like structures (wafer maps). On the downside: Within the FDA process, we lose our ability to interpret the results visually. We get it back on the other side, but we're flying blind while we're building the functional forms and have to rely on the numbers.   So, how do we change our perspective? The first step, getting the data into JMP Pro, particularly for images, has been covered by @JohnPonte in a couple of Community posts (Image Analyzer presentation and add-in) a few years back. You need a stack of data that has your response of interest, and the (x,y) coordinate pair for each pixel, die, etc. After this, we treat the images (wafer maps) as a time variable and the pixel ID (die ID) as the function ID.   Likewise, we run FDA as usual, but in this case, we're going to ignore most of the graphics -- the Score Plot and the Bar Graph with the FPC rankings being the exception. Looking at the Functional PCA table, I want to see a high score in the cumulative column (indicating that we're capturing most of the information in the data set). I'm also looking for anything unusual in the Score Plot (see the previous entry in the series).   The last step in the FDA process is exporting the FPC values into a new data table with the pixel/die ID column. From there, we can break up the ID column into the x and y components for visualization using column utilities. I am glossing over these steps, but the data sets are available on JMP Public, complete with the formula columns I used to convert the Die/Pixel coordinates into Function IDs and back again. Example 1: Wafer Map This first example is looking at defect rates in a semiconductor wafer map. The data set is part of the JMP sample data. Below is an interactive graphic of the FPC's from the study (the other steps are above). Note that FDE was able to tease out two different defect signatures in the data. The first signature is easy to see in FPC1. The second is in FPC 2 and is really weak. It's a more radial signature. FPC2 contains a lot of the random shot noise in the data. The upside of this analysis is that if we want to see what causes the signature represented by FPC1, we can just use FPC in a model/root cause analysis exercise. Example 2: Image This image set is a sample data set from the ImageJ package. It is sintered alumina particles, with each image in the original set being taken from a different depth in the thickness of the sample using confocal microscopy. The reconstructed image using the FPCs is shown below. What's interesting about this image is that the FPCs appear to emphasize features at different depths in the sample.   For fun, I took that information and did a 3D reconstruction of the sample using the first three FPCs: Conclusion So, it is possible to do some image analysis with FDA in JMP Pro. The key is making sure you consider what the Function and the "Time" domains are in the data set. More broadly, this is really a critical consideration with FDA. It can cause very different results or be the difference between a successful analysis and something nonsensical.   Next week, we're going to cover what the heck you can do with the information you get from analysis like this. We're going to marry a process DOE to functional data analysis. I hope you come back for the next episode! ... View more