Suppose you’re thinking about streaming a new movie. You read a synopsis: “The ruler of the future tells best friends Bill and Ted they must compose a new song to save life as we know it. But instead of writing it, they decide to travel through time to steal it from their older selves.” While that does sound most excellent, you read some viewer comments, too. “Lots of fun.” “The perfect movie for where we are as a people.” “Very disappointed with the weak plot.” “Please, no more.” And so forth. You weigh the positive comments against the negative and decide to watch Bill and Ted Face the Music. (How much unwatched content do you really have left on Netflix at this point?) We read viewer comments in situations like this because they tell us about people’s opinions, attitudes or feelings – things commonly called sentiment – often in relatively few words. Just a single word or phrase like fun, perfect movie or exceptionally poor can convey sentiment quite effectively, and knowing people’s sentiment can be very important. Do customers like our new product? What does the Twitterverse think about the latest news? Which provisions of the new law are most or least popular? Sentiment is at the core of the answer, and language helps us gauge it. Simply reading a text is the best way to gauge sentiment. But what happens when there are hundreds, thousands, or even millions of texts to read? We need computers for that. Computers can’t comprehend language like humans do, but they can search for words or phrases that we believe convey sentiment, score texts based on those sentiment words, and then present the scores back to us in helpful ways. This can give us a large-scale picture of sentiment that we couldn’t feasibly get otherwise. This is the essence of Sentiment Analysis, a text analytic technique coming to JMP Pro 16. Let’s see how it works. Do You Like Coffee Pods? We’ll spare you a deep dive on the latest Bill and Ted film in favor of a perhaps more everyday example: coffee pods, those little capsules used in single-serving coffee machines sometimes found in breakrooms, at continental breakfasts, or in some people’s kitchens. What do consumers think of coffee pods? We’ll analyze 9,133 coffee pod reviews on Amazon taken from a larger data set of Amazon reviews found here. To avoid controversy, we’ll collectively analyze reviews of many different brands’ pods and won’t single out any specific products. In practice, though, we’d want to restrict the domain of our analysis tightly, perhaps analyzing reviews of just one of our own products or one of our competitor’s, which would be more informative than looking at all coffee pods. We’d also be sure to exclude reviews of related things like coffee machines or accessories, or reviews focused on secondary issues like the packaging or delivery service, because what expresses positive sentiment in one domain (delivery was cheap) might not in another (the coffee tasted cheap). Now before we go on, I hear some readers asking, “Why not just look at the star reviews?” Of course, we could and, in practice, would look at the star reviews, but (a) text is a much richer source of information and (b) in many situations (e.g., posts in public forums) text is the only information available. We’ll ignore the star reviews for now to keep the focus on Sentiment Analysis. Sentiment Analysis is coming in JMP Pro’s Text Explorer platform, and it’s pretty easy to execute. Just select the Sentiment Analysis option, and JMP automatically finds sentiment words, scores each review, and produces a variety of outputs for us to fine-tune the analysis and explore the results. Launching Sentiment Analysis Fine-Tuning the Analysis Let’s take a closer look at that output, starting with a few individual reviews to see how sentiment scores are calculated and, importantly, why and how we fine-tune our analysis. Review #4297 is relatively positive, saying things like excellent, convenient and satisfied and scoring +60 overall (on a -100 to 100 scale, where zero is neutral sentiment). How did JMP Pro calculate this score? It consulted a built-in list of sentiment words and associated scores (a sentiment dictionary) and averaged the three sentiment words’ scores together. Pretty straightforward. (If preferred, a min-max scoring option calculates the overall score as the difference between the most positive and most negative words in the comment.) Review #4297 is relatively positive The algorithm that parses the text automatically handles negators (e.g., no) and intensifiers (e.g., very). Review #6034 is a nice example. The word good gets a baseline score of +60 by default, and that score increases to +84 when preceded by the intensifier really but becomes -60 when negated by not. JMP Pro adjusts sentiment words' scores in the presence of negators or intensifiers JMP Pro’s built-in sentiment dictionary is completely customizable, and it’s wise to tailor it to your domain. Look at Review #910 below. Nice and good are obvious sentiment words, but what about too strong, which is clearly negative in the domain of coffee pods? Recognizing that many words’ sentiments are domain-specific (e.g., rock solid might be positive for a car but negative for a mattress), JMP Pro’s built-in dictionary makes relatively few assumptions, letting you add to, subtract from, and modify the dictionary to tailor the analysis to your domain. You can tailor the analysis to include domain-specific sentiment words or phrases like "too strong" Customizing the dictionary is simple. To add too strong as a negative phrase, we simply highlight the phrase and click a button to assign it a score, here -30. The full analysis automatically updates, finding every instance of too strong and scoring it accordingly. This brought Review #910’s score down from +43 to +18, which seems an improvement. Adding a new sentiment phrase is pretty painless There are a variety of additional tools for customizing the analysis, including a list of potential sentiment words you might want to add to the dictionary (here, it automatically flags words like strong, weak, bitter, expensive), the ability to modify word scores however you like, customization of negators and intensifiers, and more. Sentiment Analysis should always be tailored to the particular context in which it is applied, so if you end up using JMP Pro 16 for Sentiment Analysis, you’ll want to get familiar with these tools. Exploring the Results What can we find in these data once we’re done tailoring the analysis? First, the summary shows that sentiments skew relatively positive, with a mean score of +39.5. There are 6,856 net positive reviews with a mean score of +56.9; 1,277 net negative with a mean score of -38.7. It looks like people are relatively positive about their coffee pods, and the average positive review is more positive than the average negative review is negative. While we’re about to see a number of tools for diving into these scores in more detail, it’s worth noting that all reviews’ sentiment scores can be saved back to the data table, making them available for visualization and analysis in other tools. A summary table provides an aggregated view of positive and negative sentiment   The accompaying histogram visualizes the distribution of sentiment scores There are some different ways we might want to drill down into the results further. First, which sentiment terms are appearing most frequently? Looking down the list, we see the most frequent words are generally positive, with words like great, bold and strong ranking highly. If we scroll down, we find the most frequent negative words are weak and bitter. If we click on one of these, we can see its usage in context to get a deeper understanding of people’s complaints. Exploring comments by frequent sentiment words can be highly informative   After exploring by most frequent sentiment words, we might be interested in specific features of coffee pods. That is, instead of looking at coffee pods holistically, let’s zero in on something specific like flavor. If we look to the Feature Finder, JMP Pro provides a list of words that often co-occur with sentiment terms, so we can explore sentiment related to just those words. Here, we select flavor from the list to see what people are saying about it. We can even click Score Selected Features to ask JMP Pro to rescore every comment specific to flavor only. This is a great way to drill down into a subdomain you’re most interested in. The Feature Finder drills down into sentiments related to specific features like coffee flavor Sentiment Analysis in JMP Pro 16 Sentiment Analysis, coming in JMP Pro 16, enables you to gauge how positively or negatively people feel about something, based on their written comments. Because the analysis should always be tailored to the specific context, the tool includes a variety of features to easily customize the sentiment dictionary and scoring system as you explore the data. Once the analysis is fully tailored, interactive tools enable you to drill down into the results in a variety of ways to answer whatever specific questions you have. In our case, it looks like people are mostly positive about the coffee pods they get on Amazon, and they’re particularly fans of coffee that’s bold and strong rather than weak or bitter. Want to uncover people’s feelings, opinions or attitudes? Gather some comments, turn Sentiment Analysis loose on them, and see what answers you find. ... View more

In a recent edition of Technically Speaking, we saw how to incorporate variables derived from text mining into a predictive model in order to improve model performance. (If you haven’t watched the presentation, you may want to before reading on.) The case study focused more on the text mining than the predictive modeling, so let’s now look at two main modeling steps that weren’t discussed: namely, validating the model and scoring new observations. Model validation In predictive modeling, it’s crucial to test the model’s accuracy before using it to generate predictions. We want to ensure our model isn’t “overfit”, accurately capturing the data it was trained on but generating inaccurate predictions for data it hasn’t seen before. A great meme capturing the concept of overfitting has been circulating the Internet for a while now.   One way to check for overfitting is to set aside or “hold out” a subset of our data, train the model on the remaining data, and then test the model on the holdout set. We measure the model’s prediction performance on both the training and holdout data, and if we see a large drop off in accuracy from training to test sets, we have evidence our model is overfit. We actually used this holdout validation procedure when building the Bootstrap Forest model in the case study, though we glossed over it. A table of model fit measures from the validation procedure is below. JMP provides a number of measures along which to compare training and test set performance (if you’re curious, read about them here); we saw Entropy RSquare and Misclassification Rate for the test group in the presentation. None of the measures shows a steep drop in performance from training to test (though there are some slight drops on a couple measures, which is not unusual), so we conclude our model isn’t overfit. You may notice something: there’s a “Validation” column in this table, too. What’s that about? In actuality, we split our data into three subsets here: training, validation, and test. In our Bootstrap Forest model, the training set is used to find the best splits in each decision tree, while the validation set is used to decide when to stop making new splits in each tree as well as when to stop making new trees. That is, the model fitting algorithm keeps going so long as each new split or tree results in improved performance on the validation set. Once performance on the validation set levels off, the algorithm stops. (Remember that Entropy RSquare statistic? That’s the validation statistic used in this process.) Once the algorithm stops, the resulting model’s accuracy is assessed using the test set. In JMP Pro, all this happens automatically by specifying a Validation Column in the model specification window, as you saw in the presentation. Not all models require three-part validation like this. It’s only necessary for models that use an iterative fitting procedure and a separate data subset (the validation set) to decide when to stop fitting. For models that don’t involve an iterative fitting procedure – logistic regression, for example, which we could have used here – only training and test sets are needed. JMP Pro has an easy way to divide up our data for model validation. We used the Make Validation Column utility, which performs random or pseudorandom sampling (without replacement) on our data to create each subset, with the subsets recorded in a Validation Column back in our data table. Here we used stratified random sampling to create our subsets, which is explained in the paragraph at the top of the second screenshot. There was strong imbalance in a few of our categorical variables, and we wanted to avoid a situation where, for example, few or none of the aggravated violations ended up in the test set. This would be problematic, because we need to see how the model performs on aggravated violations to get a valid measure of the model’s accuracy. You can see in the interactive Distribution report below that we have a matched representation of each stratification column across our three data subsets. With nearly 40k rows of data, there’s a decent chance we would’ve been fine without using stratification, but it wasn’t a bad idea to play it safe. Stratification is one of several different pseudorandom sampling procedures that the Make Validation Column utility can implement (see here for more). The Make Validation Column utility dropped a column named “Validation” back into our data table (which is available for download on the Technically Speaking presentation page), and we then fed that into the Bootstrap Forest platform to trigger the validation process. We didn’t find evidence of overfitting, so assuming that we’re comfortable with the model’s overall accuracy, we’re ready to use it to generate predictions for (“score”) new observations. Scoring new observations Imagine we have a group of building code violations just issued, and we want to assign a financial penalty probability to each one, which we’ll use to weight each levied penalty amount in our financial projections. For example, a levied penalty of $1,000 for a violation with a penalty probability of 0.7 would have a predicted payment of $700. Of course, the respondent ultimately will pay either $1,000 or $0, but on average across many violations, the sum of our projections should approximate our final revenue (we hope). We get the penalty probabilities out of our Bootstrap Forest model, but that model took as inputs the scores from the Topic Analysis. So we need to run our new violations through both models. This is not an uncommon practice, using the output of one model as input to another. JMP can output models as column formulas back to the original data table (under the red triangle options in the modeling platforms), and they’ll run on any new data we add to the table. To score new observations, we simply add the new violations to our table, and the topic scores and penalty probabilities are calculated automatically by the column formulas. We then create a new column formula that multiplies the penalty probabilities by the levied penalty amounts, and we get our projected payments. Below we have projections for 1,000 new violations. We predict these will net us about $1.84m in revenue, though approximately one third of that total is accounted for by only the largest 100 projected payments in that positive tail. Our projection’s accuracy will hinge in large part on what happens with those violations. (An aside: We’ve described the point-and-click way to score observations, but scoring could be automated easily through scripting in JSL (JMP Scripting Language). The model could also be deployed outside of JMP in a language like Python or JavaScript, with JMP Pro’s ability to write scoring code in other languages.) And this highlights perhaps the most important question of all: How good are our projections, really? We have fit statistics like Entropy RSquare to capture model performance, but what we really care about is financial projection accuracy. In practice, we’d have assessed this as part of the model building phase by generating a revenue projection for our test set and comparing that to the actual revenue we observed. We didn’t do this here because, unfortunately, the available data didn’t contain the initially levied penalty amounts, only the final amounts after adjudication. (In the scoring example above, we just imagined some initially levied penalty amounts for expository purposes.) A parting thought While we’ve arrived at our goal – a model that assigns financial penalty probabilities to new violations – a project like this is never truly done. Over time, new types or trends in violations may arise, new policies may alter the relationship between violation characteristics and penalty payments, or penalty payment probabilities could simply change over time due to unmeasured factors like enforcement capacity or leniency. The trends across time in the graph below might suggest the latter. (Line width represents total violations per year, which makes clear that we don’t have enough data in 2020 yet to trust the sudden rise this year.)   For many possible reasons, our model may become less accurate as time passes. We should refit our Topic Analysis and Bootstrap Forest models periodically, incorporating the latest data (and perhaps discarding older data) to ensure that our predictions remain accurate. ... View more

Learn to identify patterns and clean up data using JMP 15 or later.  See video how to use Explore Patterns to identify duplicate values, unusual values, unexpected linear relationships and spec limit anomalies. ... View more

  In this episode of Help!, Ross Metusalem looks at the plethora of online resources available for anyone interested in learning how to use (or use better!) JMP. Ross shows us how to navigate from the JMP home page to some of JMP’s most popular resources. During the session, Ross will show you valuable landing pages like the JMP Learning Library, the Mastering JMP webinar series, and the Statistical Knowledge Portal.   ... View more

  Curve, or "functional," data is everywhere. It occurs whenever we measure something over a continuum, like measuring temperature over time, pressure gradient across space, or radiation absorption across wavelength. In the image above, we have particle size measured over time while milling pigment for making LCD screens. Sometimes, the shapes of the curves we measure are important to us, and we want to know how curve shape is affected by certain factors. For example, how is the pigment size-over-time curve affected by how we run our milling process? That's a complex question, in part because curve shape is a complex thing. How do we figure out how some factors might affect it? The answer: functional design of experiments. Watch the video below to see how it's done in JMP. This video assumes that you're already familiar with the basics of DOE in JMP. If you're not, check out this on-demand webinar or, if you already have JMP and prefer a hands-on introduction, JMP's DOE Intro Kit.   ... View more

  In this episode of Did You Know?, JMP Systems Engineer Ross Metusalem explains how to get your visuals out of JMP and into your presentations or publications so they can have the most impact for your organization. If you’re currently just copying and pasting a screenshot of your visualization into a PowerPoint, Ross says “Stop!” You’re pixilating your image and negatively impacting your message. Instead, Ross show you how to use vector graphics, specifically enhanced metafile format (EMF), to get high-quality images out of JMP. He’ll also show you easy ways to customize your graphics without going back into JMP. ... View more

  JMP Systems Engineer Ross Metusalem is back to take us through another helpful aspect of … JMP Help. In this episode, Ross introduces us to what he considers the “best JMP resource available:” other JMP users! With nearly 20,000 members and more than 219,000 discussions posted, there’s a good chance your question has already been asked (and answered) in the JMP Community. Along the way, you’ll also learn how to navigate the site, search for posts and solutions, and discover top tips to consider when posting a question for yourself.   ... View more

  JMP offers top-notch Help tools and documentation on its website. But suppose you don’t know what to search for, or maybe you’re working inside the software and are curious about what a particular command or feature in JMP actually does. The Question Mark Tool in JMP is a quick and easy way to get answers immediately… from inside JMP. In this episode of Help!, Ross Metusalem shows you how to use the Question Mark Took, your direct route from within JMP to the appropriate section in Help Documentation.   ... View more

  In this installment of the Help! Ross Metusalem shows you how to use sample data and prebuilt graphs to teach yourself a variety of analysis techniques. Fully supported by documentation from JMP documentation, you can use the Sample Data Library, to learn more about JMP functionality, using sample scripts, teaching scripts, sample dashboards and so much more.   Full Transcript (Automatically Generated) Everybody, welcome to this edition of Help review of jmps help resources and how to use them effectively. I'm Ross to slim systems engineer and today we're going to talk about the sample data library, which is a repository of example data sets and pre built analyses that you can use to get some hands on hands on practice with jump. It's great for better understanding jumps capabilities and for honing your analytic skills. So let's take a look at an example around the partition platform. So for those of you who are unfamiliar with the partition platform, it is a tool for building decision tree models which are great for data mining and predictive modeling. If you're new to the partition platform, you might start here at the documentation to read up about them. After that, you may head on over to something like mastering jump to see an instructional video, but eventually, you're going to want to get Get some hands on practice. After all, it's a big leap from watching an instructional text or video to actually trying something out yourself. So we want to get some hands on practice with partition models, but we need some data. To get that practice. Let's check out the sample data library to see what it has for us.   I'm gonna go to help sample data. The window that's popped up here is an index with links to all of the sample data sets that are built into jump. So you get these automatically when you install jump. On the right, you can see we have data organized by domain. And on the left where we're going to play, we see data organized by type of analysis. For example, if I wanted to work on analysis of variance, I could check out this section where we see various data sets for different types of ANOVAs.   I'm interested in partition models. I happen to know that those live here under exploratory modeling but If I don't, that's okay. I can always go to search and find or just to Ctrl F. And with partition type then if I click find you can see it's found the first one the Boston housing data set. we cycled down we have car pole serial, the diamonds data that Julian started out with today, and a couple others as well. So numerous datasets we can use to get some practice with partition models. Let's open up the Titanic passengers dataset to see what's in there. Here we have a standard jump data table. When you open one of these sample data tables, you may notice the number of items in the tables pane in the top left here. This could include references or notes if if you find anything like that there, I highly recommend checking this stuff out because it will provide additional context. So if I hover over notes here, it'll pull up some text   or should pull up some text. Oh, I didn't have that selected. Well, sometimes when I'm impatient, I'll just click Edit. and here we can see this data table describes the survival status of passengers on the Titanic. Very good. So I want to practice partition models. I'll go to analyze, predictive modeling partition, because I've read up in the documentation that that's where the partition platform lends. Now, what do I do? Well, if I'm brand new to partition models, I might look at this launch dialog and say, you know, I'm not exactly sure where to put all these variables to make sure I get a valid analysis. So luckily, this data table has come with a few pre built analyses in the form of table scripts. This first one here is for the partition platform. So I'll click the green button to run the analysis. So now we actually have bypassed that launch window and we know that we have a valid partition analysis. If I want to see how that launch window is actually filled out. I can go to the red triangle, and under redo, relaunch analysis and I can see that okay, we took survived, that's our response variable and put it into the Y response. Roll. And then we took several factors and put them in X factor.   So this is a nice way actually, as I've learned to jump throughout the years, I've often found myself launching prebuilt analyses in the sample data library and then doing this redo relaunch trick to see how they were set up in the first place. So now I have this analysis. And if I wanted, I could, you know, play around with it, explore it a little more request a couple splits in my tree, maybe to find that, Oh, well, it looks like females at a much higher survival rate than males. And in particular, those females were in passenger, class one and two. So this is this is pretty nice. It's a great way to kind of poke around partition platform, learn more about it. But you know, there are a lot of options maybe I don't know what the split and prune buttons do necessarily, or what some of these options under the red triangle do. Could be the case that you don't feel entirely comfortable just diving right into a data set. And you'd like some more kind of hands on guides. One thing that's really nice about the data that you find in the sample data library It's referenced in the support documentation, oftentimes with step by step examples. So let me pull up the partition documentation. And if we look over to the side here, you can see we have an example. When I click that, I actually have a step by step example of launching and using the partition platform to build a decision tree model and interpret the results. You can see the first step here says, Go to help sample data library and pull up the diabetes data set. And so this brings me to something I want to point out here.   When I first went to help, I went down to sample data. But if you were watching, you may notice someone write over something that said sample data library. So what's the difference between sample data and sample data library? Well, the library is just referring really to the folder on my computer where all these sample data sets live. And so if I go to sample data library, I'm in this case in my Mac Finder window, and I can just scroll on down in this case and find that diabetes data set to work with for the exam. In the documentation, so if you want some more kind of hands on guidance, before diving into a dataset on your own, go ahead and head to the documentation, look for something that says example of partition platform or anything else. And go ahead and follow the step by step instructions first. That's the basics of the sample data library. Before we go, I do just want to point out that it actually contains more than just data tables. If you look at the buttons up here, you're going to see buttons to open various other directories, including a directory of sample scripts, or sample dashboards. So these, again, are just kind of pre built materials that you can use to poke around and hone your skills. So that's, you know, the sample data library. It's what it's all about. It's providing you the materials you can use to get some hands on practice with jump to better understand jump capabilities and to hone your analysis skills. So that's it for help today, Julian, I'll kick it back to you.    ... View more

Learn how to use the Scripting Index to more easily learn JSL, either to write your own scripts or to customize the scripts JMP automatically produces for you.   Full Transcript (Automatically Generated) Hi, everybody. Welcome to another edition of help, our review of the help resources in jmp and how to use effectively. I'm Ross, systems engineer. Today we're going to look at the scripting index, which is an excellent resource for getting the most out of jmp scripting language. If you know how to script or as we're going to see right now, even if you don't, for those who aren't familiar, jmp scripting language or JSL is a great way to extend jmps functionality, or to automate some of the things that you would normally do in jmps point and click interface. Right now we're going to look at an example of customizing a script that jmp produced for us automatically to get it to do something that we want.   So let's take a look at this data table. This is a data table that I have imported through the Excel import wizard. If you've used any of jmps import wizards before you have probably notice that there are tools where you can enter information to specify the formatting of that file, so that jmp imports it correctly. And this is one of the places where jmp produces automatically for you JSL scripts. Here we have the source script in the data table, I'll right click and select Edit so that we can take a look at what's in there. So you can see we have an open command where jmp is saying all I want to open this file, and then apply all of these settings that I had applied through the Excel wizards point and click interface.   So this is a great script that I can use to import other Excel files that I have that have the same formatting as the first one that I imported. I have one of those Excel files here on my desktop here. So you see I originally imported a file from December 2019. Here's one from January 2020. So I want to use this script to import the January 2020 file without having to go through all the pointing and clicking in the Excel import wizard. simple way I could do that. Maybe I'll actually just copy paste This filename into my script. Now I'm doing this in the table script. Normally, I would copy this out to a separate script window. But for our purposes here, this will be okay. So I've put that in.   And if I click run, jmp imports that January 2020 data file. So this is great I basically automated in a way Excel importing for myself. But this doesn't scale that well. You know, this requires me to copy paste or to type in which file I want to import. And I would rather have a solution where jmp presents me with a file open window where I can just point to the file that I want, and run the script on that. So that's something that I should be able to do in GLSL, but I don't know how so be a little help. Let's jmp to the scripting index to find some more information. So you'll find the scripting index under help scripting index. unpack what we're seeing here. In this middle column, we have a bunch of different commands. In JSL, and on the right we get a bunch of information about those commands.   On the left, we can view these commands by different categories, we want to operate on a file. So I could scroll on down to the file section here and see all the different commands that operate on files. I'm actually going to go ahead and just type something into our filter here to try to find what we're looking for. I want an open file window. So what I want to be presented with let's see what it pulls up. Well, I get a number of matches. Let me go ahead and filter down a little more to say Just give me the ones that operate on files. I got three matches, all right, open pick directory pic file, no pic file sounds like what I want to do.   So I'm gonna go ahead and take a look at that. Up at the top here on the right, we see the syntax for using pic file. Then we have a simple description here, where if you notice at the beginning says that it prompts the user with an open window returning the path name of the file. kind of sounds like what I want to do, but you know I want to learn more about how to use this pic file command before putting it Script down here, I have some example JSL code that I can run to see how this works. Let me go ahead and click this button to run the code that pops up this file open window that I want.   Let me go to the desktop and actually just point this right at that January 2020 file and click Choose and see what happens. So we're done. If we look down below, we can actually see what jmped did, it ran this pick file code. And this is what it produced. Here we have a text string that looks like where that file is living. And the file name corresponds pretty cleanly with what we actually want in our script. So it looks like PIP file is going to produce what we need. So it's good to go to pop into our script.   But before we do actually want to learn a little more about PIP file, I could go to the topic help, which will take you to the help documentation online. And if if we were to go there for the sake of time, we won't you'd learn that it says all of the arguments here all this extra stuff that maybe I don't totally understand it. is optional. Now one nice thing about the sample code that's put here in the scripting index is that it's editable, you can tweak it and then run that code in this little kind of sandbox to see what it does. In this case, I'm just going to delete all this stuff, maybe I don't even need it.   So let's just run a bear Peck file. So I'll run that again. And lo and behold, I get my file open window. I'll point to that file. I click Choose, it still returns what I'm looking for. So it looks like we're good to go. And because I'm feeling a little lazy, I don't even want to type the pic file out. I'm just going to go ahead and copy that and paste it right into my script. Now if you right JSL you probably think this looks a little funny. You know, leaving this semi colon here and all that but you know, this should work for our purposes.   So I'm going to go ahead and click Run. Alright, it's letting me point to file point to that January 2020. file, runs it through the Excel import wizard without us having to click anything and now I've imported a brand new file. So been able to use the scripting index to figure out what I needed to tweak this pre existing script just so that I can now Can I have a custom data importing tool. And I did that without writing a single line of code.   Instead, I was able to, you know, efficiently search the scripting index using the groupings and the filter to find this command that I wanted. I read a little bit more about it, I actually experimented with the code in our little sandbox down here. And then I was even able to actually just copy and paste some of this myself. So take home message JSL really powerful tool for extending jmps functionality and for automating your workflow and the scripting index, a powerful resource to help you get more out of JSL than you might otherwise. So that's it for help, Julian, back to you. ... View more

See how to design and analyze an experiment where the response variable is a function (e.g., size or temperature over time) instead of a single point measurement.   For a quick overview of functional data analysis in JMP: Blog: A 5-Minute Introduction to Functional Data Analysis 5-minute video: Functional Data Explorer in JMP 14   To go more in depth on functional data analysis: Mastering JMP: Using JMP Pro to Pre-Process Functional Data and Create Surrogate Models Mastering JMP: Using JMP Pro to Model Functional Data    Full Transcript (Automatically Generated) It's Ross here, systems engineer coming to you from Tampa, Florida. And as Julian said, In this segment, we're going to talk about functional design of experiments, which generally, is the situation in which you're conducting an experiment and your response variable takes the form of a curve instead of a single value.   So let's unpack that idea a little bit. In essence, when we're doing functional design of experiments, or F do we were asking how factors of interest affect the features of a curve that we care about, for example, we might have a load deflection curve. So here on the x axis, we have the load placed on a surface, and on the y axis, we have the deflection that we measure, we see three different curves for three different widths of the material, we might be interested in how the materials width, or some other property and material actually affects the shape of this curve. Here we have a reflectance curve for various types of mirror coatings. So on the x axis, now we Have the wavelength of light and on the y axis we have the percent of light reflected at those varying wavelengths. We might be interested in how different aspects of mirror coatings affect this curve, maybe because we're trying to achieve a certain ideal curve. or third example, one, actually from research that I've done in the past as a cognitive scientist, using sensor data in particular, Eg or brainwaves, voltages measured over time generated by the brain, we might be interested in how the type of stimulus we show someone actually affects the shape of the eg curve we mentioned.   And, you know, sensor data actually is probably a really prominent application area. For this type of thing. A lot of us deal with sensor data, whether it be voltage over time, like we have here, or temperature over time, Ph over time, and so forth. But in all these examples, we're just interested in how some factor of interest actually affects this curve and its shape or its features.   So the example we're going to talk about right now is milling pigment particles. for manufacturing LCD screens, so we start with our raw pigment. And we put it into a bead mill depicted on the left here. And we have various factors that we can control with respect to the milling process, for example, the flow rate or the temperature. And we're interested in how these factors affect the shape of the curves that we see on the right. So you see in these curves where we have time on the x axis and the size of the pigment on the Y, we see, generally speaking, a pretty rapid decrease in particle size at first once we start milling, and then it starts to level off a bit. in green, we see this specification range, so that's where we want to stop. Now we really care about the shape of the curves here. You know, we don't want to spend too much time milling. So we want to see a steep drop off at first so that we can get into that green range as fast as we can. But we also don't want to overfill so we want to see a nice steady kind of asymptote within that green range so that if we happen to leave the mill running a little bit longer We don't actually overshoot and end up with particles too small. So this is a great time to use DLP, right?   We want to know how we can basically optimize this response variable these curves by manipulating our factors. So we would first use jmps DLP tools to figure out how to systematically manipulate the factors, then we would actually record the curves across multiple runs or batches of pigment. And then finally, we would model the relationship between the two. All of this is just the basic DLP process. So here, we have an equation that looks much like the one Elisa just showed us. This is just capturing the relationship between temperature and the curve. So we have some intercept plus temperature times some slope coefficient equals and if you're with me now it's you know something about the curves shape, right? But what, how do we get the curves shape into this equation? That's kind of a big challenge that we're dealing with here in St. Louis. We need the The measure here to be a single value. So the question is, what do we measure? Well, if we're really concerned about just getting to spec as fast as we can, maybe we just measure the time at which we hit spec. But that doesn't really tell us if we get that nice asymptote to make sure we stay in spec. So maybe we see if you know, what the size of the particles is, when we see that the decrease in sizes leveled off whatever leveled off means we would have to operationalize that to maybe okay measures, you know, somewhat imperfect and certainly don't capture the shape of this curve. Maybe we actually instead regress size on time using something like a quadratic model, and then enter those coefficients in as the quote unquote curve shape. Maybe that would do better or maybe not, you can see none of these solutions are ideal. So what we really want is a better way to actually capture the shape of the curve in a single value.   And that's what jmp pros functional Data Explorer can do for us it in essence finds the primary features or shapes in a curve. translates them into single values. The basic idea is this, we take our data, we fit a model to the data, and then that model represents each curve as some average shape on the left center there, plus or minus some amount of some primary shape feature, or in technical terms a functional principal component.   From that model, then we pull out these shape feature scores or where it says FPC one, these functional principal component scores. And these actually capture, for this case, the batches in our experiment, you know, how much of that primary shape feature we would add in or out to approximate that shape. And so in this way, we can actually kind of stay true to the shapes of our curves and still enter them into analysis techniques like linear regression.   So this is on static images, it's kind of hard to understand or visualize, especially if you've never really encountered this idea before. So now I want to switch over to jmp to kind of show you how it all works. So let's bring a dataset in Here's a data set from the experiment where we manipulated our factors. So once again, these four variables here, and then we measured across multiple batches, the size at various times. So the first eight rows here represent the measurements taken from batch 2887.   Just to show you I've also included down at the bottom here, an ideal curve, if you remember I said we kind of wanted to get down to spec really fast, but then just stay there. So this curve is we're about to see actually just kind of captures that ideal state for us. What I'm going to do now is show you the end result of the analysis just so that you can get a get an idea of where we're going. And then I'll run back through and show you how to produce it using tools in jmp. So here's the output of the analysis in jmps functional Data Explorer, this is in jmp Pro. Let me walk you through what we have here the basic elements up in the left side We have each of our batches, you can see them numbered there and jmp has fit a flexible curve to each one, shown in red. It's using something called a spline model, which you can find more information on. On the right. For example, we have a one not cubic model, as you can see here. We then use that model to perform something called functional principal components analysis, which is what actually captures the average shape and this primary shape component. So we see our average shape of those curves here.   And this one component shape that actually explains 99% of the variability in all the shapes of our curves across all of our batches. Down here in the scatterplot, we can actually see for individual batches that either have a lot of that shape on the right side or actually subtract out a lot of that shape on the left, what they look like we'll get 2899 I'll just pay This here, you can see it says it has a component score of 77. So we're adding in a fair bit of this shape. And you can see it doesn't look so great. It's not exactly what we would want to see, we don't have a really steep drop off. And this is kind of a nice level trend down that doesn't ask them to like we would want.   So we can actually kind of explore that a little further down here, we have a profiler that allows us to see on the left panel, the shape of the curve that we get out of the model, given a certain value of the shape component on the right side. So let's go ahead and actually add some of that shape component in. And you can see that the shape on the left is actually changing. And as we get to a pretty high value, you can see that the models shape actually starts to look an awful lot like 2899. Fact 2899 has a component score of 77. So I could go ahead and just enter that in there. And now we can see that the models shape actually looks very much like the shape we observed.   Conversely, if we go on All the way down to batch 2887. This is one that has a pretty negative shape score negative 83, you can see the shape is a fair bit different, it actually looks a lot better, it drops off fast and then kind of levels off. If we enter negative 83. As our score, you can see that once again, the model has captured that now we have a shape that looks an awful lot like what we observed for 2887. So these scores, the 77, and negative 83. These are single values that actually capture a complex variation in the shapes that we saw across all the batches that we run.   So now we have single scores, we can actually enter into an analysis in a way that then allows us to kind of model the shape of our curves with kind of a greater degree of fidelity. And that's what we do. Down here where it says functional do II analysis. Remember our goal is to actually see how changes in our factors affect the shape of the curve.   jmp has used pros generalized regression platform to automatic fit a model for us that relates our factors to the FPC scores, those shape scores that we were looking at. And we can see for example, as I increase the percent beads, we start to get a shape that looks a little more like the one we're after. So increase percent strength, again, we start to see slight improvement in shape, just like you might have seen elsewhere and jmp we can even ask to maximize the desirability that is to find the factor settings that get as close to our target as possible. so here we can see these factor settings down below, give us a curve that starts to look pretty good. We have a nice steep drop off, and then it levels off right here. So that's functional do we in a nutshell, you have your factors of interest, you let jmp extract shape features, and then you just relate your factors of interest to those shaped features and even optimized by finding a set of factors. settings that yield the shape closest to the one that you want. So now in the last few minutes, I just want to walk you through kind of how all this works. So if you want to try this out yourself, you know where to go.   Let me close this down. First, you may be wondering, well, how did we actually design an experiment, these were experimental data is actually from a definitive screening design. This is fairly straightforward. Under the DLP menu, you'll find our normal DLP tool. So I'll go to definitive screening here, and then jmp pro 15. Under the responses, you're not going to see a functional response type. So I'll go ahead and choose that. I'll actually remove this other response this non functional response, you can see we can enter the name just like we would.   Otherwise we can set the number of measures per run. I'll just leave it at five for now. And this is flexible, you could add or remove measures per run as you go along in conducting your experiment. Then we enter our factors as we normally would have a table representing those factors settings just to speed things up. So I'll go ahead and load those factors in. Click Continue. And then I'll keep the standard design options for the DST and just click Make design, we have our design out. Now we'll make the table. So this is a jmp to a table, we see places to record size, I only specified five, so I get them. So you can see each row here represents one function, but you can go ahead and if you want maybe use tables stack as we had done to put this in a tall format, which then would allow you to create a time column and keep track of which time you took the measurements and so forth. You'll also see just like with any other deal table that you get out of jmp, you have your built in scripts to pull up the design window again or conduct your analyses and so forth. So nothing too big there.   You just make sure to specify your functional response type. Now, let's actually take a look at The functional Data Explorer platform and how we use it to analyze the data. So you'll find functional Data Explorer under specialized modeling, functional Data Explorer, I'm in the stacked or tall format, though you can enter data and other formats. For example, rows as functions would correspond to the format that we got out of the DSD platform. These first two fields here, x and y just refer to the variables that define our function. So ours was size across time, so I'll put size as y times x.   Because we're in this stacked format jmp needs to know which row belongs to which experimental runner which batch so I'll go ahead and just put batch in as the ID. Then I'll grab our experimental factors and put them in these z supplementary role so that they get passed along to functional Data Explorer.   Here's our window. You can see it looks similar to what we had before our data have all been loaded in in the green Start Here is actually the target function that I had loaded previously, which you can see down here with the nice steep drop off and then the asymptote.   So right now Trump is treating that as actual data, which it's not. So we want to make sure to tell jmp that that's actually the target function, I'll click Load. And then say that the one that I've labeled target is our target. You can see it's removed here from the plot, and kind of broken out down below.   There are a lot of options you're probably noticing on the right here for processing the data cleanup, transformation, and so forth. We don't have any of that to do here. So what we'll do next is actually fit the curves to the data or the spline model. Here we have a few different classes of models we can use. I'll choose B splines.   We have curves fit to each of our experimental runs, and the functional PCA that extracts that shape component has been done for us. Once again, just this one shape component explains 99% of variability. So we don't really need another shape component. But in other data sets, you might get multiple shaped components out. We have, again, our score plot, and our profiler that allows us to see how changes in these FPC scores affect the shape. When I'm ready to go, I'll just go next to the model title here and request the functional DLP analysis. And now down below, I have the profiler that relates our factor settings to the shape of the curve. And once again, this is done using jmp rows generalized regression. And I can actually modify the model or look into the models details further, right here. So I can go ahead and open this up. And for example, maybe launch another type of model. Or if I care to even view some of the reports, for example, the parameter estimates, so you're not stuck with whichever model is first chosen, you can in fact, kind of work with the model to get one that you're satisfied with.   That's functional do II in 15 minutes, the take home message, you have factors that you want to relate to the shape of a curve. Under the do II menu, you can create your design with a functional response type and then analyze your data in functional Data Explorer by first fitting the curves to the data, then performing the functional principal components analysis and then finally, requesting the functional analysis.   So,a pretty sophisticated technique here available in a nice easy to use package.   You know, this is a deep topic though. So if you're looking for more information, I encourage you to head on over to the jmp on air space on the user community, where for the page for this presentation, you'll find links to some more helpful info. So that's all for me. Thanks for your attention, Julie, and I'll pass it back to you. ... View more