The Engineering Mailbag Episode 5: I. DO. NOT. LIKE. SPIDERS!!! (And spider plots can take a hike, too.) I can't ignore spider plots forever, so here we go... Every now and again, we systems engineers run into interesting questions that would fall somewhat outside the typical range of JMP usage. These applications are generally clever, often bringing home how using data isn’t just for business or technical problems. Other times, the questions are just unexpected, challenging problems. These “curve balls” (as I like to call them) come in many different forms: coding problems, interesting analyses, ways of visualizing data…you get the idea. Occasionally, I’ll get questions about problems that I really don’t want to answer. Today's query is in that little pile, and I have avoided this one for years! The subject of this entry in The Mail Bag is generally regarded as a scary, unpleasant plot by the data viz community, so I guess it’s appropriate for a Halloween post. Since this type of plot shows up regularly for cosmetics and consumer analytics customers in my day-to-day work, I can't ignore it forever. So, I’m taking this as an opportunity to help out a number of users groups in New York and New Jersey. But I have to say…I really do not like spider plots. The question Like I said, I’ve received this question more often than I care to mention. So, I’m going to skip the actual question email, you'll just have to trust me that there have been several. There were also some in-person requests. And a few posts on the JMP Community. Yeah. Below is an example from a peer-reviewed paper that someone gave me as a reference for what they would like to see: (J. Inst. Brew. 2012; 118: 325–333) Author's note to any data visualization specialist that may be reading this: Please forgive me. I know this is an awful graph. The response My general response for these requests historically falls along the lines of what most data visualization experts have said, which is to discourage people from using this plot if at all possible. However, recently, I started thinking about this persistent question a little differently. I’ve since taken a different tack on the problem. My current philosophy about spider plots is that I’m open to helping people see how to make one if I can also show them better ways to present the same information along the way. As was the case with the other visualization question I answered, for me, the process of making this chart was much more instructive than the actual result. I. HATE. SPIDERS. OK, I have (or rather, had) a strong antipathy for spiders for many years. I wouldn’t quite call it a full-blown case of arachnophobia, but for many years I would dispatch anything with eight legs (and occasionally six; you can never be too sure!) with extreme prejudice. I gave no mercy. I gave no quarter. I. Took. No. Prisoners. Anyway, I eventually got over it. And, with the exception of ticks (vampire spiders!), I have made peace with my eight-legged friends. We generally just stay out of each other’s way now. I’d like to think that part of my aversion to spider plots came from my issue with spiders, but it could also be that they’re a royal pain and I instinctively knew that making one would be a bit scary. That’s not to say there aren’t some interesting coding problems involved in constructing them, particularly if you want them to have the interactivity users expect from JMP (just that I knew the code itself was going to be a bit intimidating). Anyway, let’s have a look at the problem. Scoping the problem Practicing what I preach, I started with a problem statement: I want to make a JMP add-in that displays a spider plot and other possible ways of visualizing a data set in a framework that I can add on to later. The next step is to create a punch list of the different things that I need to do: Get the dimensions for graphing from the user. Get a label column from the user for each data subset. Construct an interactive spider chart for the provided dimensions by each subset. Construct an interactive radial chart for the provided dimensions by each subset. Construct an interactive parallel chart for the provided dimensions by each subset. Construct an interactive table of pairwise correlations for the provided dimensions. Scoping the problem revealed some good news, specifically, I can offload a lot of this stuff to JMP if I use Application Builder, which will also make it really easy to add new visualizations later if needed. The challenging bits were mostly around the first two charts and making them interactive. JMP’s graphs work in Cartesian coordinate systems (x and y coordinates). Those graphs are, for all intents and purposes, in a polar coordinate system (r and φ coordinates). As a result, I am going to need some code to convert back and forth between Cartesian and polar coordinate systems. Let’s start there.   Coordinate transform functions Here’s the function that I used for working between the two coordinate systems: polarToRect = Function( {r, t},             Eval List( {r Cosine( t ), r Sine( t )} ) ); It’s fairly simple. Just what you’d find from a geometry textbook. It is a pretty nice little bit of code, in that it takes radius (r) and angle (t) and returns a coordinate pair (x,y). BTW, this function is also in an application that I reference later on that’s included in the JMP sample applications.   Because I’m working in Application Builder, just about everything else I’m going to do will be either a function or an expression. Everything about this project is also spectacularly repetitive, so it will make the iterative parts of my code easier to read if I’m just calling functions. The easy bits If you have a look at the final app source, you can see that I’m leveraging some existing parts of JMP: Graph Builder for a parallel plot, and Multivariate for the pairwise correlations. They were created, almost exclusively, using JMP generated code or capabilities in the Application Builder. So, we’ll get them out of the way first. Because of the way that Graph Builder treats columns in this case, and the fact that I have to be able to handle an arbitrary number of variables, it’s easier for me to build the Graph Builder script as an expression and then append it to the main display tree in the correct place. Here’s the code for that: drawParallel = Function( {pCol},             {default Local},                         // build the list of variables with the format "X( :col1), X( :col2)" as a string             // also building a list of plot elements             colStr = "";             gElemStr = "";             For( i = 1, i <= N Items( pCol ), i++,                         If( i == 1,                                     colStr = colStr || "X(" || Char( pCol[i] ) || ")";                                     gElemStr = gElemStr || "X(" || Char( i ) || ")";                         ,                                     colStr = colStr || ", X(" || Char( pCol[i] ) || ", Position( 1))";                                     gElemStr = gElemStr || ", X(" || Char( i ) || ")";                         )             );                         // Get the grouping variable as a string             xVarStr = Char( gCol );                         // Insert the strings into the main string             Eval(                         Parse(                                     Eval Insert(                                                 "parallelVLB << Append(Graph Builder(                         Size( 766, 256 ),                         Show Control Panel( 0 ),                         Show Legend( 0 ),                         Variables( ^colStr^, Color( ^xVarStr^ ) ),                         Elements( Parallel( ^gElemStr^, Legend( 3 ) ) ),                         SendToReport(                                     Dispatch(                                                 {},                                                 \!"Graph Builder\!",                                                 OutlineBox,                                                 {Set Title( \!"Parallel Plot\!" ), Image Export Display( Normal )}                                     ),                                     Dispatch( {}, \!"X title\!", TextEditBox, {Set Text( \!"Variables\!" )} ),                                     Dispatch( {}, \!"graph title\!", TextEditBox, {Set Text( \!"\!" )} )                         )             ))"                                     )                         )             ); ); Notice that all I’m doing is formatting the variables as strings and plugging them into the right spots in the code and then running it. The end result is a fully interactive parallel plot in my app with very little coding on my part. I tried to script up a parallel plot manually; it worked, but just barely. I don’t recommend it. Just use Graph Builder and Append(). I’m also using the Eval Insert() construct.   It’s a little easier to read than constructing the string using concatenations. The multivariate plot is just a simple Parameterization of the Multivariate Platform in JMP. I didn’t even have to write code to get this bit! There’s an Advanced Mastering JMP webinar that shows how to do that with just mouse clicks. Again, I’m just doing these bits this way for convenience and because I really didn’t want to spend a huge amount of time on this part of the app. It’s not being lazy – it’s being an efficient coder. JMP gives you the code, so you might as well save yourself some time. The radial plot The radial plot is actually a port (with some modifications) of an example app that’s included in JMP’s samples directory. The big modification is that I’m using Marker Seg() instead of Marker(). And, that’s actually an important point. It’s possible to make Marker() interactive using some mouse capture commands and some logic, but Marker Seg() handles all this automatically. You just have to tell it which data table you need it to monitor. drawFlies = Function( {nP, pCol, obj},             {default Local},                         // Create a list of angles for the spider plot             th = ((1 :: nP) - 1);             th = Shape( th, nP, 1 );             th = 2 * Pi() * th / nP;                         //Get the data from the table as a matrix.             sel_dat = J( N Row( DataTable1 ), N Items( pCol ), 0 );             For( i = 1, i <= N Items( pCol ), i++,                         sel_dat[0, i] = pCol[i] << get as matrix             );                         // now get a matrix with two columns and nvars rows to keep track of the min and max of each variable             min_max = J( nP, 2, 0 );                   // column1: minimum, column2: maximum. row for each variable chosen             For( i = 1, i <= nP, i++,                         min_max[i, 1] = Min( sel_dat[0, i] );                         min_max[i, 2] = Max( sel_dat[0, i] );             );                         // Create some empty matrices for the converted values             xMat = [];             yMat = [];                         // Convert the data matrix             For( i = 1, i <= nR, i++,                         current = sel_dat[i, 0];                                                 // for each variable, subtract the minimum and then divide by the range.                         adj = J( N Row( current ), 1, 0.0 ) + 1.0 * ((current - min_max[0, 1]`) (min_max[0, 2] - min_max[0, 1])`);                         adj = adj`;                         total = Sum( adj );                         {x, y} = polarToRect( adj, th );                         x = Sum( x ) / total;                         y = Sum( y ) / total;                         xMat = xMat |/ x;                         yMat = yMat |/ y;             );                         // Draw markers             annotateRad = Expr(                         // Matrices and book-keeping variables                         xCoord = xxx;                         yCoord = yyy;                         obj[FrameBox( 1 )] << Append Seg( Marker Seg( xxx, yyy, Row States( DataTable1 ) ) );             );                         // Substitute values into the annotation expression             Substitute Into( annotateRad, Expr( xxx ), xMat, Expr( yyy ), yMat );             annotateRad; ); As you can see, A LOT of the code is dealing with the radial transforms, etc.   The interesting bit for this discussion doesn’t happen until you’re almost at the end of the function. (Hint: Look for the Draw Markers comment.) I’m appending a Marker Seg() to the Graph Elements stack. If you right-click on the radial plot in the finished report and select Customize, you can see it in there: The reason that Marker Seg() is so powerful is that it has that Row States() reference in it. Remember, everything in JMP is linked through the data table via row states. So, by telling JMP which data table the report needs to communicate with, I can sync up all the charts in the report and have all the interactivity I’m used to with JMP. And it's just by going from Markers to Marker Segments (which blew my mind when I got it working).   The spider plot OK, we’ve got nothing else that I can talk about except that dang spider plot. So, let’s get into it. Since a spider plot is basically a parallel plot wrapped around central axis, we’re more or less going to have to make a parallel plot – with all the points projected onto the polar coordinate system. And, if that sounds painful, you’re right. It is. It’s also really, really hard for your brain to process radial information. So, that one change – from Cartesian to polar – makes spider charts significantly harder to read. Anyway, I’ll save the discourse on the merits of this graph for another time. On to the code! The good news is that since I’m really only dealing with one dimension per variable, I have full control over the angular part of the coordinate system. So, there’s some transformation work there, but it’s not that bad. It gets messy when you need to consider that the user might want a wider scale displayed than the data actually covers, e.g., the values are between 3 and 5 but the possible scale is 1 to 7. I was able to handle that by looking for an axis column property and then getting the max and min values from there. Since that’s not a critical point in the narrative, you can see how that was done in the source code I’ve included with the add-in. The radial spokes and reference lines are just lines drawn using the Line() command. The hardest part of this whole thing was making the webs for each row interactive and getting the legend to work. I made the lines interactive by employing a Line Seg() and a Marker Seg(). The legend was a repurposing of a really old piece of JMP I ran into recently called a Row Legend. Let’s look at each of them. Here’s the code for creating the lines on the spider chart. As with the radial chart, it takes a lot of data manipulation just to get it into a format that makes sense to graph:   drawTrails = Function( {nP, pCol, obj},             {default Local},                         // Create a list of angles for the spider plot             th = ((1 :: nP) - 1);             th = Shape( th, nP, 1 );             th = 2 * Pi() * th / nP;                         //Get the data from the table as a matrix.             sel_dat = J( N Row( DataTable1 ), N Items( pCol ), 0 );             For( i = 1, i <= N Items( pCol ), i++,                         sel_dat[0, i] = pCol[i] << get as matrix             );                         // now get a matrix with two columns and nvars rows to keep track of the min and max of each variable             min_max = J( nP, 2, 0 );                   // column1: minimum, column2: maximum. row for each variable chosen             // Check if the user wants to use axis column property values for max and min.             If( axisProp == 0,             // Directly calculate the values from the data table.                          For( i = 1, i <= nP, i++,                                     min_max[i, 1] = Min( sel_dat[0, i] );                                     min_max[i, 2] = Max( sel_dat[0, i] );                         )             ,             // If the user wants to use axis column property values                         For( i = 1, i <= nP, i++,                                     // Pull the column name and check if the axis property is defined                                     col = pCol[i];                                     axisFlag = Contains( col << Get Properties List, Expr( Axis ) );                                     //If the axis property is present pull the values from the property                                     If( axisFlag == 1,                                                 axisPresent = col << Get Property( "Axis" );                                                 min_max[i, 1] = Eval( Extract Expr( axisPresent, Min( Wild() ) ) );                                                 min_max[i, 2] = Eval( Extract Expr( axisPresent, Max( Wild() ) ) );                                     ,                                     // if the property is not present, calculate the values directly from the data                                                 min_max[i, 1] = Min( sel_dat[0, i] );                                                 min_max[i, 2] = Max( sel_dat[0, i] );                                     );                         )             );                         // Create some empty matices for the converted values             xMat = [];             yMat = [];                         // Convert the data matrix             For( i = 1, i <= nR, i++,                         current = sel_dat[i, 0];                                                 // for each variable, subtract the minimum and then divide by the range.                         adj = J( N Row( current ), 1, 0.0 ) + 1.0 * ((current - min_max[0, 1]`) (min_max[0, 2] - min_max[0, 1])`);                         adj = adj`;                         // an error check form missing values in the vector                         For( k = 1, k <= N Rows( adj ), k++,                                     If( Is Missing( adj[k] ),                                                 adj[k] = 0                                     )                         );                         {x, y} = polarToRect( adj, th );                         xMat = xMat |/ x;                         yMat = yMat |/ y;             );                         // Draw Lines and Markers for spider plot             annotateSpider = Expr(                         // coordinate matrices                         xMat = xxx;                         yMat = yyy;                         nParam = nnn;                         nPoints = N Rows( xMat );                         nGroups = nPoints / nParam;                                                                         // Reshape the matrices to make them easier to work with                         xSMat = Shape( xMat, nGroups );                         ySMat = Shape( yMat, nGroups );                                                 // Create an empty string to hold a list (as a string)                         pathList = "";                                                             // Draw the lines as polygons using the reshaped matrix                         For( i = 1, i <= nGroups, i++,                                                                                     // Get the coordinates for a given path and add the first value to the end to close the path                                     xCoord = xSMat[i, 0];                                     xCoord = xCoord || xCoord[1];                                     yCoord = ySMat[i, 0];                                     yCoord = yCoord || yCoord[1];                                                                         //                                     obj[FrameBox( 1 )] << Append Seg( Line Seg( xCoord, yCoord, Row States( DataTable1, {i} ) ) );                                                                         );                                                                         // Draw the markers                         For( i = 1, i <= nP, i++,                                                                                     // Get the coordinates for a given path                                     xCoord = xSMat[0, i];                                     yCoord = ySMat[0, i];                                                                         // Run Markers Seg (need to move and reformat the matrix)                                     obj[FrameBox( 1 )] << Append Seg( Marker Seg( xCoord, yCoord, Row States( DataTable1 ) ) );                         );             );                         // Substitute values into the annotation expression             Substitute Into( annotateSpider, Expr( xxx ), xMat, Expr( yyy ), yMat, Expr( nnn ), nP );                         // Run the annotation expression             annotateSpider; ); The important bits are the Line and Marker segments toward the end. I had to draw each connecting line individually. There is a Path Seg() that would have made this really easy, but it creates a filled polygon if you try to color it. So, Lines and Markers it was! By using Graphic Segments, it's really easy to hook back to the data table through the Row States operator. Now, because all the graphs are looking at the row states from the main data table, it becomes possible to color them all simultaneously by assigning colors to each Row State in the data table.   When you right-click on some of the graphs in JMP, there will be the Row Legend option. It’s been in JMP for a while and does two things. First, it colors the rows in the data table by column (like the Red Triangle Menu option in the data table does). Second, it creates a little legend next to the visualization. That’s all I had to do to get the colors in sync with one another across four graphs. Here’s the code: drawLegend = Function( {gVar, obj},             {default Local},             // extract the column name with the grouping variable             colName = gVar[1];                         // pass the variables into the expression             annotateLegend = Expr(                         obj[FrameBox( 1 )] << Row Legend( ggg, Color( 1 ), Continuous Scale( 0 ) )             );                         // substitute into the expression             Substitute Into( annotateLegend, Expr( ggg ), colName );                         // run the expression             annotateLegend; ); Since all my graphs are looking at the same data table for row states, they all automatically inherit the coloring the Row Legend assigns! Super slick.   Wrapping things up And that’s it. For scripters, the main thing I’d like you to get out of this is the power of the Graphics Segments. A lot of the bits that make JMP so special are wrapped up in what those functions can do. For everyone else, there’s now a spider plot add-in. I’m pretty proud of how it turned out. But, please don’t use it. There are much better ways of visualizing that type of data. 😉 Author's note No spiders were harmed in the writing of this blog, although some may have taken umbrage at being lumped in with insects (six-legs) or ticks (also arachnids, but they’re vampire spiders, so I’d like to think that even spiders hate them). ... View more

In March of this year (the dumpster fire that is 2020), my world (like pretty much everyone else's) went bananas. Full-blown, off-the-rails bananas. I live in upstate NY and, because of the pandemic, my state collectively decided to have a state-mandated three-month staycation. I’ve never seen anything like it. And for an extrovert like me, it was tricky to navigate. Since I live in a house full of introverts, I had to find some ways to deal with my situation while giving them their space. I turned to exercise. I’m normally a pretty active person (even if my pre-lockdown waistline disagreed). I love working out and strength training. My gym went virtual during the lockdown so that we could keep in shape…or at least not get out of shape. At the same time, since my customers are all in NY and NJ (thus also subjected to staycations), I wasn’t traveling   ̶   at all. Which really annoyed me, since it torpedoed my plan to go on a pastrami bender at Katz Deli for my 40 th birthday. However, the lockdown also meant that I could get in a workout every day. I needed something to fill the hours and it was either cleaning the basement or burpees. My daughter also came down and joined me in our classes, which gave my wife a bit of a breather. (Alas, since the workout space I was "assigned" is in the basement, I did end up cleaning that anyway.) The staff at the gym also took the shutdown as an opportunity to help us improve our overall health. They helped us examine our health and wellness habits, tracking how much we slept, daily activity, and food intake. Now, me being me, I looked at the spreadsheet they provided us for this data and said, "I bet I could put this into JMP and get some interesting insights." And, me being me, I also have access to JMP and JMP Public, so I decided to tinker with some control charting and data visualizations. Let’s have a look! Weight and Body Composition My top line metrics (KPIs, if you like) were my weight and body fat composition. The weight bit was really easy. I have a good scale that measures weight to +/-0.2 lb. It also measured other body composition values like Percent Body Fat. I was, however, really skeptical of the absolute measurement for body fat composition. There’s a lot of data about impedance-based measurements being a little wonky, so I was more interested in the trend than the absolute numbers it provided. Here’s my top-line dashboard (created using Graph Builder, Dashboard Builder and JMP Public 😞 😞 I have different phases in my charts because of the different goals in each phase, Phase 1 was for establishing baseline values in my diet and learning how to get my diet dialed in for Phase 2. Phase 2 was about getting down to roughly 205 lbs. by the end of the 12 weeks. I averaged around 1.5 lbs. per week pretty consistently, despite some day-to-day variations. As I write this, Phase 3 has just begun; it has a slightly more aggressive goal of reaching ~180 lbs. by the end of November. Right now, I’m averaging about 2 lbs. per week. I don’t have a huge amount of data yet, so that rate might change as things settle in. As you can see, the body fat numbers do trend down. But they are really noisy. This is consistent with my research. Lots of factors can impact body composition based on impedance measurements, basically anything that changes how electricity would pass through my body would impact this measurement. So again, a statistically significant downward trend is what I was looking for; the rest is pretty much meaningless. Nutritional Control Charts After grad school, I went into manufacturing and learned about control charts, which is probably the reason I looked at this data in JMP. The spreadsheet my coach provided me had color coding for when my nutrition values were outside the range he wanted me in. When he explained his spreadsheet to me, I said to myself, "That’s a set of spec limits! I can do that in JMP!" And, yes, I really did say that to myself. And, yes, I immediately reverse-engineered the spreadsheet and ran it through Process Screening. Here are the results, again with the phases: The phases that I mentioned earlier are more apparent here. You can see things are all over the place in Week 1 of Phase 1, when I’m just getting in the habit of logging things and getting the process together. You might call this Early Stage Process Development, which is always noisy. Week 2 of Phase 1 is where I’m dialing things in (Process Targeting). And then we transfer into Phase 2 where everything is stable, and we just run the process. Phase 3 was a retargeting of the process, which included changing the spec limits for my nutrient categories. The dashboards you see above have the Phase 3 targets and acceptable ranges on them. (Side note: Strictly speaking, I shouldn't have Spec Limits and Control Limits on the same chart.  It's a bad habit I picked up somewhere - don't get into it.  If you want to know why have a look at our STIPS statistics course.) Forecasting and Goals Because I’m pretty consistent with everything in this program, I can do a pretty good job of forecasting the weight-loss trend overall. (See below.) I did this by modeling the weight loss by day as a function of phase, which I can plot in Graph Builder. More importantly, I built a nested regression in Fit Model that allowed me to build a model on the entire data set taking into account each phase of the journey. Why Does All This Matter? That’s probably the question that you’re asking. Why would I want to model and control chart my weight-loss journey? Well, for one, the graphs are pretty cool. We are visual creatures, and a graph of the journey is much more engaging than a table of numbers or a spreadsheet. The other reason is more pragmatic. Weight fluctuates. I don’t eat the exact same thing every day. I don’t exercise exactly the same way every day. All kinds of things impact a person’s health data on a given day or even time of day. The point with modeling the data is that I get a better sense of a normal range for me. Since modeling the data gives me prediction intervals and confidence intervals for fits, I can also check to see what might be normal noise and what might be something I need to address with my coach. So, all together, I did this for two reasons: 1) data visualization is more impactful and 2) modeling gives me a better way to know if something is outside of expected behavior. Wrapping Up And that’s about it. I’ve still got a way to go before I hit my target weight. But the numbers look good, as do my trends. JMP made this analysis really easy (even pulling in the data from the Google Sheet my gym used). I showed the numbers to my coach, and he found it to be an interesting way of approaching wellness. ... View more

This new capability has me breaking out my thesaurus, scrambling for new superlatives. So yeah, so much for being done with the series… Normally, when I write a series like this, I write the whole thing – start to finish – before I start publishing anything. Normally, I don’t want to leave my readers hanging while I write the next entry. Normally, I try to avoid the stress of writing under a deadline. Alas, there is nothing normal about this entry. This entry is being written as the other entries are being published. I normally wouldn’t attempt something like this, but the information that I’m going to share here is not just salient to the discussion, it completely changes how I’m going to approach this topic going forward. As a refresher, here are the links to the other entries: The 1st thing I wish I knew when I started scripting: There are different kinds of scripts The 2nd thing I wish I knew when I started scripting: Getting help The 3rd thing I wish I knew when I started scripting: The Answer Button is your friend The 4th thing I wish I knew when I started scripting: What to do when things go wrong The 5th thing I wish I knew when I started scripting: How to string things together The (other) 5th thing I wish I knew when I started scripting: How to package things up And now, let’s turn nearly everything I just finished telling you over the past weeks on its ear. There is a cycle to how JMP releases work. Usually, the Product Management team doesn’t start discussing the new release with the general public until about two or three months before the release date. So, you can imagine my shock when I was given the green light to write about this new capability – a full six months before JMP 16 is slated for release. As I sat in a meeting where the developers demonstrated this new capability to JMP staff, I shared my excitement about this feature and my eagerness to show it to customers. Imagine my surprise when, later that day, I was asked to write an article about it. So, what is this magical feature that has my heart all aflutter? What has me breaking out my thesaurus, scrambling for new superlatives? What is the new feature that’s going to completely alter how I introduce people to the world of JMP scripting? Well…and, stay with me here…it’s actually the log. A preview of things to come JMP 16 will introduce a new log, currently called the Enhanced Log. But, that’s a bit like saying a smartphone is a thing to make phone calls (you do remember they do that, right?). The Enhanced Log tracks a fair bit of what you do when you interact with JMP and (here’s the awesome part) will generate JSL for it with a mouse click! Let that sink in for a second. This goes way beyond Source scripts and the red triangle Save Script options. They’re not trying to boil the ocean, so some stuff isn’t going to be captured in the first release; data import and data table manipulation are the primary goals for JMP 16. But it covers a lot of ground right out of the gate. Let me show you what this thing is already able to do (with six months of development time to go). Turning on the Enhanced Log First things first, turning on the Enhanced Log is as easy as going into the preferences in JMP (under JMP > Preference for MacOS; File > Preferences for Windows) and changing the Log Style drop-down to “Enhanced.”   Click OK and you’re off to the races! (note the screenshot below is from an early release and the log now has its own menu item toward the bottom of the list.)     Now open the log (Window > Log for MacOS; View > Log for Windows). You’ll see something that looks a little different than you’re used to. I’ve cleared mine out in the screenshot, just so you can see the different regions. The top part is where actions will be recorded, which I’ll cover in a moment. The bottom part is where the JMP generated JSL will appear. How that fits into our story I’ve put everything into a JMP Project, just to keep everything organized. When the Enhanced Log goes into a project, it’s automatically put at the bottom and goes into a side-by-side configuration (Enhanced Log on the left; the JSL output on the right). Since this isn’t meant to be a blow-by-blow discussion of how I did things, I’m just going to go through a few steps and point out some things. But I’m going to use the same example we used back in my third post. If you’ve been following along, have a look back and compare how easy this is now. First, let’s get some data into JMP. I opened Big Class Families.jmp. The Enhanced Log area has an entry that shows the action was a user click (under Type), what it did, where the action came from in JMP (under Origin), and what happened. It also has a nice time stamp. Now have a look at the JSL area. There’s an Open(…) command (properly commented!) ready for you to use! No diving through the Scripting Index required! Let’s continue with our analysis. I made a Distribution with the Height, Weight, Sex, and Age columns. In the Enhanced Log, there is now an entry showing that a user command started with a data table and resulted in a Distribution Platform window. In the JSL area, I’ve got the properly formatted (and commented!!!) entry to reproduce the Distribution platform. Now, let’s do a Fit Y by X of Weight vs. Height. The same story – there’s an entry in the Enhanced Log and JSL areas. But, that’s not generally why we open Fit Y by X, is it? We want to do line fitting or hypothesis testing. If we do that, there isn’t an update to the Enhanced Log or any new JSL code. We appear to have hit a roadblock. But wait! We haven’t, have we? We haven’t followed the JMP Workflow yet! So (following the workflow) the question is, how do we get the Enhanced Log to capture the changes we made to Fit Y by X by fitting a line? As always, the answer is found by PUSHING THE ANSWER BUTTON! For those who missed my first blog series, I call the little red triangle the Answer Button.) Since we’re in a JSL blog series, we’re going to the Save Script To… part of the little red triangle menu, where we find a new entry, To Log. Clicking that puts a new pair of entries in the Enhanced Log for Fit Y by X with the fitted line options on. So. Freaking. Cool. Side note: Did you notice that it’s also handling the glue operators (;)? Now, the last bit. How do we get that JSL out of the Enhanced Log? Well, I just asked a question, so let’s push the answer button for the Enhanced Log. And, there we find a Save Script > To Script Window option. And we’re done! I could delete the extra Bivariate call (the first one, without the line) and put the two report windows into a New Window() function, but the point here is that I got 80%, maybe 90% of the way to a completed script without writing a lick of code. IT EVEN COMMENTS FOR YOU!!! (I’m totally geeking out at the moment…hold on a sec…I need to take a breath.) Here's the finished code, if you want to play with it (Note: I did change the file path so that it would work on other versions of JMP): // Open Data Table: Big Class Families.jmp Open( "$SAMPLE_DATA/Big Class Families.jmp" ); // Launch platform: Distribution Distribution(             Continuous Distribution( Column( :height ) ),             Continuous Distribution( Column( :weight ) ),             Nominal Distribution( Column( :sex ) ),             Nominal Distribution( Column( :age ) ) ); // Launch platform: Bivariate Bivariate( Y( :height ), X( :weight ) ); Bivariate( Y( :height ), X( :weight ), Fit Line( {Line Color( {230, 159, 0} )} ) ) Where to go from here…again Here we are at the end of the blog series…again. I promise this time is for real. So, what’s your homework for this entry? Well, you might be tempted to just sit on your hands and wait for Enhanced Log to arrive. I know it’s tempting, but I recommend that you practice what we’ve gone over in the earlier episodes. It’s important stuff, and you should use what you’ve learned. Good luck and happy coding! ... View more

Level: Intermediate   Mike Anderson, SAS Institute, SAS Institute Anna Morris, Lead Environmental Educator, Vermont Institute of Natural Science Bren Lundborg, Wildlife Keeper, Vermont Institute of Natural Science   Since 1994, the Vermont Institute of Natural Science’s (VINS) Center for Wild Bird Rehabilitation (CWBR), has been working to rehabilitate native wild birds in the northeastern United States. One of the most common raptor patients CWBR treats is the Barred Owl. Barred Owls are fairly ubiquitous east of the Rocky Mountains. Their call is the familiar “Who cooks for you, who cooks for you all.” They have adapted swiftly to living alongside people and, because of this, are commonly presented to CWBR for treatment. As part of a collaboration with SAS, technical staff from JMP and VINS have been analyzing the admission records from the rehabilitation center. Recently we have used a combination of Functional Data Analysis, Bootstrap Forest Modeling, and other techniques to explore how climate and weather patterns can affect the number of Barred Owls that arrive at VINS for treatment — specifically for malnutrition and related ailments. We found that a combination of temperature and precipitation patterns results in an increase in undernourished Barred Owls being presented for treatment. This session will discuss our findings, how we developed them, and potential implications in the broader context of climate change in the Northeastern United States.       Auto-generated transcript...   Speaker Transcript Mike Anderson Welcome, everyone, and thank you for joining us. My name is Anna Morris and I'm the lead environmental Educator at the Vermont Institute of Natural Science or VINS in Quechee, Vermont. I'm Bren Lundborg, wildlife keeper at VINS center for wildlife rehabilitation and I'm Mike Anderson JMP systems engineer at SAS. We're excited to present to you today our work on the effects of local weather patterns on the malnutrition and death rates of wild barred owls in Vermont. This study represents 18 years of data collected on wild owls presented for care at one avian rehabilitation clinics and unique collaboration between our organization and the volunteer efforts of Mike Anderson at JMP. Let's first get to know our study species, the barred owl, with the help of a non releasable rehabilitative bird serving as an education ambassador at the VINS nature center. Yep, this owl was presented for rehabilitation in Troy, New Hampshire in 2013 and suffered eye damage from a car collision from which she was unable to recover. Barred owls like this one are year-round residents of the mixed deciduous forests of New England, subsisting on a diet that includes mammals, birds, reptiles, amphibians, fish and a variety of terrestrial and aquatic invertebrates. However, the prey they consume differs seasonally, with small mammals composing a larger portion of the diet in the winter. Their hunting styles differ in winter as well, due to the presence of snowpack, which can shelter small mammals from predation. Barred owls are known to use the behavior of snow punching or pouncing downward through layers of snow to catch prey detected auditorially. Here's a short video demonstrating this snow punching behavior. I've seen in that quick clip barred owls can be quite tolerant of human altered landscapes, with nearly one quarter of barred owl nests utilizing human structures. There are also the most frequently observed owl species by members of the public in Vermont, according to the citizen science project, iNaturalist, with 468 research grade observations of wild owls logged. As such, barred owls are commonly presented to wildlife rehabilitation clinics by people who discover injured animals. The Vermont Institute of Natural Sciences Center for wild bird rehabilitation or CWBR is the federal and state licensed wildlife rehabilitation facility located in Quechee, Vermont. All wild living avian species that are legal to rehabilitate in the state are submitted as patients to CWBR and we received an average of 405 patients yearly from 2001 to 2019, representing 193 bird species. 90% of patients presented at CWBR come from within 86 kilometers of the facility. Of the patients admitted during the 18 year period of the study, 11% were owls of the order Strix varia, comprising six species, with barred owls being the most common. However, year to year, the number of barred owls received as patients by CWBR has varied widely compared to another commonly received species, the American Robin. Certain years, such as the winter of 2018 to 2019 had been anecdotally considered big barred owl years by CWBR staff and other rehabilitation centers in the Northeastern US for the large number presented as patients. One explanation proposed by local naturalists attributes the big year phenomenon to shifts in weather patterns. When freeze/thaw cycles occur over short time scales, these milder, wetter winters are thought to pose challenges to barred owls relying on snow plunging for prey capture. Specifically the formation of a layer of ice on top of the snow can prevent owls from capturing prey using this snow plunging technique as the owls may not be able to penetrate this ice layer. In order to feed...I lost my place. In order to feed the animals may therefore use alternative hunting locations or styles or suffer from weakness due to malnutrition, which could lead to adverse interactions with humans, resulting in injury. This study was undertaken to determine if a relationship exists between higher than average winter precipitation and the number of barred owls presented during those years at CWBR for rehabilitation. Though there are several possible explanations for the variation in the number of patients associated with regional weather, we sought to determine if there was support for the ice layer hypothesis by further investigating whether barred owls presented during wetter winters exhibited malnutrition as part of the intake diagnosis in greater proportion than in dryer winters. This would suggest that obtaining food was a primary difficulty, leading to the need for rehabilitation, rather than a general population increase, which would likely lead to a proportional increase in all intake categories. Initially we expected that there would be a fairly simple time series analysis relationship to this. We went and looked at the original data for the admissions and just to compare as, as Bren said, just to compare the data between the barred owls and the American robins, you can see for bad years, which I've marked here in blue, except for the the gray one which is actually had a hurricane involved, we can see there's a very strong periodic signal associated with the robins. We can see that the year-round resident barred owls should have something resembling a fairly steady intake rate, but we see some significant changes in that year to year. Looking at the contingency analysis, we can see that the green bands, the starvation, correlates fairly nicely with those years where we have big barred owl years. Again, pointing out 2008, 2015, 2019, these being ski season years instead, which I'll make clear in a moment. 2017 doesn't show up, but it does have a big band of unknown trauma and cause, and that was from a difference in how they were triaging the incoming animals that year. The one, the one trick to working with this is that we needed to use functional data analysis to be able to take the year over year trends and turn them into a signal that we can analyze effectively against weather patterns and other data that we were able to find. Looking here, it's fairly easy to see that those years that we would call bad years have a very distinctive dogear...dogear...dogleg type pattern. You can see 2008, 2017, 2019, 2015. Again, Most importantly, those signals tend to correlate most strongly with this first Eigen function in our principal component analysis. You can see quite clearly here that component one does a great job with discriminating between the good years and the bad years with that odd hurricane year right in the middle where it should be. You can also look at the profiler for FPC one and you can see that as we raise and lower that profiler, we see that dogleg pattern become more pronounced. The next question is, is how do we get the data for that kind of an analysis? How do we get the weather data that we think is important? Well, it turns out that there's a great organization that's a ski resort about 20 miles away from here that has been collecting data from as far back as the 50s. And they've also been working with naturalists and conservation efforts, providing their ecological or their environmental data to researchers for different projects, and they gave us access to their database. This is an example of base mountain temperature at Killington, Vermont, and you can see that the the bad years, again colored in blue here, tend to have a flatter belly in their low temperature. You can see for instance, looking at 2007, the first one in the upper left corner, you can see that there's a steep drop down, followed by a steep incline back up. Whereas 2008, which is one of the bad years for for owl admissions, we have a fairly flat, and if not maybe in a slightly inverted peak in the middle. And that's fairly consistent, with the exception of maybe 2015, throughout the other throughout the other the other years. So I took all of that data and used functional data explorer to get the principal components for our responses. We end up having, therefore, a functional component on the response and a functional component on the factors. This is an example of one of those for the...what turns out to be one of the driving factors of this analysis, and you can see it does a very nice job of pulling out the principal components. The one we're going to be interested in in a moment is this Eigenfunction4. It doesn't look like much right now, but it turns out to be quite important. So let's put all this together. I use the combination of generalized regression, along with the autovalidation strategy that was pioneered by Gotwalt and Ramsay a few years ago to build a model of this of the behavior. We can see we get a fairly good actual by predictive plot for that. We get a nice r square around 99% and looking at the reduced model, we see that we have four primary drivers, the cumulative rain that shows up. That makes sense. We can't have rain without...we can't have ice without rain. Also a temperature factor, we need temperature to have a strong...to have ice. But also we have the sum of the daily snowfall or the daily snowfall. That's a max total snowfall per year, and the sum of the daily...the daily rainfall as well. And taking all of this, we can put together start to put together a picture of what bad barred owl years look like from a data driven standpoint. We can see fairly clearly. I'm going to show you first again what a bad barred...what a bad year looks like from the standpoint of the of the of the the admission rates. And we can see here. Let me show you what a bad, bad year looks like. That's a bad year; that's a good year, fairly dramatic difference. Now we're going to have to pay fairly close attention to the... We're gonna have to pay fairly close attention to the other factors to see because it's a very subtle change in the the temperatures, in the rain falls that trigger this good year/bad year. It's it's kind of interesting how how tiny the effects are. So first, this is the total snowfall per year. And we're going to pay attention to the slope of this curve for a good year and then for a bad year. Fairly tiny change, year over year. So it's a it's a subtle change, but that subtle change is one of the big drivers. We need to have a certain amount of snowfall present in order to facilitate the snow diving. The other thing, if we look at rain, we're going to look at the belly of this rainfall right here, around, around week 13 in the in the ski season. There's a good year. And there's a bad year. Slightly more rain earlier in the year, and with a flatter profile going into spring. And again, looking at the cumulative rain over the season, a good year tends to be a little bit drier than a bad year. And lastly, most importantly, the temperature. This one is actually fairly...this is that belly effect that we were seeing before. We see in early years or in good years that we have that strong decline down and strong climb out in the temperature, but for bad years we get just slightly more bowlshaped effect overall. And I'm going to turn it over to Bren to talk about what that means in terms of barred owl malnutrition. Malnutrition has a significant negative impact upon survival of both free ranging owls and those receiving treatment at a rehabilitation facility. Detrimental effects include reduced hunting success, lessened ability to compete with other animals or predator species for food, and reduced immunocompetence. Some emaciated birds are found too weak to fly and are at high risk for complications such as refeeding syndrome during care. For birds in care, the stress of captivity, as well as healing from injuries such as fractures and traumatic brain injuries can double the caloric needs of the patient, thus putting further metabolic stress on an already malnourished bird. Additionally, scarcity or unavailability of food may push owls closer to human populated areas, leading to increased risk for human related causes of mortality. Vehicle strikes are the most common cause of intakes for barred owls in all years and hunting near roads and human occupied habitats increases that risk. In the winter of 2018 to 2019, reports of barred owls hunting at bird feeders and stalking domestic animals, such as poultry, were common. Hunting at bird feeders potentially increases exposure to pathogens, as they are common sites of disease transmission, it may lead to higher rates of infectious diseases such as salmonellosis and trichomoniasis. Difficult winters also provide extra challenges for first year barred owls. Clutching barred owls are highly dependent on their parents and will remain with them long after being able to fly and hunt. And once parental support ends, they are still relatively inexperienced hunters facing less prey availability and harsher conditions in their first winter. Additionally, the lack of established territories may lead them to be more likely to hunt near humans, predisposing them to risks such as vehicle collision related injuries. Previous research on a close relative of the barred owl, the northern spotted owl of the Pacific Northwest, shows a decline in northern spotted owl in fecundity and survival associated with cold, wet weather in winter and early spring. In Vermont, the National Oceanic and Atmospheric Administration has projected an increase in winter precipitation of up to 15% by the middle of the 21st century, which may have specific impacts on populations of barred owls and their prey sources. The findings of this study provide important implications for the management of barred owl populations and those of related species in the wake of a change in climate. Predicted changes to regional weather patterns in Vermont and New England forecast that cases of malnourished barred owls will only increase in frequency over the next 20 to 30 years as we continue to see unusually wet winters. Barred owls, currently listed by the International Union for Conservation of Nature as a species of least concern with a population trend that is increasing, will likely not find themselves threatened with extinction rapidly. However, ignoring this clear threat to local populations may cascade through the species at large and exacerbate the effects of other conservation concerns, such as accidental poisoning and nest site loss. These findings also highlight the need for protocols to be established on the part of wildlife rehabilitators and veterinarians for the treatment of severe malnourishment in barred owls, such as to avoid refeeding syndrome, and provide the right balance of nutrients for recovery from an often lethal condition. Rehabilitation clinics would benefit from a pooling of knowledge and resources to combat this growing issue. Finally, this study shows yet another way in which climate change is currently affecting the health of wildlife species around us. Individual and community efforts to reduce human impacts on the climate will not be sufficient to reduce greenhouse gas emissions at the scale necessary to halt or reverse the damage that has been done. Action on the part of governments and large corporations must be taken, and individuals and communities have the responsibility to continue to demand that action. We would like to thank the staff and volunteers at the Vermont Institute of Natural Science, as well as at JMP, who helped collect and analyze the data presented here, especially Gray O'Tool. We'd also like to thank the Killington Ski Resort for providing us with the detailed weather data. Thank you.   ... View more

You've created a script. Now how can you share what you’ve created? Welcome back! If you’ve made it this far, you’ve probably got a working script of some sort. Congrats! If you’ve just discovered this series, you might want to put a pin in this article and start from the beginning. Here are links to the ones you may have missed: The 1st thing I wish I knew when I started scripting: There are different kinds of scripts The 2nd thing I wish I knew when I started scripting: Getting help The 3rd thing I wish I knew when I started scripting: The Answer Button is your friend The 4th thing I wish I knew when I started scripting: What to do when things go wrong The 5th thing I wish I knew when I started scripting: How to string things together So, you’ve got a nice new bit of code. What are your plans for it? If you’ve gone to the trouble of creating a script, you should probably consider how to package it for others (or yourself) to easily use. In this last entry, I want to give you some ideas about how to share what you’ve created. JMP has several options depending on how you want to share it. Sharing your work There are four or five methods (depending on who you talk to) for packaging scripts for others to consume: JSL files, Query Builder files, JMP add-ins, JMP applications, and JMP Live and/or JMP Public reports. For the level of scripting we’re doing here, some of these methods are a bit out of scope (particularly applications; we’ll save those for another series). What I’d like to do here is give you some options and good resources for each method and work through some examples for the ones that will give you the most bang for the buck in the current phase of your scripting journey. JSL files This is what people think of when they talk about working in JSL. If you’ve been following along in the series, this is where your script currently lives. The thing is, JSL files aren’t the best way to store or share files. They can be accidentally edited. They require someone to know how to run a script. And, to be honest, for some people, seeing the code is a bit intimidating. JSL files should probably only be used for the most trivial of tasks or for prototyping methods for a larger project. There are exceptions, of course. If you’re creating a library of functions or something, it’s going to exist as a JSL file. There are also bits of JMP that let you provide JSL directly (for example, the code folding keywords file), but these tend to be the exception rather than the rule. My personal bias is to stop at a JSL file only as a last resort, if I’m collaborating with another coder on something or if the code is entirely for my personal use (e.g., a snippet for testing something).   Query Builder files JMP has Query Builder support for most file types through ODBC drivers or JMP data tables. If your script involves a data manipulation step, chances are that you could probably offload a bunch of the heavy lifting to the Query Builder file and use the post-query script (more on that in a second) to do the bits that Query Builder can’t handle. Using these files as a packaging method simplifies the workflow and makes the script more robust. They also make the script portable. Plus, you gain the ability to use the built-in filter prompts to make your code even more flexible. Let’s look at a simple example. I’m going to use the JMP data table Query Builder under Tables > Query Builder rather than the one for ODBC or SAS connections just to keep things simple, but the workflow would be similar for those cases, and there are resources to show how those work. First, let’s set up a Table Query. I’m not going to write that out. The JMP documentation team did a great job with an example for the SATByYear sample data. Just follow the instructions in this example and stop after you run the query, but keep the Query Builder open. Once you have the data table, create a Graph Builder map: Open Graph Builder (Graph > Graph Builder). Drag State to the Map Shape drop zone below the Y-axis. Drag % Taking (2004) into the Color drop zone. Go to the red triangle next to Graph Builder and select Redo > Column Switcher. Select % Taking (2004) as what you want to switch out and click OK. Select all the continuous variables as what you want to switch between and click OK. Select Done to close the control panel. Now, that’s a lot of clicking around for a graph and a very common view for this kind of data. Using our new scripting skills, we can have Query Builder reproduce this graph (and any additional ones we may choose to add later). Here’s how you do it: Select Save Script > To Clipboard from the Graph Builder. In the Query Builder select the Post-Query Script tab in the bottom half of the window. Paste (CTRL+P for PC or CMD+P for MacOS) the script into the Post-Query Script tab. Save the Query Builder file by clicking Save. Now, close the data table Query Builder created and Graph Builder. Click Run Query. See how that works? By using Query Builder, you got some interactivity for free in the form of the filtering dialog and were able to significantly increase the efficiency of this workflow. If someone had the two data files, you could send the Query Builder file to them and it would produce the same result. JMP add-ins If you want to make your code show up in JMP, an add-in is the way to go. Add-ins show up in the Add-in menu in JMP, which will only appear after you’ve installed your first add-in. They are stored in the JMP file system, so the user doesn’t have to remember where a script file is. And they provide a certain element of polish to a completed project. Admittedly, they can be a skosh intimidating at first, but they’re really easy to work with using the Add-in Wizard. I’m going to create a really simple add-in that uses this code:   // Set up the namespace Names Default to here(1); // open the log window and clear it Open Log(); Clear Log(); // write my message Write(“Hello World!”); The code will open the log and print a message. We’re going to call this add-in “My First Add-in.” The first step is to open an Add-in Builder (File > New > New Add-in). After that, it’s just a matter of filling in the form. There are five tabs along the top of the interface. We’re just going to need the first two for our add-in. The first tab (General Info) is more about bookkeeping and getting JMP to know where things are. The second tab actually sets up the bits that are in the JMP GUI. Under General Info, the Add-in Name and Version fields are pretty self-explanatory. You can limit legacy versions of JMP from running your code with the Minimum JMP Version. You can also limit your add-in to run on MacOS or Windows using the Host. I recommend setting the Minimum Version and Host to match the configuration of the system that the code was written on (unless you have the ability to test on other configurations). It’s easier than trying to make sure your code is bulletproof for all situations, particularly when you’re just starting out. The Add-in ID is a unique folder name that is created in the JMP file system when the add-in is installed. JMP recommends using the “Reverse DNS” naming convention (no idea why). It basically means you create something that looks like a website address and reverse it. For the add-in, we’ll use com.myAddin.MyFirstAddin. Here’s what the first tab should look like: Under the Menu Items tab, you just need to click on the Add Command button. Then under Details, give the menu item a name. If you want to be fancy, you can give it a tool tip to explain what your add-in does when you hover over it. Then you just copy and paste your JSL into the Action area. If you’re using the code snippet above, your Menu Items tab should look like this: And that’s it. Just save the file, and you’re done! Because the default action (on the General Info tab) is to install the add-in when you save it, clicking Save will both create a file to send around to others and install it on your own instance of JMP. Once you’ve saved the add-in, you should see an entry like this in your add-in’s menu:   JMP applications So, you want to make something that looks like it’s a native part of JMP? Application Builder is, in my opinion, the most flexible and efficient method for generating high-quality, custom JSL solutions. It provides you with the ability to control every aspect of the software, leverage existing parts of JMP, and use custom-designed routines. That flexibility and power does come at a cost in the form of a learning curve. That said, you shouldn’t let it intimidate you. There are lots of resources out there for getting started with Application Builder, and the things that you can do with it are incredible. It’s just going to take some work to get up and running. There’s also a “lite” version of Application Builder built into JMP. You’ll know it as the drag-and-drop tool called Dashboard Builder. Once you’ve got a bit of experience with JSL, taking the Application Builder course from SAS Education would not be a bad way to learn about this very powerful tool. I’m also toying with doing another “Five Things” series on this topic at some point, so keep an eye out for that! JMP Live and JMP Public While JMP Live and JMP Public are heavier on the "consume" end of things, you can create and publish dashboards and reports for others to consume using this method. For the most part, you’re just using the bits of JSL that I’ve shown you in this series. Dan Valente did a great blog post on scripting the JMP Live (and JMP Public) publication process, so I’m not going to belabor the point. Homework Well, you made it! The last homework assignment is really easy. Package up your code, give it to a colleague, and have them try it out and see if it works. If it doesn’t, look at the error messages and figure out why. If it does, then you’ve made your first JMP add-in! Nice work. Where to go from here Like I said earlier, this series was not meant to blow your hair back and dazzle you with my scripting prowess. It was meant to point your nose in the right direction, give you a good map, a pat on the back, and some help to reach out to in case you get stuck. It’s a starting point; there’s still a lot more road ahead of you than there is behind you. So, from here, the trick is to keep trying and experimenting. Keep taking people’s add-ins and applications apart to see how they work. Keep trying new things and looking for opportunities to make your workflow more efficient with little scripts. The more you do that, the better you’ll get and the more approachable the bigger projects you’re thinking about will seem. Just keep at it. ... View more

Ready to learn about stringing everything together into an actual script? Welcome back to my series on the basics of scripting in JMP. This is the fifth entry in a series that is structured for you to work along. If you’re just joining us, you’re really going to want to have a look at the earlier entries in the series, because this one’s a doozy. Here are the links to them for you to review: The 1st thing I wish I knew when I started scripting: There are different kinds of scripts The 2nd thing I wish I knew when I started scripting: Getting help The 3rd thing I wish I knew when I started scripting: The Answer Button is your friend The 4th thing I wish I knew when I started scripting: What to do when things go wrong All right? Still here? Did you look at the earlier episodes? Excellent! Onward to glory! From the homework a couple of weeks ago, we should all have a bunch of JSL snippets from using Save Script to Script Window or the Scripting Index to get pieces of code that we need. You did do your homework, right? Good! Let’s get into stringing everything together into an actual script. But first, we gotta have a little chat. One of Mikey's coding pet peeves Okey-dokey, folks. Let me, just for a moment, get up on my soapbox about something. When we write code, we must always comment it! We can’t make any assumptions about the person who will read our code after us, including ourselves. Back in college, I took a Visual Basic coding class. The first thing we learned was how to comment code and write pseudocode. Pseudocode, by the way, are comments that you write before writing code to help you map out what you need to do (they are similar to the punch list I had you write in that first post). The instructor called commenting and pseudocode “being a good citizen in the coding community.”   We were even graded on how well we commented the code we submitted. It’s that important. The truth of the matter is that you rarely have complete control over who sees your code. Case in point: In this blog series, I have given you step-by-step instructions on how to reverse-engineer someone else’s add-in. I can guarantee that some (if not most) of the people who wrote those add-ins were not expecting others to peek under the hood at their code. I can also guarantee that you’re not going to find as much commenting in the code to help you figure out what’s going on as you’d hope. You need to comment your code to make sure that someone who views it later can understand what you’re doing easily. Yes, commenting is a pain. And, yes, it takes extra time. But, no, you can’t just skip over doing it. It will pay dividends in the long run. And I do practice what I preach here. I’ve got add-ins that I wrote five years ago that I can still understand because I did detailed commenting (and I still remember how tedious it was to do!). So, those of you who already code, make sure you’re commenting your code so that someone else can understand what you’re doing. Those of you who are learning, get into the habit and stay in the habit of heavily commenting your code. Now, let me get down from this soapbox and show you how. Commenting in JSL Let’s open up JMP and open that Big Class Families.jmp data table we’ve been using. If you didn’t save the script example from the last session, go ahead and recreate it. You should have something that looks like this: There are two ways to put comments into your JSL code. The first is to comment out individual lines using a double forward slash ("//"). Anything after the slashes until a carriage return (hitting return on the keyboard) will be ignored. I generally use this method to do my short inline commenting. It’s also useful to turn off single lines of code that you don’t want to run. Let’s have a look at what that means: The second way you can comment is used in cases where you want to turn off large blocks of code or to generate large blocks of comments and text. This type of comment is called a comment block. It’s done using an opening comment flag made of a forward slash and an asterisk ("/*") and a closing comment flag made of an asterisk and a forward slash ("*/"). Here’s an example of how this might be used: Note that I’m not discussing commenting style here. That’s actually up to you. You’ll develop your own style as you go along and may even have different commenting styles for different purposes. My inline comments (like the ones in the examples) are completely different from the commenting I do in something like an application. The point is to write the comments and to be internally consistent in a script with how you do it. The hardest bits in this entire series Really, this entire series has been leading up to these next items. There are three core things you need to know to get started in JSL. The first is commenting (see the sermon earlier in this article). The second and third are how to name things and how to send messages to them. I suppose it’s possible to code up your own ANOVA algorithm (a colleague of mine tells a very entertaining story about that). But, it’s a lot easier to tell a data table to run an ANOVA using Fit Y by X through a message, like we learned in the previous episode. That basic workflow (name something, send a message to it) is a lot of what JSL is about. Yes, you can get into logic and loops, GUI development, etc., but brute force scripting really only requires names, messages, and a little searching the Scripting Index. Naming things Let’s start with naming things. You name things by giving JMP a name following it with an equal sign and then writing the text, number, code, etc., you want that name to represent. The name is called a variable. You can think of variables as containers that you can put stuff into or as nicknames for open reports in JMP. Let’s look at an example using an ANOVA. To create the code for an ANOVA, first run the analysis in Fit Y by X (under Analyze), then turn on the ANOVA test, and then save the code out to a script window. Here’s the report for that using Big Class Families.jmp and the resultant JSL code saved from the red triangle menu (height as Y, sex as X, and selecting Means/Anova/Pooled t under the red triangle) :   This little JSL snippet will produce an ANOVA for any data table that has the same column names as the original data table. (We’re not going to worry about parameterization in this series, though. It’s a bit out of scope.) Now, what if you’ve got a bunch of data tables open and you want to control which data table you want to run that ANOVA on? Well, you can edit the script by giving a name to the data table you want to run the ANOVA on and then sending the JMP-generated code as a message. If you were to look in the Scripting Index (see the article on getting help), you would find a couple of ways to do this. You could either call a data table directly or grab a reference to the data table JMP is currently focused on by using Current Data Table(). There is a third option that’s a little more advanced, where you give a data table a name as you open it. I’ve put examples of all three below (and again note the use of comments to convey information!):   Calling the data table by name will work as long as there is a data table open that has the name you are trying to use. This can be handy if you have a list of data table names to work from. It can also be handy if you’ve hidden or minimized your data table. In either case, you can still send messages to the data table even if you can’t necessarily see it. This method uses the form: <variable name> = Data Table(“<data table name>”); Probably the most useful way to assign a variable to a data table for our purposes is to use the Current Data Table () function. This function grabs whatever is the active or current data table and assigns it to the variable that you’ve indicated. Once the table is assigned to the variable, it doesn’t matter if another data table becomes “current” (e.g., by creating a summary table, you’ll always have a reference to the first data table until you redefine the variable). It has a simpler form: <variable name> = Current Data Table(); The last method is useful if you want to assume that the data table may not be open when someone runs the script. By using this method, JMP will generate a file dialog that allows the user to select the file they want to open. When the user confirms their choice, the file will be opened and assigned the variable name you’ve given it. Again, it’s a simple form:   <variable name> = Open(); Using the Open(…); method can also be handy if you’re comfortable working file paths and directories. You can put the file path inside the parentheses to tell JMP where to look for the file, like this: Also, note that I’m using something called a Path Variable in the Open(…); command to shorten the string. There’s a section on them in the Scripting Guide. Sending messages Now, once you’ve named something, it’s possible to send it a message. It works similarly to sending a text message to someone on your phone. When you send a text, the first thing you need to know is their name, email address or phone number of the recipient, something to identify them. You enter that into the messaging app and then you send the message. Here’s how to do it using the Big Class ANOVA. I’m going to have some fun and have JMP ask me to open a file in this case:   Let’s unpack what we’re seeing here starting at line 5. The first thing that we see is the variable name of the data table we want to give an instruction. Just like when we send a text or give an instruction to a person, we have to identify who we are talking to. Next there are two “less than” symbols (<<). We’ve already seen something like this with the Glue Operator (;). The “<<” symbol is referred to as the Message Operator. It lets the message recipient know that what follows is a message. After that, we see a function called Oneway(…); with the arguments that the function needs to create the ANOVA report. Note: you can get all the information on the options for that function in the documentation or Scripting Index, but you don’t actually have to write any of it out longhand. Simply run the analysis through once and grab the finished script (with all the options) using one of the Save Script to... commands I showed you earlier. Some odds and ends Okay, there are three last details that need to be addressed. The first is the issue of stringing things together. For instance, if you wanted to do an ANOVA and create a Graph Builder in the same script, you’d need to warn JMP that further instructions are to follow. To do this, use that Glue Operator (;) that I keep mentioning. Like a semicolon in written English, the semicolon at the end of a command lets JMP know that there are more instructions to come. Glue is found at the end of an instruction like Bivariate(...);, or Distribution(...); , or giving something a name: stuff = 5 + 6;. As you may recall from earlier posts, JMP includes the semicolon automatically when we use Save Script…To Script Window, but when scripting freehand, this happens to be the most common error I commit during scripting – forgetting to glue things together. The second detail is ensuring your scripts are good citizens in a JMP session. This is done by assigning all the variables you create to a namespace. A namespace is a little sandbox that keeps the variable names that you create from interfering with other parts of the software. The default namespace is called the “Here” namespace. In scripts it looks like this (see line 2): This instruction should always go at the beginning of a script (until you learn when not to use it!) to keep your variable names from colliding with other scripts or parts of JMP.   Before you start pulling out your hair on this point, here's a TL;DR version of what I just said: Just remember to always put "Names Default To Here(1);" at the beginning of your script and you'll be fine for the moment. If you want to learn more about Namespaces, there's a paragraph on the topic in this other post I wrote a while back. The last detail I need to discuss is how to change a column name in a saved script. It’s really simple. Look in your saved script. You should see your column names with a colon (“:”) in front of them. That colon indicates that they are column names (it can mean other things, but for now, it means column names). To change a column name, just change it, but remember to put the colon in front. If your column names contain special characters, you might run into something like :Name(“…”) when JMP outputs the script for you. That’s how JMP handles the fact that your column name has characters that would cause problems if it wasn’t clear that they were part of a string. Even though it’s bad coding form to use :Name(“…”) in our scripts, the software gets to use it because it doesn’t have a choice. You should try to choose better column names.   But you forgot about... OK, I realize some of you are probably getting a little twitchy at this point. You may be thinking, “YOU FORGOT ABOUT FUNCTIONS!!!” or “BUT WHAT ABOUT EXPRESSIONS!!!” or even, “YOU TWIT!!! YOU COMPLETELY GLOSSED OVER LOGIC AND LOOPING!”   First off, it’s not nice to call names. Second, I didn’t forget about any of that. I consciously omitted it. The thing is, this series is not meant to blow your hair back. Like I said toward the beginning of the series, it’s meant to point your nose down the right path, give you a good map, some snacks, a pat on the back, and show you where to look for help in case you get stuck. It’s a starting point. (You have my permission to get a snack at this point, btw.) For those of us who script regularly, a lot of what I’m talking about here is so foundational – so basic – that it’s muscle memory. The thing we’ve forgotten is that, just by using the foundational knowledge I’m covering here, a neophyte scripter can cover a lot of ground. Anyway, if you’re hyperventilating because I didn’t cover anything super fancy, relax. We’re going to go way down the rabbit hole in the next series. For now, let’s get back to the discussion and talk about this week’s homework. Homework All right – here we are. The big homework exercise. This week, you’re going to take the bits of code that you made last week and glue them together. Do this one bit at a time. If your script is going to live outside a data table (as a JSL file), then start with opening your data table and putting it in a variable. After that, regardless of where your script is going to live, add a bit and check that it does what you expect. Then add more and check again that you get the expected result. Remember the Community, help files, etc., are your friends. If you get error messages, look at the log (View > Log for PC and Window > Log for MacOS) and the previous episode. Good luck and see you next week! ... View more

What should you do when things go awry with your scripting? Welcome back! If you’re just joining us, you’ve jumped in right in the middle of a series that aims to get people started with using JMP scripting language. Unlike the previous entries in the series, you probably don’t need to read the others for this one to make sense, but if you want to play along and do the homework from the other entries, here are the links The 1st thing I wish I knew when I started scripting: There are different kinds of scripts The 2nd thing I wish I knew when I started scripting: Getting help The 3rd thing I wish I knew when I started scripting: The Answer Button is your friend Also, a big thanks to Christian Stopp (@christianstopp) for suggesting this entry in the series. This part of the series is very much a standalone entry. But, in the flow of things, this is probably the point where you’ve started to play with some code and may have seen an error message or two. So, before we start with any serious code wrangling, let’s cover what do to when things go sideways. The most common mistakes in JSL Justin Chilton (@Justin_Chilton) and Wendy Murphrey (@Wendy_Murphrey) are among the best scripters at JMP. At the 2020 Americas JMP Discovery Summit, they (virtually) presented a paper titled, “The Most Common JSL Mistakes and How to Avoid Them.” The PowerPoint file for that presentation can be found here on the Community. For the new scripter, this is 14 pages of solid gold. In the context of what we’re doing in this series, some of it is a bit out of scope but will be invaluable for the next steps in your scripting journey. Just do me a solid, download the pdf, put it somewhere that you won’t forget about it, and give the paper a thumbs up on the Community page. Mike’s most common JSL mistakes As a bit of a riff on Justin and Wendy’s work, I want to add a couple of things to their list. These are pretty basic and will generally cause a script to crash before it’s even run (which is why they don’t show up in the Discovery Summit paper). Glue Missing Glue Operators are probably the most common JSL mistake out there. Like I mentioned before, the Glue Operator is the semicolon (“;”) that you see at the end of every instruction in JSL. If you’re saving scripts to a script window using the red triangle menu (see the previous post ), JMP will add the glue where it needs to go. If you’re writing code manually, there’s a trick that you can use to make sure you haven’t forgotten one somewhere.   Right-click in the Script Editor and select Reformat Script .   This trick will actually catch a lot of basic syntax related errors in the script. There is also a keyboard shortcut when you are in the Script Editor (it doesn’t work in “script editor boxes” though). I prefer the keyboard shortcut myself. For Windows, it’s CTRL+M. For MacOS, it’s CMD+M. Now, before I show you how to unpack what Reformat Script tells you when there is an error, let me show the other error that drives me nuts. Since they both use a similar debugging workflow, this will make sense in a second, I promise. Parentheses Getting your parentheses (brackets, braces, etc.) out of place (or missing) is probably the most insidious bug to track down. And I do mean it is a special kind of frustrating. There is one tool that you have available in JMP to help you with tracking down mismatched pairs of parentheses. When you put your cursor on the outside of a parenthesis, the Script Editor will highlight it (and what it thinks its match is) in blue. If the Script Editor is confused, or can’t find a match, it will highlight the parenthesis in red.   So, if the blue match for a parenthesis is not where you expect it to be, or if the parenthesis is highlighted in red, you probably have a problem. When you get errors for missing parentheses, you might be tempted to just keep adding them to the end until the error goes away. Let me just say that’s a bad idea and move on to some better strategies for this issue and for the issue of missing glue. And they both revolve around setting up JMP’s Script Editor. Setting up (and using) the Script Editor for success The JMP Script Editor (we saw it in the previous post) has a number of features that make working with code and debugging easier. I have these on by default in my preferences, and I’d suggest you do the same. To access the JMP Preferences Dialog, select Preferences from either the JMP menu (MacOS) or File menu (PC).   I’ve highlighted the options that I find really useful in the screenshot. The code folding comes in handy for larger scripts. The line numbers and autocomplete options are more important to our current discussion. How to use the log Now that we have our Script Editor set up, let’s learn how to use the log. First, to open the log, go to View > Log (Window > Log on a Mac). This will open a window that looks similar to a Script Editor. It is different from the Script Editor in that it shows the output and results of the commands run in the Script Editor. If you would prefer to keep your input and output in the same window, you can turn on the Embedded Log in the Script Editor by right-clicking in the Script Editor and selecting Show Embedded Log.     Let’s have a look at how it works using this script: The script is pretty simple. Aside from line 2 and the comments, it was created using Save Script…To Script Window. If I run Reformat Script (also found by right-clicking in the Script Window), I get the following error message: In the log, you get something similar: In the gray area, you can see what was input. Below it, you see a message about something unexpected and a suggestion about what could be the problem. In this case, JMP thinks there is missing glue (;). It also tells you where the unexpected input came in the script with a line number. Since JSL is run sequentially (from top to bottom), this means that the error is somewhere above line 5. Because we’ve got the line numbers turned on, we can see that line 2 is missing the Glue Operator, which means JMP isn’t expecting a new command to start at line 5 and is caught by surprise. After adding the semicolon at the end of line 2, running Reformat Script shouldn’t return any errors. Similarly, if we’re missing part of a bracket pair (e.g., parentheses), like this (notice line 7):   we would get this error message: with a log message: These messages from JMP are a little less clear-cut, but here’s how to interpret them. First, from the error message, we know that the problem is in the Distribution. Between the log and the error message, it’s most likely that there is a missing parenthesis somewhere. The log says that the something unexpected happened at line 11 (the “Column …” can sometimes be helpful, but you have to count characters to use it). The color highlighting the parentheses shows that the first one is missing its pair (note the red color on the parenthesis next to Distribution in line 5 after placing the cursor to its left). Work forward using the highlighting to figure out where the missing one should go (line 7 in this case). For the most part, the log is a really useful window into what JMP thinks is going on. However, if you want to check your JSL for errors or work through the code stepwise, there’s another tool for that. How to use the JSL Debugger The JSL Debugger is a bit of an odd beast. If you’ve accidentally hit the debugger button:     you probably already know that it locks up the main interface in JMP while running. When I started scripting, I really didn’t use it a lot. Now that I’ve had a few chats with the development team about it, I can say that I’m coming around to it. Let’s talk a bit about it and what it can do. First, that bit about it locking up JMP. It’s a little confusing, but what it’s actually doing is creating a second instance of JMP to run the debugging. The main instance of JMP isn’t going to be responsive as long as the JSL Debugger is active. If you close the JSL Debugger, you’ll get control back. When you launch the JSL Debugger, in addition to another instance of JMP, you get the JSL Debugger window. This window contains your code, lists of the different variables, a log, and other information. It looks something like this: When you’re debugging, this window is your eyes and ears into what your script is doing in JMP. If you’ve got a second screen, move it over there, so you can always see it (sometimes it can get buried behind other windows). Now, I could go crazy with describing everything that you could do with the JSL Debugger, but the documentation has a great chapter (with examples!) that you can find here. There is also a good blog post by Melanie Drake (@melaniedrake) that covers some of the cool things that you can do with the JSL Debugger. So, if you want to go into the weeds a bit, have a look at those. For our purposes, I want you to pay attention to these buttons: The first two run the script. The first one runs the script to its end or to where it encounters a breakpoint (a point that you want to stop evaluation if you don’t want to run the entire script). You can set a breakpoint in your code by right-clicking next to the line number in the JSL Debugger. A red dot will appear next to that line (the breakpoint needs to be on a line of code, not a comment!).    If you were to run this script using the first Run button, it would run until line 10 and then stop. A second press of the Run button would then finish the script. The second Run button runs the entire script without paying attention to breakpoints. The red Stop Button exits the debugger, which is useful if you opened it by accident or have finished your debugging session. The Repeat button resets everything to run the script again (e.g., if you want to change where a breakpoint is and then rerun the code). Homework The homework this week is pretty light. I want you to get the Script Editor set up, find the log, and start playing with the JSL Debugger using some of the example scripts from the documentation. Have a look at which variables pop up in the tabs on the lower left of the debugger window. Also, find the Log tab in the lower right (so you know where it is). Next week, we’re going to get into the last bit of the “coding” we’re going to cover in this series. ... View more

Make the Answer Button (the red triangle) your friend. Welcome back to my series on the basics of scripting in JMP. This is the third entry in a series that is structured for you to work along. If you’re just joining us, you’ll probably want to have a look at the earlier entries in the series. Here are some links to those: The 1st thing I wish I knew when I started scripting: There are different kinds of scripts The 2nd thing I wish I knew when I started scripting: Getting help Have a look; it will help you with what’s coming up. All right, buckle up. Thus far we haven’t actually worked with any code (unless you count the JSL Scripting Index). That ends today. Today we are going to use the JSL script window. I know that can be anxiety-inducing to some of you. Let’s take some deep breaths and get to it. Don't script for the sake of scripting! Let’s be honest about something: Coding is not an efficient process in the short term. For one-off analytics and graphics, drag-and-drop tools are going to almost always beat out coding in terms of ROI (the cost of someone going through a development exercise is just too great). There are those who insist on scripting regardless of whether a GUI solution is faster. I don’t know why, and I am not in their number. I will consider coding when a workflow is difficult or circuitous enough that using a GUI introduces a significant chance for errors down the road. I will also consider coding when I’m going to do something over and over again and I can justify the time through the efficiency I gain by automation. That’s not to say I don’t code just to see if I can do something. Or that I don’t choose a more convoluted way of coding just to learn something new. But those cases are the exception rather than the rule. Now, I’ve probably tweaked a few people’s noses with that generalization. The problem with this kind of blanket statement is that it’s pretty black and white. The truth, just like everything in life, is that there’s a gray area where the value proposition is a toss-up between a point-and-click solution and coding. The gray area also happens to be where most aspiring coders find themselves. They know they can probably gain some efficiency through scripting, but they just don’t know where to start or they are worried that the time lost to the initial learning effort will be too great. What’s nice about the design of JMP is that it gives us tools that we can use to start introducing JSL into a workflow without losing too much efficiency when manipulating JSL. And, like all things in JMP, it starts with The JMP Workflow. The JMP Workflow (scripting edition) The JMP Workflow generally ends with a user going to a red triangle menu to answer a question. Another function of the red triangle menu is to provide methods for users to save their work (save scripts).   There are three main ways to save scripts: To Data Table: This method saves the scripts to the data table as "Table Scripts."   These scripts save the method used to generate a given report in JMP and are portable (i.e., can be copied and pasted between data tables with the same column names). To Script Window: This method saves the information needed to reproduce a given report to a script window. It is useful for multiple-step operations or producing multiple reports. The contents of a script window can be turned into a table script as well. To Clipboard: This method is useful for moving the code needed for a report into other JMP Platform (e.g., Query Builder, Application Builder, etc.) or for generating prototype code for larger projects. It turns out that really efficient coders will leverage these red triangle options extensively during their coding sessions. They use them to speed up their coding (why pound keys when you don’t need to?) or generate platform prototype code that would otherwise be easy to mess up and difficult to debug. For the beginning coders, these options provide different opportunities to learn about and utilize JSL while easing into the details of working with it. This is where we want to start our coding adventure. Save Script > To Data Table If you happen to be a JMP user in the NY or NJ area: Hi! I’ve probably visited your company at some point. (If not, drop me a PM; we need to fix that.) And, when I’ve come on site, you’ve probably heard me harp on this point:   You MUST save your work, and by “save your work,” I mean use the Save Script…To Data Table option under the red triangle menu. Going back to my first experience with automating a workflow, this is how I started: by saving a bunch of standardized Graph Builder reports to a JMP table that I would regularly update. I cannot stress strongly enough the of power this one little tip. It’s like having Dr. Who’s Sonic Screwdriver. The possibilities are endless. Let’s look at a quick example: Open Help > Sample Data Library > Big Class Families.jmp Open a distribution dialog: Analyze > Distribution Put age, sex, height, and weight into   Y, Columns and click OK.   Under the Distributions red triangle menu, select Save Script > To Data Table   Give it a name (or just use the default). If you use the default in this case, it will show up in the data table as Distribution 2 because there’s already a Distribution script in the data table.   Now look back at the data table in the upper left corner. You should see your table script in the list with a green check mark next to it.  In the context of this series, those table scripts can be thought of as little applets (mini applications) or macros. They are directly portable between data tables (just copy and paste), provided the column names used in the script are the same. They can also be edited (right-click on one and select Edit). So, they are also portable to data tables that don’t have the same column names if you are willing to edit the column names in the script, which is not a big deal. We’ll cover how to do that in the next post. Table scripts can also be used to string smaller scripts to automate a larger workflow. We’ll also cover that in the next post. Save Script > To Clipboard We’re going to cover the last entry in the red triangle menu: to clipboard item. We’re going to cover this one next because the entry is identical to the to Script Window option, but a lot less instructive. That’s not to say it’s not useful, just that you can’t see what’s going on. So, for the sake of time, here’s what it does. It takes the code that is created by Save Script…To Data Table and puts it in the clipboard so that you can put it in another script somewhere. It’s handy. And, as your scripting skills grow, you’ll figure out where it’s useful for you. For the moment, I want to back up into the more interesting and useful option: saving to a script window. Save Script > To Script Window The next entry in the Save Script to… red triangle menu, we are going to cover what is probably the most instructive of the three I’m discussing. Saving a script to a script window does just what it says: It creates a Script Editor window (we’re going to go deeper on the Script Editor in the next episode) and populates it with JMP-generated JSL to recreate whatever was in the window you were working from. The important point with this entry is that you can study the code and learn from what JMP generated. Let’s look at a trivial example using Big Class again. Go ahead and click the green button next to your saved script to rerun the analysis. Then, in the report window, select Save Script > To Script Window from the red triangle menu. Now, look at what JMP has written in the Script Window. JMP has created a Distribution Report: Distribution(…);. JMP has put two Nominal Distribution subgroups into the report: Nominal Distribution(Column(…)) and two Continuous Distribution subgroups: Continuous Distribution(Column(…)). At the end of the script, JMP has put an incredibly important piece of code: the “;”. It’s called the “Glue Operator” and we’ll cover it in more detail in the next episode. For now, you just need to know that it’s a way to signal to JMP when it’s running the code that another instruction is coming. The fact that JMP puts that on the end automatically makes the novice scripter’s life a lot less complicated. The last point that makes this entry so useful is what happens if you leave that script window open. Go ahead and close the Distribution Platform, but leave your Script Editor open. Now, using the open data table, create a Fit Y by X report (under Analyze) using Height as the X variable and Weight as the Y variable. Next, select Save Script to Script Window again. Did you notice how the window was updated with another entry? That script will now recreate the two windows you were just working with, once you know how to run one. Speaking of which… Running scripts So, you’ve got some code in a script window, now what? You probably want to be able to run the code that you’ve created. I mean, you could just leave it up to impress people that walk by your desk, but it’s probably better to run it. There are a few ways to run code in JMP: the main menu, keyboard shortcuts, the tool bars at the top of the scripting window, the Enter key, or just right-clicking. Let’s quickly go through each of these. The main menu and keyboard shortcuts The Run Script command is in the same place in Windows and MacOS. If you’re in a scripting window, go under Edit > Run Script. The hotkey for this is CTRL+R for windows and CMD+R for MacOS. The tool bar button at the top of the scripting window If you look at the top of the script window, you should find a Run Script button (unless you’ve got the tool bars hidden). In Windows, it looks like a little green play button, similar to the ones for table scripts. In MacOS, it looks like a little JSL document with a JMP Man and the green play button. Using the Enter key Note here that I mean the button in the lower right part of the number pad on an extended keyboard. If you push that button, JMP will run the line of JSL where the cursor is located. I used this one a lot before I got comfortable with the JSL Debugger. Right-clicking Selecting some code and right-clicking will bring up a context menu that has a Run Script option toward the bottom. I recommend that you try the different forms of running scripts and find the one that you like best. I’m partial to the hotkey, except when I’m in Application Builder, where I prefer using the tool bar options. Over time, you’ll probably gravitate to one or two that become your favorites. Also, as a side note: All of these options do slightly different things depending on whether you have a section of code selected, your cursor is at the end of a line, or your cursor is on a blank line. So you might want to experiment a bit with that, too. Wrapping up for this week All right, believe it or not, you’ve now got enough information to be insanely dangerous with JSL...in a good way, of course. Let me show you what I mean. Let’s use that script we created today. I’m going to make one little tweak by wrapping a New Window(); function around the distribution and bivariate functions JMP created. You can learn more about New Window(…); in the Scripting Index. Note that the first part of New Window(…); starts on line 1 and the last part is on line 10! I’ve got my Script Editor set up to draw a nice little line connecting them so I can see where they are (more on that later). Then, let’s copy the text for that script and go back to the data table. Select New Script from the red triangle menu in the uppermost left corner of the data table. Paste the script text into the dialog, give it a useful name and click OK. If you’re following along, you should see your report in the script area of the data table. If you run it, you should get something like this: And that’s the little trick I used to turn a two-day analytical marathon into a 15-minute exercise in cutting and pasting. I spent a little time getting the data into a JMP data table that I could easily update. I then exported all the graphs I needed for that report into a single JSL window using Save Script to Script Window. I wrapped them in a new window (so that they were all in one place) and put that script in a Table Script in the data table. Was I working with JSL? Yes. Was I writing lots of code? No. Did I increase my efficiency? Definitely.   Now that we are getting into the actual coding, I need to stress this last point. Coding is about increasing your efficiency: How efficiently you can solve problems. How efficiently you can create reports (so you can get back to solving problems). How efficiently you can work with your data. The code that we are going to create in this series is computer-generated or it’s simple. And that’s OK. It’s about efficient code not sexy code. When I was in college, one of my professors (a fellow musician) had a quote on the wall outside his office: Music that is simple is no better or worse than music that is complex. The same goes with code. I was reminded of this a while back by the person that taught me coding. So, remember: Code that is simple is no better or worse than code that is complex. All that matters is that it meets its intended purpose and increases the efficiency or productivity of those who use it. Homework For this week’s exercise, you’re going to look back at your problem statement. Think through the punch list you created and generate code using the workflows we’ve covered today. The code can be the actual code that you are going to use or prototype code that you think you’ll need to modify a little to answer your problem statement. Try to generate as many of the bits and pieces you think you’ll need using the red triangle menu and save them out as .jsl files. If you get stuck or can’t find a way to get JMP to generate the code you need, reach out for help, either in the Community or in the comments here. Don’t forget to search the Help, and don’t forget that you can search the Community, too! Until next time, good luck and happy coding! ... View more

These are my favorite helps for JSL. Welcome back! Apparently, the esoteric content from my first entry in this series didn’t frighten you off. Great! If you’re just tuning in to the series, I’m discussing the five things I wished someone had told me when I started learning to script in JMP. (In all fairness, I was told a lot of this stuff in the JSL course I took from SAS Education and interactions with the instructors after the fact.) The series is set up for you to follow along with some homework at the end of each episode. So, go back and have a look at the previous entry in the series so you’re not wondering what’s going on or check out the whole series.  Getting Help In my day job, I talk to people about JMP. All day long. I talk to people that have never seen the software before and people who have been using it since before I knew how to use a mouse. It’s a lot of fun, and I learn a lot from all the questions that I answer. One of the things I cover with everyone (because some people refuse to use Google and some just don’t know any better) is where to get help with the software. A lot of programming books I read actually fail to address this incredibly important topic or shove it in the last chapter, almost as an afterthought. My position on this topic, particularly in the context of JSL, is more pragmatic. There is no way we can cover everything in the context of this blog series (or even the next blog series I’m planning, for that matter). There’s just too much to cover. So, before we follow the white rabbit any further down the hole, here are my favorite helps for JSL. The Scripting Index Open up JMP and go to Help > Scripting Index. Congratulations! You just accomplished one of the most important tasks for learning JSL. This one tool has got me out of more jams than I care to mention. Every scripter I know has this open during a serious coding session. Every. Single. One. Just get used to having it open. It is the collected knowledge (with examples!) of the JMP developers. It’s updated every release with any changes to JSL or new commands for new platforms in JMP. Why is this so important? Unless you’re scripting all day, every day, there no chance that you will know everything about JSL. This tool fills that gap. Here's a little example. Open up the Scripting Index, and type "powerpoint" in the filter field. The bottom entry in the second column says "Save Presentation." This gives you the form of the command export a report as a PowerPoint File and an example of how to use it.    JMP Tutorials and Sample Data Now, in JMP go to Help > Sample Data. Note the buttons along the top of the interface. One of them says, “Open the Sample Scripts Directory.” Guess what’s in there? Examples of completed scripts! If you want to know how to get a script to add a formula column to a data table, there’s an example in there. If you want to know how to get JMP to connect to an external dll file, there are a few examples in there. There is nothing wrong with starting from these scripts or using them as templates when you are getting started! That is why they are there. JMP Documentation JMP includes a massive amount of documentation, more than 6,500 pages in JMP 15. At ~1,500 pages, the JSL resources make up roughly 20% of that documentation. There are two books in the JSL resources: “The Scripting Guide” and “The JSL Syntax Reference.” This content is also fully searchable on the web, either here The Online Scripting Guide or The Online JSL Syntax Reference. These resources have a lot of useful information and examples about specific use cases. They are good counterpoints to the Scripting Index; the Index is a tactical tool, while the JSL documentation is more of a strategic tool or general reference. The two books provide a lot of material on the different things that JSL can do. In practice, you would look to the Scripting Guide for things like how to work with scripting graphs or how to use Application Builder. You would look to the Syntax reference if you wanted to know all the options for the Bivariate() function or how to use the F Sample Size() function. The Syntax Reference has a lot of overlap with the Scripting Index.   Reverse Engineering Scripts and Add-Ins Now, let’s take a break for some trivia: Did you know that there are multiple copies of da Vinci’s Mona Lisa? It’s true. Da Vinci ran an art studio with many apprentices who studied his work and attempted to replicate it.   The Mona Lisa (1503–1516) by Leonardo da Vinci    Isleworth Mona Lisa (16th Century) by Workshop of Leonardo da Vinci    Prado Mona Lisa (16th Century) by Workshop of Leonardo da Vinci    Portrait of a Woman (16th Century) by Raffaello Sanzio (a.k.a. Raphael) Here’s another one: Did you know that Michelangelo made his money early in his career by copying other artists’ work? He studied the work of prior masters to improve his skills. He also took commissions to recreate ancient works for people (the Renaissance equivalent of ordering a high-quality touch-up print). OK, one more: Do you know what the difference is between an art student and an art forger? One copies art to learn the techniques while the other tries to pass their work off as being done by the original painter. Do you see a trend here? I am of the opinion that there is nothing wrong with borrowing from other people’s work with full transparency and proper attribution. Why should you reinvent the wheel? Find someone else’s wheel and see if you can make it better or better fit your needs, as long you give them a mention somewhere. That’s how you learn this stuff! So, what does that have to do with JMP scripting? Well, there’s a wealth of information put out by people that have studied scripting for years (or actually wrote the code for JSL). It’s all there for you to learn from. But how do you do that? The easy way is to look in the JMP Community in the JSL Scripts section of the File Exchange or the JSL Cookbook. Now, what if you want to have a peek at what people are doing in their add-ins? You can do that, too; it just takes an extra step. The thing most people don’t know about the JMP add-in transport file (the ones with the .jmpaddin file extension) is that it’s just a zip file with a custom file extension. Unless the author has encrypted their code, there is nothing stopping you from taking a peek under the hood. Just change the “.jmpaddin” to “.zip” and let your operating system (OS) unpack it for you. After the OS has unpacked the zip file, you’ll find a number of files in the folder (the ones you want have the .jsl extension). Tear into them (and pray that the person you’re cribbing from does a good job of commenting their code; more on that later). You can also have a look at add-ins that you've installed already. You just have to go diving into the file system. The location can be a little hard to find (especially if your IT Admin's have made it hard to go into hidden directories). But, if you can get there, you can find the possible location of your add-in directory here (there are two possible locations each for MacOS and Windows). In that directory, you'll find a bunch of folders (one per add-in) that are named like a backward URL. Inside each folder are the JSL files for a given add-in. Go ahead and have a look!   Authors note (15Jul2020):  After publishing this article, @Craige_Hales, the author of the Uncharted Blog and JSL wizard, reminded me in the comments section that you can get to installed add-ins directly from the JMP GUI.  Have a look at the comments below for details! Paid Resources (Courses and Books) Strictly speaking, it is possible to learn JSL with just the resources provided in the JMP installation and help on the JMP Community. If all you want to do is string together some bits to make your life a little easier, then that will probably be enough. The thing is, there is a learning curve with JSL. If you want to take your coding beyond what we’re covering in this series, one way or another, you’re going to have to fight up; there are no shortcuts here. You can get some help using the resources I’ve mentioned already, but you’ll still have to put in the time. That said, this is one of the places where I really recommend using some paid resources if you can. If nothing else, you’ll be more committed by putting some skin in the game. The first one is the Introduction to JMP Scripting Language course offered through SAS Education. I took this course when I started scripting seriously, and I highly recommend it. There is a distinct value in sitting in a classroom and being guided through increasingly more complex exercises. It also gives you someone you can ping if you get stuck on something. (I still talk with my instructor all the time. Hi, Di!) And the course notes are a really helpful resource. There are also two books out there that are really helpful: “Jump into JMP Scripting” and “The JSL Companion.” Conclusions That’s the end of this week’s lesson. And it’s the last of the preliminaries. Next week we’re getting into the code. Your homework for this week is to look through the resources I’ve outlined and get to know them. Specifically, start looking at the Scripting Index. Using your problem statement from the last lesson, search through the index for some possible tools or examples that you think might be useful. ... View more

Once more unto the breach! I was talking with my dad a few weeks back. Not to date him, but he’s been using JMP since version 1.0. I remember seeing the 3.5” discs sitting next to our Mac II-se when I was first learning to use a mouse. The same machine was on display at the 2019 Discovery Summit conference in Tucson, Arizona. Here’s a picture of it:     Anyway, my dad and I were discussing my first Five Things blog series. He mentioned that he wished someone would write a series about how to get started with scripting in JMP. At the time, I was already in the process of writing some other articles, so I tucked it away for a future project. Life, however, had other plans. A week later, I was asked to help with a customer presentation and, to my surprise, one of the topics they wanted me to cover was an introduction to scripting! The universe loves jokes, apparently. Anyway, here we go again! Following the format of my first Five Things series, here are the five(ish) things that I think everyone should know when they are starting to script. Up first, a conceptual thought. See a need, fill a need Before you start scripting, you need to consider the need. Odds are, if you want to script, there is some reason. Maybe you’re doing it to make yourself more efficient at work. Maybe you’re looking to standardize analytical methods for a group. Or maybe you are, in fact, one of those rare people that decided to pick it up for grins and giggles. None of these are bad reasons, although the last one may make things a little more difficult (I’ll explain why in a bit). The point is that you should have a reason in your head and a specific purpose for your first script. There are scripts, and then there are scripts There are, in my mind anyway, two different types of scripts: those that act on existing data, and those that create data. Scripts that work on existing data interact with a data table and existing platforms in JMP to create a new user defined output. These can be things as simple as wrapping a window around multiple reports or as complex as a custom-built report in Application Builder. The key commonality is that scripts like this are modifying existing content. The second group are scripts that create data. Scripts that create data can be simulation tools, custom graphics, tools for managing metadata in a JMP file, etc. This group is a little more nebulous, but the common theme is that the scripts extend JMP in ways beyond enriching existing data or analyses. Most of the time, these scripts are going to create new content from scratch. Scripts that modify existing content are generally where people start when they think about learning JMP scripting language (JSL). This is also where we are going to start our discussion. So, let’s take a step back and think about why you are interested in learning JSL. Give me a sentence, Vasily. One sentence only, please. Take a moment and think about your need. Which pain point are you hoping to address with JSL? Do you now see why scripting for grins and giggles (at least at first) can be tricky? You’re much more likely to fight up the learning curve if you’ve got a reason to reach the top. I’m not saying you can’t pick up JSL for the heck of it, just that the fight is harder without the extra motivation. The next step, and the final one that we’re going to cover in this episode, is what you do with that need. The simple answer is that you are going to do two things. First, you are going to write down that (possibly nebulous) idea that’s driving you to learn JSL into a single sentence – a problem statement. And, I do mean just one sentence. Make sure you are able to crystallize the idea into something tangible that you can build off of. It will also help with reining in scope creep. Here are some examples: I want to automate all the graph generation for my weekly report into one script. I want to automate the formatting work for a customer provided data table. I want to generate a set of control charts from a data file provided by my factory computer system.  Note that I’m not talking about specifics; I am simply stating the pain point that has me looking at JSL. And then I stopped writing. Make your punch list Got your problem statement? Good! Now we are going to write a series of specific deliverables for your script. It will be our punch list to help us locate the resources SAS provides for learning JSL. Do not list any graphical user interface (GUI) or user interface (UI) specifically. List the actions that the script needs to perform. Here are examples for the problem statements above: Problem Statement 1: I want to automate all the graph generation for my weekly report into one script. Punch List: Pull a data set from our production database containing the last three weeks of data. Make a graph of time vs temperature for my product. Run an analysis of time vs temperature using a polynomial model. Export the graphs and reports to a PowerPoint file. Problem Statement 2: I want to automate the formatting work for a customer provided data table. Punch List: Rename columns based on internal standards (using table from specs). Assign units to the needed columns (using table from specs). Convert Lat and Long to Geographic coordinates. Change defective units column from continuous to ordinal. Problem Statement 3: I want to generate a set of control charts from a data file provided by my factory computer system. Punch List: Import data from network location. Assign spec limits. Build control charts. Note here that I’m not boxing myself in too much. I’m just building a punch list of things I need the script to do. Since I’m assuming that you’ve never worked in JSL before, I’m also assuming that you don’t know how you’re going to attack these problems. By writing them down, you know what you need to look for in the help and resources. Which is where I will pick up in the next installment of this series. Closing thought for this week Since this series is going to span a few weeks, I decided to make it more of a work-along series than something you just read. So, take these last points I made as homework. If you want some accountability or feedback, put them in the comments below. I’ll chime in if I can help or let you know if you might be setting yourself up for some trouble down the road. For bonus points, open up a script window in JMP (File > New > Script) and put your problem statement and bullet points into the script window, one per line with a double forward slash (“//”) at the start of the line. You may see the text color change to green for those lines in the JSL Script Editor. We’ll cover what you just did in an upcoming episode. Oh, and if you’re wondering who Vasily is and why he should be giving you a singular sentence have a look here. ... View more

  In his final installment of Things I Wished I Knew When I Started Using JMP, Systems Engineer Mike Anderson reveals the secrets of JMP Masters. Mike starts the episode sharing the one common thread among the most proficient JMP users: They. Save. Their. Work! By “save their work,” Mike means they use the automatic code generation capabilities in JMP. This simple action lets users recreate their analyses in their current data set and allows them to rerun the analysis with new or updated data. Finally, Mike shares a little about special keys, in particular the broadcast key. Pressing these special keys allows you to do things like standardize graphs, close all the Outline Items (those title boxes with the gray arrows next to them), export multiple fit equations into a single column in the data table, or even turn all the summary statistics for all the subgroups in a report into a single data table. In short, you’ll be able to increase your efficiency in all kinds of unexpected ways.   ... View more

  According to JMP System Engineer Mike Anderson, “A chart or graph should have a clear purpose. If it doesn’t meet that purpose, it’s not a good chart.” To illustrate this example, Mike shows us a spider diagram of five sensory attributes for 14 types of beer to see if we can answer some simple questions. Ultimately, it’s pretty obvious it’s not easy to get that information out of the spider plot. More importantly, Mike asks: “How hard did you have to think?” A useful visualization, Mike says, must meet five criteria: It’s comprehensive, truthful, “not ugly,” obvious and effortless. So, can his example be presented in a better way? Mike shows us a number of possible candidates so you can judge for yourself.   ... View more

  You’ve probably used JMP’s Distribution Platform, but chances are you’re only scratching the surface of its usefulness. In this Episode of Things I Wish I Knew When I Started Using JMP, Mike Anderson gives us a tour of all the capabilities of this powerful platform, including how to customize the information you view, how to turn on various capabilities to perform various statistical tests on your data and get even more out of The Distribution Menu’s visualizations and more. Mike concludes his session by answering several questions for the viewing audience.   ... View more

  According to JMP Systems Engineer it’s the first thing you should do when handed a new dataset – graph your data. In this episode of Things I Wish I Knew When I Started Using JMP, System Engineer Mike Anderson shows you where to find and how to use the Analyze Menu in JMP. Learn how the Analyze Menu can help you understand the basics of your data and variables (Distribution and Fit Y by X); discover more complex questions including regression (Fit Model) and Machine Learning and Artificial Intelligence Methods (Predictive Modeling, Specialized Modeling, Screening, Multivariate Methods and Clustering); and better understand process, quality and people (Consumer Research).   Full Transcript (Automatically Generated) Morning, everyone. It's a beautiful Monday here in New York, the sun shining finally. And today we're going to be going over another of the things that I wish I had known when I started learning about  jmp. Also something that I like to tell a lot of people when they're first getting started using  jmp, I've been working with a lot of people that have been taking the demo downloads that we have lately. And this is one of the things I tell them on the first day. Because this can be a little bit confusing coming from different software's because the way  jmp considers the Analyze menu is organized entirely differently than the way you would expect coming over from somebody else's software.   As always, I'm going to be taking your questions. So if you've got any questions, find that q&a window and at the end of the session, I'll try and answer as many of them as I can. I got one cut that came in over the weekend from actually from the first episode of this series, where they wanted to know how I did something and I'll be going through that as well, too. To show them how we did that, and we're going to dust off some fun and go into image analysis to do it.   Alright, so first thing, the Analyze menu. When I first started learning the Analyze menu, it really didn't make sense to me until and it actually didn't make sense to me until I started teaching other people to use the software. And then as I started kind of refining my idea of what the the analytical workflow is, a lot of the ideas that we talked about in this series came from those early experiments and how to teach  jmp on my own. I started to understand how the Analyze menu is laid out and that there's a method to this apparent madness.   And it's there's a, what I want to share with you today is a general guideline about how to kind of guess where those things are that you're looking for. Kind of moving your guessing around in the menu from just wild guess to scientific, wild guess And we'll we'll leave the the middle of the layout for the sake of the recording. So let's get started. Let's look at let's, what I've got right here is a screen capture of the Analyze menu right here. So I can use my little laser pointer and point around. And let's take a second let's step back. When you're handed a brand new data set. What's the first thing you should do? Well, if you were here, for the first episode of this series, you would know the first thing you should do is graph your data. Well, if we make zero assumptions about our data, if we make zero assumptions about shape, about relationship about anything, the first type of graph that you probably would run into is something called a histogram.   And it makes no assumptions. It's great for understanding how data is shaped looking for outliers doing basic statistical tests. There's lots of things you can do with just a histogram. If we come under the Analyze menu, the first thing we run into is this platform called the distribution. And that graph that goes with that distribution. Remember from the second thing, we talked about the the  jmp workflow, get a graph, look at the graph, ask questions about what you see and then push the answer button. The graph that comes with the annal with distribution platform, is a histogram. So that's if we want to just look at one thing now if you look at your data, and then you moved a little bit further down the kind of the, the chain of logic that you might be following or that the inquiry that you might be looking at, you might start proposing relationships.   I think a relates to be I think there's a correlation between A and B. Those are all possible things. And those by various relationships, we're starting to make inferences about whether or not something is a response or a factor whether or not it's an X or Y. In that case, we would come down to fit y by x. And there because of some, some clever coding, a lot of different plot, a lot of different comparisons are available all of them comparing one variable to another, most of the time making an assumption that one of them is response, and one of them is a factor.   So it would seem at first glance, at first glance, that the Analyze menu is organized by the number of variables you're trying to compare. But then we get to tabulate and then we get to text Explorer, and that that logic kind of breaks down. So let's let's skip those two for just a minute. And let's move down to fit model and what we're doing Here as we're proposing more complex relationships, it's multivariate Yes, but we're composing, where we're proposing different levels of complexity in our models. And that's the trick. That's the key. The  jmp analyze menu, at least, from what I've seen, and the way that I, the way that I teach it is that it's organized as you go down the menu. You're asking more and more complex more and more specialized problems. Which is why down here at the bottom, we have people consumer research, because let's be honest, people are generally kind of nutty, in very difficult to predict things.   And because of that the data associated with people is invariably very complex and very hard to understand. So we've got some very specialized tools in that platform down there in that subgroup down there at the bottom, to help us navigate those complexities that come with the human condition. And so let's look at look at this a little bit down in the middle here. Let me bring up a little bit of an antic annotated version of this. So we talked about one variable, two variables, fit models regression. In here we have machine learning and AI type models. We also have multivariate platform, screening is kind of the odd man out, except it under the hood, a few of the screening platforms use machine learning technology so that that makes some sense as to why they're there. Then we get into prop quality and process. So that's a more specialized tool. And like I said, people down at the bottom, because it's incredibly complex data.   Now, let's scoot back up to this transfer to the tabulate and text Explorer. And this is a problem in the scheme, because those don't necessarily make sense for them to be right there. Unless you understand that tabulate one of the roles that tabulate has, is to transform data from one form to another. It's good for doing cross tabs, it's good for doing well not so much cross tabs, pivot tables and things like that. So its goal is to do summary to reshape data so that other parts of the software can take the inputs to transformative tool, text explores the same way. A little cue about the role of a platform in  jmp is when we put explorer at the back of something. One of the goals of the tool, one of the goals of the platform is to transform data from something in this case text which is really tricky to work with, to something that we can use for fit model that we can use for doing sentiment analysis that we can use for doing all these different things. So those two bridging tools, those two those To platforms there are bridging their goal is to transform from repair to fairly simply, fairly simple, simplistic type one, not simplistic, but fairly simple type analyses to more complex multivariate analyses that come along.   So let's take a quick let's take a second and do a pop quiz here. I'll leave that up delay to give everybody a cue, and you can follow along. Well, I guess I'm not gonna leave that. Let's start with a question. If you wanted to compare one variable to a target, where would you look in the Analyze menu? I'll give you a second to think about that. got an answer. Okay. one variable, that's the key. So we're going to look on where there is one variable, analyze distribution. What would we do right there? Well, let's just take some data. And let's take some data here that I've got this is just processed data that doesn't really doesn't matter. For the sake of the discussion, we'll just grab something. Click ok. We're in the distribution platform now awesome.   And if I follow my workflow, I'm gonna look at my data I'm gonna say I, you know, as normal enough horseshoes and hand grenades. We could, we could look at that and more in depth if we wanted to. But we come under here for this option that says test mean and I, I'm going to propose that my hypothesized mean is 30. And when I do that, I get down here at the bottom, my T test for the me. Okay. So that's part that's step one. Let's try the net. Let's try one more. Now, what if we wanted to get correlation coefficients? So we're comparing two variables. But we want to do it across all the continuous variables in a data set. Where would you go for that? Give me a second. Think about it again. Okay. If you said by variant, you could do that.   You could do that in by various. It's possible. For come under by under or excuse me, if you if you say fit y by x are saying I'm going to compare two variables. Yes, you're comparing two variables, but you're comparing two variables and then two variables, and then two variables, and then two variables, and then you've got all these common tutorials. combinations of two variables. So while you can do this, you know, let's just grab some, let's grab a bunch of continuous variables here. Again, we're not really worried about what's going on here. Let's just due to, you could come into by very, very easily, and come under the red triangle, and then come down to the density ellipse. And that will give you the correlation. I could do it there. That's true. But then you'd have to do it for F pairwise for every combination in this data set. If you've got 150 different combinations of 150 different continuous variables, you're going to be at that all day.   That's not what we want to do. So we come a little deeper, our problem is more complex. We're coming down further in the in the model, do we need to transform our data? Now we don't do we need a model. That doesn't sound quite right. modeling, modeling, screening, possibly, but multivariate methods, multivariate might be the right place to look for this. And again, there there may be if you're just starting out with  jmp, there may be some, some hunting and pecking some a little bit of guessing around. But the purpose of this little trick is to kind of give you an idea about where you need to guess. So if I dump all my continuous variables in here, Wallah, I get all of my correlations. And I can even if I want to be fancy, I can put them on this scatterplot matrix in here. Just like that, and they're colored so it's easier to look at what's what's Important. Okay. All right. And that's that's the trick about the that's the trick for today. That's the thing we want to cover for today. And if you're interested, this particular graphic, it's up on the blog article that comes with this. It's look at the blog article for this, for this content today is linked in the  jmp on the air segment for this.   So you can get to this article, you can get to that little graphic if you need some help at any time. All right. Now I want to come to and while I'm doing this question, if you have any questions, I've got the q&a open here. Go ahead and type them in the q&a. Instead of taking a two minute break. I'm going to cover a question that I got earlier in the week from someone that wanted to know how I did something in the first episode. But in the meantime, if you've got questions, feel free to type them in to that to the q&a and I'll keep an eye on that while I'm going. All right. The question I got was how Did I make this 3d graphic? And it wasn't the specifics about the 3d graphic itself? That's just scatterplot. 3d, that's easy to do. The question was, how did I get the data? out of my images? Which is a very interesting question. And the short answer is that on Windows, this is really easy to do. So let me pull over my windows enter in my password here before I pull it over.   Now what the problem is, so here's a here's  jmp on Windows. On Mac, the workflows a little bit harder because of a there's some differences in in the underlying code. But the the trick the question here is how do I get Get if I have a series of when I'm opening if I have a series of images or TIFF images and I want to bring those into  jmp, let me just open up all my show all my files here. And in this case I do I have my images set in here and what's called a multi file TIFF. So in this TIFF just like in a GIF, you know what, let's leave that let's leave the argument about GIF or GIF off the side for the moment. In this single file, there are multiple images, it's a movie more or less. This is really handy for for taking thicknesses of so if you're doing X ray tomography, you can set up the image stack in here and it will play through in certain graphic softwares or it's a great way just to keep everything together when you're doing image analysis. But the question is how do we get this data in? Well, in either case, john Ponty a few years back made a great tool called the image analyzer. And this on the Mac, or on the PC side, sorry, this, this makes this job really easy to do.   So I'm going to navigate, I'm going to go choose I'm going to grab my TIFF file here. And it's going to read all the bits attached to this image, spit it out as a data frame with 2 million rows, but also create some images so you can kind of see what's going on inside of this file. So on Windows easiest pi, go grab the image analyzer, it's super easy to do. on the Mac, you would have to do this the image analyzer add in is available but on the Mac It only pulls in the first image, it won't pull in a multi image stack. So you'll have to bring these in one at a time using that image analyzer and then concatenate them. Or if you're into the if you're in the process of doing image analysis, you probably know about a tool called image J. And this is what this is the way I did it. And what you can do is there's a macro out there, I can come in and open my data. Let's go open recent.   There's my image subset, same data set, I can come into my macros. And I have a macro right here that does the same thing runs through all the images, extracts all the bits and puts them into a CSV file that I can then bring into  jmp really easily and I'll post That batch, macro as well on on the community later on this afternoon. So that's how I built those things. Now I'm not seeing any questions here. So we're going to go ahead and we will, we'll wrap up. One thing I do want to say is, again, the point of the primary point of today's content is that's how you analyze or navigate the Analyze menu. Next week, we're going to spend a little bit more time on that first part of the Analyze menu on the distribution platform and go over some of the things you may not have known are capable of doing you're capable of doing distribution platform.   ... View more

The Engineering Mailbag Episode 4: Pythons and Pandas, Oh My! Every now and again we systems engineers run into interesting questions that would fall somewhat outside the typical range of JMP usage. These applications are generally clever, and often bring home how using data isn’t limited to solving business or technical problems. Other times, the questions are just unexpected, challenging problems. These “curve balls” (as I like to call them) come in many different forms: coding problems, interesting analyses, ways of visualizing data… you get the idea. I haven’t written from the mailbag in a while, but this particular question really seemed to be something that needed to be addressed. The tool I used to answer the question really isn’t discussed as much as I think it should be. So, this is an opportunity for me to help out a colleague from New York and show everyone something interesting that you might not be aware that JMP can do. The Question Mike[JMP], Often, I need to do groupbys where I need to grab say the max value everything grouped by some label where the Python Pandas equivalent would be something like: df.groupby(['Something', 'Something Else']).max() Another might be something like pandas.nlargest with a groupby, where I would want say the top 5 things out of a groupby. And after a couple minutes of fiddling with a column formula I give up and export the data to reimport it into JMP later. Tabulate sort of does this, but not always in the way I want. So a "How to JMP like Pandas" document would be super helpful for me, and might be helpful for JMP with how widespread Pandas has become… Thanks, Nick My Response Hi, Nick! There is a trick that you can use to get more flexibility out of formulas that mimics a lot of what [Pandas] can do. It involves telling JMP what columns you want to group by. You do this by right clicking on the column name and selecting New Formula Column > Group By.  After doing that, any formula columns you create by right clicking on the column name and using New Formula Column> … will be grouped by the grouping columns you set.  After that, it’s just a matter of filtering or subsetting using data filters or local data filters. Let me know if that helps! Best, M Pandas: Cute Mustelid or Data Analysis Tool All right, for the uninitiated, Pandas is not a cuddly, chromatically challenged mustelid. Pandas (usually abbreviated as “pd” in python scripts) is a “…Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series [source]." It basically fills the role that the data organization and manipulation parts of JMP (data tables, Row and Column menus, etc.) do in the Python programming environment. So, it’s pretty important to Python data analytics workflows. Now, if you are looking to run Python from within JMP, this ain’t the blog article for you. Go have a look at this white paper. If you are looking to break yourself free from doing data transformation in a command line, read on. DataFrames and Data Tables Still here? Awesome. Okay, the rest of this article is going to go through some of the common bits of Pandas and their analog in JMP. It’s not a comprehensive list by any means; it’s intended to point your nose in the right direction when thinking about the two environments. The first bit I want to cover is the “DataFrame.” In Pandas, this is where everything is stored. Like data tables in JMP, it is essentially a collection of data columns with a name (column name), etc. You act on columns and rows (transform things, summarize, etc.) but you’re always acting on either the DataFrame as a whole or some part of it. In JMP, the primary data container is the data table. You can act on columns and rows in the same ways you would in a Pandas DataFrame. In both, you are able to deal with missing values in the data table. In JMP, they are designated as either a cell with a “.” in it (for numerical data) or a blank cell (for character data). You can transform data, summarize it, etc. (more on those bits later). In both cases, the more you tell the software about the data the less leg work you have to do, which brings me to my next point. Column Properties and dtypes In the matter of column properties and data types (dtypes in Pandas), JMP wins hands down. Pandas inherits most of its data type understanding from another Python package called NumPy, which is cool. NumPy is pretty sturdy code, but it’s really limited in this regard. It handles about eight types of data (see the table below[source]) with somewhat limited support for different formatting (it gets pretty confused by currencies without help, for instance). Kind of Data Data Type Scalar Array String Aliases Documentation tz-aware datetime DatetimeTZDtype Timestamp arrays.DatetimeArray 'datetime64[ns, <tz>]' Time zone handling Categorical CategoricalDtype (none) Categorical 'category' Categorical data period (time spans) PeriodDtype Period arrays.PeriodArray 'period[<freq>]', 'Period[<freq>]' Time span representation sparse SparseDtype (none) arrays.SparseArray 'Sparse', 'Sparse[int]', 'Sparse[float]' Sparse data structures intervals IntervalDtype Interval arrays.IntervalArray 'interval', 'Interval', 'Interval[<numpy_dtype>]', 'Interval[datetime64[ns, <tz>]]', 'Interval[timedelta64[<freq>]]' IntervalIndex nullable integer Int64Dtype, … (none) arrays.IntegerArray 'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64' Nullable integer data type   Strings StringDtype str arrays.StringArray 'string' Working with text data Boolean (with NA) BooleanDtype bool arrays.BooleanArray 'boolean' Boolean data with missing values The column properties in JMP comprise about 43 pages of the Using JMP book. They are significantly more extensive and are good at recognizing data types as they are imported. Of particular interest to a new JMP user is the fact that column properties extend well beyond the data types that Pandas uses. It can handle a host of metadata that trigger actions across the software. Now, to be fair, you can attach attributes to a Pandas DataFrame, but since the DataFrame is just a container, you have to either code in the support for the attributes or hope someone put in the support in the package you are using. All that is already done for you in JMP. Tabulate, Column Formulas and GroupBys The truth is that the previous two items don’t really matter that much in the context of the question I was trying to answer. For our purposes, the data table/ DataFrame is just a box to put stuff in. The column properties/types can help with analytical considerations, but, again, that’s not important for the question we’re working with here. The bit of Pandas that we are interested in is called GroupBy. And it’s for slicing and dicing data in preparation for doing other analyses. A quick example using the JMP Big Class data table and getting the average height and weight of boys and girls by age might look like this:         # Import Pandas import pandas as pd # Bring in the data data = pd.read_csv("/Users/mike.anderson@jmp.com/Desktop/Big Class.csv") # Put the data into a DataFrame df = pd.DataFrame(data) # Run the group by, get the means, and show the results summ_df= df.groupby(['sex','age']).mean() print(summ_df)         With the output looking something like this:         height weight sex age F 12 58.600000 100.200000 13 59.000000 95.333333 14 62.600000 96.600000 15 63.000000 102.000000 16 62.500000 113.500000 17 62.000000 116.000000 M 12 57.333333 97.000000 13 61.250000 94.250000 14 65.285714 103.857143 15 65.200000 110.800000 16 68.000000 128.000000 17 69.000000 153.000000         Normally, JMP users would look to Summary or Tabulate as their primary tools for these data transformation operations. They might even use the Formula Editor to create a column. But, there is a secret weapon in the data table (and in the platform column list) that you can use to do just about everything you’d want to do in Pandas and more. That secret is the ability to right click on a column name and select New Formula Column. This works anywhere that there is a column list! From there you have many different options for doing various transforms, such as combining columns, doing cumulative sums in rows, etc.   That, in and of itself, while handy, is not particularly magical. The place where things get really interesting is when you turn on the “Group By” flag for a column in JMP. This will automatically make any formula columns you create aware of the levels in a grouping variable. Just think about that for a second -- you can create a formula column anywhere in the software that can be stratified based on a grouping column. Let’s look at an example of what this means from another question I received recently: An Example: TWO QUESTIONS FOR THE PRICE OF ONE!! About the same time the initial question came through, I got a question from another customer about how to do a numerical integral (area under a curve or AUC) for a large collection of samples. Moreover, they were willing to provide me with a sample data set they had published previously for a journal article in Environmental and Molecular Mutagenesis. It turns out that what they wanted to do was easily accomplished using Group By Flags and Function Columns. Here’s the procedure (you can follow along using the data from the JMP Public site 😞 First, let’s graph the data. The details of the study aren’t important for this discussion, though it is interesting reading. The short version is that it’s a large collection of chemicals with different concentrations (Conc. (uM)) and a series of genetic responses. For our exercise, they’re interested in calculating the AUC for each of the genetic responses. Here’s a look in Graph Builder with a column switcher: The first step in the process of calculating the AUC is setting up the grouping. We’re going to right click on the Chemical column, select “New Formula Column,” and then “Group By.” We’re going to do this again for the Biomarker column. Note that order can matter here. We’re grouping first by Chemical and then Biomarker within Chemical. After that, it’s just a matter of using the “New Formula Column” in the data table to build up the calculation. Going forward, I broke each step out into individual columns for the sake of clarity. And, if you’re really interested, I’m using the Trapezoidal Rule Riemann Sum for the math here. Ok, back to the calculation. The first thing we need is the distance between each point in the x-axis (the point-wise change in x). We’re going to select the Conc (um) column, right click on the title and select New Formula Column>Row>Difference. Now open the formula in the new column that JMP created. It should show something like this: Note that JMP is handling calculating between rows and has adjusted the formula to reset when it detects a new Chemical or a new Biomarker type. Next we need the average value of Value between each data point and the previous one. Right click on the column header and select New Formula Column>Row>Lag. Open the formula and modify it a little so that it’s calculating the difference between rows and not just giving the value of the previous row. That’s shown in this screenshot: Those two columns (I renamed them dx and dy in the data table) need to be multiplied together to get the area of a segment. Select the two columns, and right click on the column name of one of them. Select New Formula Column>Combine>Product. The new column gives us the approximate area under the portion of the curve between each data point and the next one. I renamed that column iAUC(for instantaneous AUC) in the data table. The last step is to sum up the values for each section. This is done by right clicking on the iAUC column and selecting New Formula Column>Row>Cumulative Sum. Because we selected the Group By columns at the beginning of the exercise, the tally is reset for each sample in the data set. From here we could use Tabulate or Summary to create a table with the max values for each sample (the approximate total area under the curve). Here’s an example for a common chemical: Final Thoughts And, that’s it. You can easily replicate behaviors in Pandas using structures that exist in JMP. Just remember to use the Group By variable and you’re off. No mustelids, chromatically challenged or otherwise, required. Editor's note: Have you read the other installments in our Engineering Mailbag series ? See how Mike helps a user who was using JMP  to decide whether to sign up for a snow removal service and develop unique visualizations for STEAM education programs. ... View more

  In this episode of  Things  I Wish I Knew When I Started Using JMP ,  Systems Engineer  Mike Anderson   introduces the 4-step JMP Workflow,  an easy way  to create and understand  JMP reports.  The process is simple, but  powerful .   Step 1 – Make a  report.  Step 2 – Look at the graph; are you seeing what you want to shar e  within  five  second s ?   Step 3 – Ask questions about what you see.  Step 4 – Push the  “ Answer  B utto n .”   Using data from  Futurama , one of his favorite shows,  Mike d emonstrate s  how easy, and effective, the 4-step JMP workflow can be. ... View more

  Learn some of the ways you can work with your text data in JMP and JMP Pro.   Full Transcript (Automatically Generated) Morning, or good afternoon, everyone. My name is Mike Anderson. I'm a systems engineer for New York. And today I'm going to be showing you a few of the things that I love about text Explorer. I got really passionate about this particular tool back when it was when it was first introduced. And it's it's been kind of a hobby for me to kind of get go around with it and play with the play with text explore a little bit.   The for the sake of argument here, I'm going to use some data that I've never used before. I was working on this over the weekend trying to pull this data set together. And this is being stuck at home like everybody else. I've been going through some of my old favorite TV shows one of those back in grad school was Futurama. So I've pulled all of the the plots and plot summaries from Wikipedia for all of the Futurama episodes, and that's what we're going to be analyzing today. Now before we go into this in too much detail, let me give you a little bit of an idea about what text explores for and it all lines up in that last name in jmp, there are two platforms that are called explorers. There is text exploring, there's functional Data Explorer. These platforms, while they can do an analysis among themselves, they're one of their primary features is to transform data from one usually tricky format to analyze into something that's easier to work with in jmp.   So that said, Well, we can do some very interesting things in the text Explorer. One of the things that I love about it is its ability to transform text data into something that I can do sentiment analysis with. And I'll there's some resources on that I'm not going to go down that road. That's that's a whole nother rabbit hole to go down. But one of the things that make that I love about text Explorer is that facility to take text which is challenging to analyze in a in a statistical package to something that we can analyze that we can model We can crunch in a little bit more in a more comfortable setting. So let's go ahead and let's get started with this. And I'm going to show you first the how to get the platform set up. So we'll go into analyze text Explorer.   And it's as simple as dropping in the free text that you've got in your data set. So text explorer goes into we go into there first, we can we have a bunch of options that we can use to filter down the data, we can get rid of really, really short words, we can work with phrases and things like that as well. There's a great tutorial by Nick Shelton on how to do all of the basics that's out on the web. So I'm gonna skip over that just for a second and get to kind of the highlight reel. of the things I love about text explorer in general. The first thing that I love is the word cloud. Okay, this is when people talk about text analysis. This is the first thing that pops to mind. It can be an potful it can be incredibly powerful. It's a great tool for visualizing this data and it makes it kind of easy for people to consume. In this case. The first thing we see out of the gate we see the main characters we see Philip j fry we see Ben Bender, we see Leila, we see these main characters that are in almost every episode, so it's not surprising that they are going to that they're going to show up a lot in the plot summaries. Now removing that information is really easy. I can right click on something and say add a stop word and that excludes that word from analysis. If I do that with the just the by removing just these three, just fry bender and Lila   I get up to something where I can start picking out different themes that are present within the dataset. Now, I've colored this based on the overall Internet Movie Database rating for The Internet Movie Database rating for the for the individual episodes. And you can see that in some cases, things that involve Farnsworth or involve involve robots, you can see that those tend to be a little bit lower on the rating scales than things that involve the professor's or for instance, time travel or something like that some of the more kind of out there sci fi type topics that are associated with this with this show. And that brings me to the next thing that I love about this, that I love about this platform. And this is a pro capability. But what I love is the ability to come into after I've gotten rid of the words that don't necessarily matter. Once I've got into got the dataset curated, I can come in and say do something called Latent Semantic Analysis. And that's a lot of words to say, we're going to analyze all of this, all of these terms that we've that we've looked at, and turn them into things that maybe we can look at it from trends are phrases and words that tend to show up together. And that's what we've got down here.   Now in this case we don't see a lot, that's good, you dramas a show was kind of all over the place, there was no real formulaic element to it in terms of the different plot elements that it had into it. So expecting to see a lot of interesting things in here wasn't something I was looking for. What I was looking for curious about was how the topic analysis, which is one of the options for under SVD would be able to handle this kind of data. And what it does in this case is it starts picking out individual episodes, not by picking them out based on their on on the row of information but just based on the words and phrases that tend to show up together. It starts reconstructing the synopsis for different for different people. So it's, and that's kind of a good idea for what for what topics are in this case, they're not necessarily topics, they're words and phrases that show to show up together very frequently.   And you can use that with your own personal Insight with your subject matter expertise to read into that and start figuring out things that are that are important within this dataset. And let me show you one where we've got some common themes that tend to show up a lot within a data set. This is the same type of analysis. This is the this is a actually one of the sample data sets in jmp, and it is looking at survey of pets.   And what we can see here is we this, this top plot now actually has some structure to it has some things that are interesting to it. And the way I explain this to people is each one of these kind of tendrils that goes off along along each side. You can see there's three of them here. And you can see three One, two, maybe three, maybe four over here for different three or four different topics, or three or four different common sets of documents that show up within the data set. And this helps you when you're looking at data where you're looking for commonality when you're looking for common themes that show up within a data set like survey analysis, or when you're analyzing reviews from products or when you're looking at, you're looking at your social media. Those kind of common threads and themes can provide insight for you and we can come in and just grab them really quickly. Here I'm using the lasso tool. And I can pull up the text and I can look in here and I can see what's going on. Aside from the strange. The cat, the itinerant cat entry here, there's a couple of them. This thread is talking about guard dogs. And we can get that just from the the different terms that tend to show up together within the data set. And that's that's the crux of the things that I like about the crux of the things that I like about this, this platform, the ability to visualize a very challenging data set the word cloud, the ability to take in, dig into that data set for its for different themes or trends you may not know are there. And the third is its ability, as I said, first to transform data from one form to another to let us do things like sentiment analysis and things like that. The way we do that, in this case is we come up to the red triangle, and we come down to where it says save document term matrix. And what that will do is it will take all of the terms that we've taken in our data set and create columns of it within the original data set for each of those terms, and those become the factors that we use when we're building a model. For for instance, sentiment analysis. And this is a bit of a high This is a bit of a high level over View there are some great tools for working with sediment and working with with text explorer out there on the web. There's two mastering jmp seminars already recorded. You can go and look at those today. Those are going to be those links are going to be in the community page for this episode of jmp on air. Julian, I'll take it back to you now. ... View more

    This add-in combines a number of visualizations related to consumer analytics, including Spider Plots, a multivariate plot on normalized values (riffing on @danschikore's RadViz add-in), a parallel plot, and pairwise correlation analysis.  The add-in installs as "Parker Analysis" (e.g. Peter Parker) under the add-in menu.   The setup is straight forward - just drop the variables of interest into the variables field and the sample/person/product ID in the Label field.  A summarized version of the Candy Bar Nutrition sample data set is included to demonstrate the table set up.     Since collected data can sometimes not extend the full width of an available scale (i.e. possible values ranged between 1 and 7, but observed values only ranging between 2 and 4) the platform has an option to use the Axis column property.  Select the checkbox in the lower left of the launcher and the tool will scan the variables for Axis settings and scale those parts of the Spider plot accordingly.       Comments, complaints, and thoughts for improvement are always welcome!   Updates: 19May2020: Switched from adhoc parallel plot to using Graph Builder's Parallel Plot and added a bar chart with a column switcher.     ... View more