JMPer Cable articles

Innovation Manager Lee Laim on Micron’s analytics culture and the symbiotic relationship of domain expertise and statistics

Meg_Eberle — Thu, 18 Nov 2021 15:20:38 GMT

Lee Laim says domain expertise and statistical approaches complement each other at Micron.Lee Laim began his career at Micron Technology more than 14 years ago. Starting as a semiconductor yield engineer, he worked to monitor reliability and post-assembly yields on the production of dynamic random access memory (DRAM) products, flag yield limiting process deviations and perform electrical characterization and fault isolation. Embracing industrial statistics early on, Lee quickly rose through the ranks to become Innovation Manager.

Today, he is responsible for implementing the company’s analytics vision via Industry 4.0 initiatives that drive quality improvements and cost savings at every stage of the manufacturing workflow.

Though his background is in semiconductor engineering and spectroscopy, Lee has become adept at statistical analysis, data mining and machine learning – all of which he says complement domain expertise at Micron.

I sat down to talk with @leelaim about the role he’s played at Micron in advocating for the deployment of statistical approaches.

Meg: You first joined Micron as a yield engineer, likely with not all that much experience in industrial statistics – at least comparatively speaking! But fast forward 14 years, and you’re in a role devoted entirely to analytics transformation and Industry 4.0. How would you say your enthusiasm for analytics has shaped your career path so far?

Lee: The amount of data we collect requires constant innovation in data storage, retrieval and analytics. I feel like my current role as Innovation Manager is a direct result of relentless focus on connecting data to the business needs in my previous roles. Knowing how the data is generated, collected and stored is one-third of the battle; another third is knowing how to access, refine and interpret the data; and the final third is being able to share the insights with the larger network.

Meg: What a great point about drawing a direct line between data and business needs – that’s really the essence of analytics culture: getting people to think about how they can use data to solve business challenges. What kinds of successes have helped you to sustain momentum for this kind of thinking?

Lee: Improvements around simplifying data accessibility for more users through APIs has made it easier to get ideas off the ground and into production.

Meg: What is the value in having a workforce of engineers and scientists who, in addition to being domain experts, are also statistically literate? Has JMP in any way changed how domain experts at Micron apply their knowledge and skills?

Lee: Domain expertise and statistical approaches complement each other at Micron. Domain experts will always have the upper hand in abstract thinking, while the computer is unrivaled in its ability to churn through massive amounts of data. JMP is a platform that allows our domain experts to code their insights to deploy on large data sets, leading to greater impact in less time.

Meg: JMP offers a lot different resources for learning and engaging with other users – I’m thinking about things like the Early Adopter program and Discovery Summit. In your experience, what value do you get from participating in these kinds of programs?

Lee: With the Early Adopter program, we get to learn about new functionality in upcoming releases. We also get to see different ways of using existing platforms. For example, using images as markers in the Graph Builder provides an intuitive dimension to the analytics.

Meg: What advice would you give to someone looking to cultivate a more mature analytics culture in their organization?

Lee: Analytic vision has to be driven from the top. Alignment of platforms, methods, and training enables faster development cycles.

Meg: How about advice for someone who’s just starting out with JMP?

Lee: It takes some time to understand how the different data types affect different analytic platforms. Once data types are understood, use the scripting and statistics index to learn and apply analytics to your data sets.

High-speed train versus scenic train: Drag-and-drop control charts versus dialogs

tonya_mauldin — Fri, 05 Nov 2021 17:04:37 GMT

Will you take the high-speed or scenic route for creating your control chart?When traveling by train from Paris to Geneva, you have two options: high-speed train or regional train. The high-speed train will get you there in 3 hours and 5 minutes. The regional train takes 8 hours and 35 minutes. While the high-speed train is the fastest and most comfortable, the regional train does not require you to reserve a seat and allows you to stop in beautiful cities along the way like Belfort, Basel, and Neuchatel to explore. Control Chart Builder is somewhat like the regional train. It is an innovative tool that allows you to visually create control charts one piece at a time. This is great for exploratory and discovery purposes.

However, there may be times you choose to take the high speed train. Sometimes, you may already know the type of control chart you are looking for, and you need a quick method of producing the chart rather than adding pieces one at a time. Dialog options were added to Control Chart Builder in JMP 15. These dialogs are a bit like the high-speed train. They are a faster method of obtaining the desired control chart. JMP 16.1 added a few new features to these dialogs.

The IMR dialog now has a sample label role.
The IMR dialog has a checkbox for sort by sample label.
The XBar dialog has a constant subgroup size field.
The XBar dialog allows the user to pick the dispersion chart type.

Let's take a closer look.

Dialogs

To understand the advantage of using the dialogs, let’s go through an example. Open the Washers data table found in the Quality Control sample data folder in JMP.

dt=Open("$SAMPLE_DATA/Quality Control/Washers.jmp");

We know our goal is to create an NP control chart of # defective. To do this using the drag-and-drop interface of Control Chart Builder, select Analyze->Quality and Process->Control Chart Builder. Drag # defective to Y. You get an IMR chart.

Drag lot to Subgroup. Since the response is the number defective, change the chart type from Shewhart Variables to Shewhart Attribute. Now you get a U chart.

To take into account the number of trials, drag Lot Size to n Trials. Now you get a C chart.

Since we are dealing with events that have two outcomes – defective or not defective – and the number of trials is is fixed, we should be using the binomial distribution. Change the Sigma to Binomial. Now you get a P chart.

To plot counts rather than proportions (NP Chart), change the point statistic from Proportion to Count.

You finally get the NP chart you knew you wanted in the beginning. Whew! That was a lot of clicking and a lot of different chart types that were generated during this process.

If you know what chart type you wish to create ahead of time, it is often easier and quicker to use a dialog entry into Control Chart Builder. For this example, select Analyze->Quality and Process->Control Chart->NP Control Chart. Specify # defective as Y. Specify Lot as Subgroup. Specify Lot Size as n Trials.

Click OK. You get the same NP chart as above with many fewer mouse clicks!

IMR dialog additions for JMP 16.1

Two new options were added to the IMR dialog in JMP 16.1. Open the Hollywood Movies table found in the sample data folder.

dt=Open("$SAMPLE_DATA/Hollywood Movies.jmp");

Select Analyze->Quality and Process->Control Chart->IMR Control Chart. New for JMP 16.1 is the Sample Label role and the Sort by Sample Label checkbox. Specify Rotten Tomatoes Score as Y and Movie Name as Sample Label. By default, the Sort by Sample Label checkbox is checked. Uncheck this box. This means that the subgroup role will be sorted in data table row order rather than alphabetical order.

Click OK.

You get an IMR chart using Movie Name as the sample label. The subgroup role is using data table row order rather than alphabetical order for Movie Name.

XBar dialog additions for JMP 16.1

Two new additions were made to the XBar dialog in JMP 16.1. Open the Coating table found in the Quality Control sample data folder.

dt=Open("$SAMPLE_DATA/Quality Control/Coating.jmp");

Go to Analyze->Quality and Process->Control Chart->XBar Control Chart. New in Version 16.1 is the Constant Subgroup Size field that appears in the lower left and the Dispersion Chart drop down that also appears in the bottom left of the dialog. Specify Weight for Y. This data table contains a subgroup column: Sample. Each level of Sample has 4 observations. Some data tables do not contain a subgroup column. If this data table did not contain the column Sample, we could get the same analysis by specifying 4 for Constant Subgroup Size (rather than Sample as Subgroup). Let’s do that here. Specify 4 for Constant Subgroup Size. Change the chart type from Range to Standard Deviation.

Click OK.

You get an XBar and S chart with a subgroup sample size of 4.

Conclusion

Dialogs can be a quicker way to obtain control charts if you know ahead of time what type of chart you wish to create. JMP 16.1 added several new features to these dialogs:

Sample label role for IMR charts.
Sort by Sample Label checkbox for IMR charts.
Constant Subgroup Size field for XBar charts.
Dispersion Chart dropdown for XBar charts.

Next time you need to create a control chart, will you be taking the regional train (Control Chart Builder) or the high-speed train (dialogs)?

Dr. Georg Raming of Siltronic on how analytics enriches the work of a domain expert – and why resources from JMP can add value

Meg_Eberle — Wed, 03 Nov 2021 18:31:27 GMT

Dr. Georg Raming of SiltronicGeorg Raming, PhD, is an electrical engineer and Six Sigma Black Belt at Siltronic AG, one of the world’s top manufacturers of silicon wafer technology. Based in Burghausen, Germany, @Georg works on the development of silicon crystal growth processes and is responsible for supporting JMP use for more than a hundred users across Siltronic AG. He holds a PhD in the simulation of electrothermal processes.

Meg: As a Six Sigma Black Belt and someone whose work and research have incorporated applied statistical methods for many years, you’re at a position in Siltronic where you can act as change agent and resource for those who are new to statistics. But introducing statistical methods can be disruptive at first. What helped you to garner support for data initiatives with both your leadership and colleagues?

Georg: Siltronic AG has always been strong in statistics, as this is really a must in the semiconductor industry. But JMP has made it a lot easier and attractive for users to deploy standard statistics as well as advanced methods.

At Siltronic, our management was aware of this quite early on. Soon it became evident that with JMP, more comprehensive analysis and better decisions were possible, and that gave us the drive to implement it widely.

Meg: What were some of the hurdles you had to overcome in that initial implementation phase?

Georg: At first glance, JMP looks quite complex. And it is different from Excel – which is also its strength. But, for sure, users need to change how they think and act. However, there is more than one measure users can take to overcome this hurdle.

I’ve found that it’s quite important – albeit not entirely sufficient – to motivate users by giving them a vision of what is possible. A small share of users did learn JMP independently, but for the others, I had to deploy a good communication strategy and provide opportunities for support and personal training. That is what really helped us to gain speed.

Meg: One of the most common objections we hear from newcomers to JMP is that some fear statistical approaches will supplant domain expertise. Is that something you’ve had to address at Siltronic?

Georg: To me personally, this argument does not sound convincing. Domain expertise is always necessary to solve problems and is only improved and accelerated by statistical approaches.

In my opinion, this is a communication challenge: Statistical analysis should be done together with domain experts, not against. JMP’s ease of use makes it easier for domain experts to utilize statistical methods on their own.

Meg: You mentioned support and training resources, and I’m wondering how you’ve leveraged your relationship with the JMP organization – via things like Discovery Summits, STIPS and users groups – to build analytics capability.

Georg: I’ve used all of these resources from JMP and found them to be great and effective. In fact, it has really helped us to overcome the learning curve. It’s so important to be at the leading edge of a dynamically improving software.

Of course, the personal connection with the JMP team is also helpful and much appreciated, particularly when solving more advanced problems.

Meg: How has advocating for analytics transformation impacted the course of your career?

Georg: Using analytics and advocating for analytics approaches are a part of my work, so I can’t say that it has had any direct impact on my career trajectory. But it has enriched both a lot of my work and my network.

Meg: What advice would you give to someone looking to cultivate a more mature analytics culture in their organization?

Georg: Get management support on using data analytics – and communicate that support. Establish visible leadership in using statistical methods and clearly demonstrate to colleagues how statistics is supporting their work and improving decision making.

Meg: What advice would you give to someone who is just starting out with JMP?

Georg: Think about and define what you need JMP for. Then try to find a small first task that can be done regularly with JMP and concentrate on that small part of JMP until you feel more comfortable with it. Also, I recommend seeking out an experienced user and investing in building a network so that users within your company can support each other.

New subgrouping features for Process Screening available in JMP 16.1

tonya_mauldin — Tue, 02 Nov 2021 13:00:00 GMT

Rational subgrouping is the process of organizing the data into groups that were produced under the same/similar operating conditions. Several new subgrouping features were added to the Process Screening platform in JMP 16.1 to make this analysis more flexible and user friendly.

You can now specify multiple subgroup columns.
Individual and Moving Range charts with a subgroup role now create a detail table that includes sample label column(s).
An option was added that gives you the flexibility to sort by subgroup columns or not.

To see examples of these new features, open Semiconductor Capability.jmp found in the sample data folder.

dt=open("$SAMPLE_DATA/Semiconductor Capability.jmp");

Multiple subgroup columns and sample label column

Select Analyze->Screening->Process Screening. Specify NPN1 as Process Variables. For Subgroup, specify lot_id and wafer.

In previous versions, only one subgroup column was allowed for Process Screening. Beginning with JMP 16.1, multiple subgroup columns can be defined for the Process Screening analysis. Click OK.

From the red triangle, select Save Details Table. In addition to a Subgroup column, the table contains columns for all subgroup variables: lot_id and wafer. In previous versions, only a Subgroup column was created. There was no identifying information for lot_id and wafer in the details table.

Sort by Subgroup

Select Analyze->Screening->Process Screening. Specify NPN1 as Process Variables. Specify wafer as Subgroup. Set the Control Chart Type to XBar and R.

Notice the new checkbox in the lower left “Sort by Subgroup”. By default, this box is not checked meaning the data will not be sorted.

Click OK. From the red triangle at the top of the report, select Save Details Table.

We see that the first subgroup corresponds to wafer=1. The 25^th subgroup also corresponds to wafer=1. This is because Process Screening did no sorting of the data. If you look back at the original data table, you will see a similar arrangement where wafer=1 for rows 1-5 and wafer=1 for rows 121-125.

Let’s compare this to checking the Sort by Subgroup box in the dialog. Select the original Semiconductor Capability data table. Select Analyze->Screening->Process Screening. Click Recall. You should have the same dialog as previously pictured. In this dialog box, check the box in the lower left for Sort by Subgroup before clicking OK.

Now when you select Save Details Table, you find that there is only 1 subgroup for wafer 1 for the NPN1 column.

Comparing the Process Screening results, you will find that the given statistics are different depending on the sorting options used. Rational subgroups define how the analysis is implemented.

Conclusion

JMP 16.1 provides more flexibility with regards to subgroup variables.

Multiple subgroup variables can be defined.
Detail tables from IMR charts with a subgroup variable contain a subgroup label column for each subgroup variable.
The new Sort by Subgroup command determines if the platform sorts the data prior to the analysis or not.

Developer Tutorial: From sports to stats to generalized regression in JMP Pro

gail_massari — Wed, 10 Nov 2021 16:28:59 GMT

JMP Statistical Developer Ryan Lekivetz () recently shared details and answered questions about how and why you can use covariates to build better designs using JMP; this was part of our Developer Tutorial series. JMP Systems Engineer Tom Donnelly () was on hand during Ryan's webinar to share user situations where covariates are in play. The video and resources from the webinar are now available.

Starting Nov. 4 (and continuing Nov. 11 and 18) JMP Statistical Developer Clay Barker () will present the next three Developer Tutorials. One of Clay’s key contributions to JMP Pro is in the area of generalized regression variable selection techniques that specifically address modeling correlated and high-dimensional data that include more variables than observations.

I spoke with Clay when scheduling his first session, where he will discuss how to use JMP Pro’s Generalized Regression to make the most of your designed experiment. He’ll make sense of some of the related statistics.

Clay made his way to a statistics PhD through sports – a slam dunk for the JMP team! The game clock is still running for JMP Pro 17, where he and the JMP Development Team are enhancing generalized regression.

See the video of Clay's Nov. 4 Tutorial (and get slides and JMP Pro Journal) where he:

Shares his expertise about capabilities he has helped develop in JMP Pro.
Explains what the Generalized Regression capabilities aim to achieve for users designing experiments.
Describes how and why JMP Pro techniques (aka platforms) use specific statistical approaches.
Discusses which capabilities might be most statistically useful for certain situations.
Answered questions.

Want to see some of Clay’s Discovery Summit presentations?

2018 Tutorial: Generalized Regression – The Most Flexible Modeling Tool That You're Not Using
2019 Presentation: Not Quite Normal – Choosing the Best Distribution for Modeling Your Response

How do we analyze defective subpopulations in JMP's reliability platform?

JerryFish — Mon, 25 Oct 2021 15:36:05 GMT

Suppose you are making parts, and some small percentage of those parts are susceptible to an early life "defect" that causes them to fail, while the rest of the population doesn't have the defect and will never fail because if it. How do you handle that?

A paper presented recently at Discovery Summit addressed this very problem. The author of the paper was Dr. David Trindade, and he spoke on analyzing Defective Subpopulations (DS).

A DS can occur when a manufacturing process introduces a defect into a small portion of the total parts produced. For example, perhaps a motor manufacturer buys electrical brushes from several vendors. Unknowingly at the time of assembly, one of the vendors (supplying perhaps 15% of the brushes for this particular motor model) provided brushes that would ultimately fail early in the life of the motors. Brushes from the remaining vendors were all OK. So there is a small fixed portion of the population of motors that will fail early in life.

We could analyze the reliability of the motor using JMP's Life Distribution platform, and assume that there are two "causes" of failure by perhaps fitting two Weibull curves to the data. One of the curves would fit the early failure data, while the second would take care of the rest of the motors.

But this isn't exactly correct. The Weibull distributions would assume that the entire population was susceptible to either type of failure for the entire lifespan of the product, when in fact only a few motors would experience the early failures. Once the motors in this subpopulation have all failed, no more motors would fail due to this "bad brush" cause.

That is where Defective Subpopulation distributions come in. Here, we assume that only a portion of the products have this initial, early life failure defect. The result is a better model of the reliability of the product or system.

If you are interested in seeing more on this subject, I encourage you to read Dr. Trindade's excellent paper. Enjoy!

Predictive Modeling Unsession at Discovery Summit Americas 2021

Ross_Metusalem — Tue, 28 Sep 2021 14:02:32 GMT

If you’re interested in predictive modeling, the upcoming JMP Discovery Summit includes a session you won’t want to miss. Join your fellow attendees on Oct. 4 from 1-4 p.m. ET for the Predictive Modeling Unsession, the best opportunity all conference long to engage with industry peers and JMP staff around our shared enthusiasm for using data to predict outcomes.

What’s an “unsession,” you ask?

An unsession is akin to an unconference, in which attendees set the agenda and control the sessions without any prespecified topics or formal presentations. The idea is for attendees to talk about whatever they want to talk about, however they want to talk about it. Think of the Predictive Modeling Unsession like a mini unconference inside Discovery Summit.

Here’s how it’ll work at this year’s Predictive Modeling Unsession:

Anybody interested in predictive modeling gathers on Zoom.
Attendees propose discussion topics and then rate their interest in those topics. Proposing a topic is optional, and the whole proposal and rating process will use online forms and take just a few minutes.
Topics will be ranked according to attendee ratings, and we’ll work our way through the topics starting at the top.

Topics can be anything related to predictive modeling. They don’t even have to be about JMP! A few examples: “Peeking inside the neural net black box”, “Tips for setting decision thresholds”, “XGBoost has lots of hyperparameters. Which matter most?”. But don’t take these examples too seriously. Topics are up to you.

If your topic comes up, you’ll be given the floor to kick off the discussion. Don’t be nervous: this isn’t a formal presentation. (After all, that’s what the rest of the conference is for.) You could just pose a simple question and ask for thoughts. You could demo something you’ve been working on in JMP and solicit feedback. You could share a quick tip you find valuable and give others the chance to ask questions or share their own tips. It’s up to you.

Is this unsession for me?

Yes.

If you’ve read this far, you must have at least some interest in predictive modeling, so this unsession is for you. Don’t know anything about predictive modeling? That’s fine! Come learn. Just want to sit in without participating in the discussion? No worries! You can just come and listen if you prefer.

One thing to note: if you want to propose a topic, make sure to be there for the 1 p.m. ET start time. All can come and go as they like, but for logistical reasons we’ll only accept topic proposals at the start.

The Predictive Modeling Unsession is your chance to talk about what you want to talk about with your fellow attendees. Come make the conference — or this session, at least — whatever you want it to be.

JMP developers tell all this fall – and answer your questions, too

gail_massari — Wed, 22 Sep 2021 15:32:23 GMT

Do you miss the JMP Developer Tutorials offered at JMP Discovery Summit?

Well, they are back, with a slight difference. Instead of being offered at the upcoming online Discovery Summit Americas, this October, November and December we are offering six Developer Tutorials as part of the Mastering JMP online series.

Although you’ll be on Zoom rather than sitting in a conference room with other JMP users, the developers will structure their presentations to include as much dialogue and Q&A as you find useful. We’ll hold sessions at a time – 11 am US Eastern Time – that we hope is convenient for users in the US, UK and Western Europe. And, of course, all JMP users worldwide are welcome.

The developers still aim to give you insight into the workings and statistics behind JMP and to learn about issues facing you as users – issues that the JMP Development Team might address in future JMP releases. And, they will answer your questions about why and when to use the approaches they have integrated into JMP and/or JMP Pro.

The first session on Handling Covariates Effectively When Designing Experiments will be presented by Ryan Lekivetz () on Oct. 21.

I spoke with Ryan as he was planning his session. He shared how he decided to become a statistician when living in Western Canada and how JMP subsequently lured him to North Carolina. I think you’ll agree that his ability to make design of experiments (DOE) understandable to even a first grader is remarkable. Please enjoy his short interview (audio only). But watch out, his creativity is highly infectious.

Want to learn more from Ryan?

Chocolate Bar Smackdown – US vs Canada
Eggstra! Eggstra! A new designed eggsperiment - JMP User Community
Father's Day fun with toy cars and DOE - JMP User Community
How to create an experiment design that is robust to a linear time trend - JMP User Community
Journal of Quality Technology mentioned in the interview
Natural Sciences and Engineering Research Council of Canada mentioned in the interview

Intel’s Don Kent on what makes a mature analytics organization, the importance of mentorship and joining the “JMP family” at Discovery Summit

Meg_Eberle — Mon, 01 Nov 2021 12:53:02 GMT

Don Kent works for Intel Corporation’s Technology Development Group as manager of a full-stack data science team. Prior to joining Intel, he served as Principal Data Scientist at IMFlash Technologies, a joint venture between Intel and Micron.

With his background in engineering and solid-state physics, Don has been a part of the semiconductor industry for two decades and a JMP user for nearly as long. An outspoken advocate for analytics for many years, he helped found the Wasatch Front JMP Users’ Group and co-hosted countless JMP meetups at IMFlash.

I reached out to to learn about his analytics journey and the advice he gives to newcomers in industrial statistics.

Meg: You’ve been a JMP user for almost 20 years! In that time, you must have seen quite a few changes not only in the software but also in the field of industrial statistics more broadly. How do you think your personal analytics journey ultimately shaped your career path?Don Kent is manager of a full-stack data science team at Intel.

Don: My advocacy for analytics has largely paralleled my journey of career growth. Looking back, it started with a keen interest in learning more about statistics. As a novice, the challenges were at the personal level, focused around finding mentors and making connections with like-minded folks. Much of the time in my earlier years was spent with user groups or other communities, both in contributing and in learning as much as possible.

It’s interesting how, as you look back, you can see a few key milestones that stand out. For me, those milestones are related to someone encouraging me to learn and to grow my skills – either from one of the amazing mentors I’ve known or, for example, from a manager who encouraged my journey with a complete boxed-set of JMP 4 books (which are now PDFs!).

As my knowledge, responsibilities and confidence in statistical methods have grown, so too have the opportunities I’ve had to impact the organizations I worked in. I see a clear correlation between my analytic growth and the career opportunities I’ve been afforded.

I started my career with Intel as a reliability engineer with a background in semiconductor device physics. However, through my interest and study of analytics over the last 20 years, I've been able to transition my career to managing a data science team within one of Intel's Technology Development groups. That opportunity to both create and lead an analytics team would not have been possible without the mentorship of others and the confidence of my current manager.

As I've grown in skills and knowledge, the dynamics have shifted from personal advancement to more interpersonal advancement. Now I find more rewards in mentoring and encouraging others in their journeys.

Meg: Yes, mentorship is absolutely key – no matter what role or industry we’re in, we can always learn from those who have walked the path before us and encountered similar obstacles along the way. Now that you’re in the position of mentoring others, what would you say are some of the primary hurdles you’ve learned from?

Don: One that comes to mind is related to spatial pattern analysis. At the time, I was working on a project at a former employer, and I remember going to my manager and showing him a JMP training course called "High Dimension Data Analysis." I explained how I thought that course would significantly increase my knowledge in this area and help with the project. Unfortunately however, my manager highlighted that at the time the company did not have the budget for the training.

I was of course dismayed, but after some consideration I realized that my education was in my own hands. So I paid for the course myself and took the class. It was one of the best courses I've ever taken, and it significantly helped that project. Even more importantly, coming to that decision was a real turning point in my career. It helped me focus in on what was important and helped bring clarity to the direction I ultimately wanted my career to go.

Meg: That provides me with a nice segue to ask you about something we often don’t talk about much, which is the wealth of resources JMP offers and the hidden value it brings to our customers. Can you speak a bit about how you’ve leveraged those learning resources and opportunities to interface with the JMP organization over the years?

Don: JMP really has the entire ecosystem to enable an individual to get involved at many different levels. I personally have utilized so many of the features of the JMP ecosystem that it’s hard to really overestimate the value, so to be brief I will mention one.

For those who have never attended a Discovery Summit – I just can't recommend it enough! The JMP team actually feels like a family, and they bring you into that family, which is a pretty amazing experience. This is all on full display at Discovery. Every time I’ve attended the Summit, I’ve taken home not one thing but many things, and I consider it a key conference. The access to tutorials, trainings and presentations is amazing.

The JMP team also offers opportunities to visit with the developers directly. They love to hear from customers and understand how they can make the product just a bit better. Of course, the keynotes are always by current thought leaders, and I personally treasure that one dinner sitting and visiting with Stu Hunter as one of the highlights of my many wonderful Discovery experiences.

Meg: That is so wonderful to hear! Speaking from the perspective of someone inside of the JMP organization, I have to say that I too thoroughly enjoy Discovery Summits. They are a real highlight of my job.

One of the most common objections we hear from newcomers to JMP is that some fear statistical approaches will supplant domain expertise. How has having JMP changed the way you and your colleagues apply domain knowledge and skills?

Don: Within our organization, the Technology Development team is eager to utilize JMP for many reasons – not least of which is that it significantly helps them complete their tasks. In fact, I would venture to say that the concern from our side is that the team needs to combine their expert knowledge of the semiconductor process with the correct JMP platform (and statistics) to answer a desired question.

Due to certain semiconductor hierarchies – for example, die within wafer within lot – certain data analysis can go a bit sideways if the proper hierarchy (or expert knowledge) is not correctly utilized. For example, using the proper lot-wafer-die nesting structure in a variance components analysis can make a large difference in the result.

Also in data mining. Choosing full lots (all wafers) for training and/or validation data sets will reduce the likelihood of information leakage when data mining at wafer level. We have to understand these complexities and combine expert knowledge with the correct JMP platform and statistical approach. As our statisticians like to say, "When in doubt, consult your local statistician."

Meg: Speaking as that statistician or go-to person for analytics support, what advice would you give to someone looking to cultivate a more mature analytics culture in their organization?

Don: What does a mature analytics culture look like? For us, we look at the culture across a few vectors – and the key is the team – the individuals in the organization must have the analytics knowledge or skills to enable them to perform their jobs with confidence and efficiency. This requires training and the knowledge to best use the tools they have, tools like JMP. They don't need to know everything. And that is where the analytics team can fill in the gaps.

Another highlight of a mature organization is the availability of KPIs and metrics for the organization. These should be standardized, available and visible across the enterprise to help direct the entire team. Finally, look for advanced applications that involve machine learning or utilize recommender engines – organizations that are utilizing these advanced tools have evolved data ecosystems, whereas organizations that are not quite there should work on improving the fundamentals like data integration, pattern/feature identification, automation and predictive analytic systems. Of course, having a dedicated team can significantly help drive these initiatives.

Meg: Speaking of new initiatives, introducing new analytics workflows can be disruptive at first. But, as we know, the benefits of an effective analytics strategy outweigh the initial costs. What kinds of ROI or improvements have helped to build and sustain momentum for data initiatives at Intel?

Don: In terms of adding value to the organization, the first step is generally a gap analysis – understanding where you, your team and the organization currently are compared to the direction they should travel. “It's a journey not a destination” can be a tremendous motivator and shine the light on where improvements and efforts will have the most value and help define the ROI.

Identifying those gaps can be time-consuming but, in my experience, involves meeting with and listening to customers and business partners. As an analytic organization, it is our job to translate the business need into a data strategy or data problem. By engaging and listening to our customers and partners, we are empowered to work with the organization to help synthesize the best analytic strategy. Once that strategy is determined, then executing on it becomes the focus. This is where engineers and statisticians excel and can shine.

One motivator that builds momentum is progress toward goals. Simple things like successfully deploying an add-in or adding a feature that was requested by a customer builds trust and progresses the organization’s goals. Often to successfully deploy these features requires partnering with other teams like the training group, who can help with professional documentation and communication. Metrics can also be powerful tools to gauge momentum and adoption and are used extensively to help understand and sometimes shape customer engagement.

Meg: As we wrap up, what advice would you give to someone who is just starting out with JMP?

Don: Be curious, use the "Help" section of JMP – and do so frequently! The documentation within JMP itself is the best place to begin, and the sample data sets and books are a treasure. From there, check out the resources online at jmp.com, check the training schedule and get engaged with a users’ group – try virtual if in-person is not available in your area.

Joining by datetimes

wendytseng — Tue, 07 Sep 2021 14:41:02 GMT

Why would you want to join data sets by datetimes?

One of the common data preparation challenges scientists and engineers face is matching different data sets by dates, times or datetimes (also referred to as timestamps) (Figure 1).

Figure 1. Example of dates, time and datetime data.

Scientists and engineers often get their metrics from a variety of different measurement systems / pieces of equipment. To analyze their data, they need to combine all the metrics in a single table so that they can look at relationships across all the metrics.

When you have a unique identifier column or set of columns (e.g. Batch_ID, Lot_ID, UPC, serial number, etc.), joining is relatively simple and can be accomplished using the Tables > Join function in JMP or the virtual join function in JMP.

It is more challenging when you need to align dates, times or datetimes between data sets. Often the task involves sensors that track metrics continuously over time (for example, temperature, pressure, pH, glucose). To understand how these metrics may impact product endpoints (sometimes referred to as “critical quality attributes”), it will be necessary to create summaries* of those metrics over appropriate time periods. The summaries are then joined with the corresponding product endpoints.

Let’s work through a case study showing how to accomplish this type of joining.

Case Study

Watch the video, or read about the case study below the video.

The process we are trying to understand and improve makes a finished product called “Forever Young Elixir.” The critical quality attribute is Potency.

A product is being made in two steps. First, the raw materials go into a machine that applies pressure to the raw materials. Then, the assembled raw materials move to a second process step where temperature is applied to create the final product.

Both process steps can be controlled, but today there is a fair amount of variation that we believe is impacting the finished product’s critical quality attribute potency. We are interested in understanding how pressure and temperature impact potency of this product so that, ultimately, we can identify an acceptable operating range for each of the process steps. There are temperature and pressure sensors that monitor the process continuously.

We will focus on temperature in this example because the process for working with the pressure data is analogous.

We have a data table that has each lot, the timestamp (datetime) for when the batch was finished and its potency (Figure 2).

Figure 2. Potency data collected for each finished batch along with the timestamp for batch completion.We also have data from a temperature sensor. Temperatures are recorded at one-minute intervals (Figure 3).

Figure 3. Potency data collected for each finished batch along with the timestamp for batch completion.

Ultimately, to correlate Potency with Temperature, we need a table with each batch’s Potency and the average* Temperature during the period when the product was made.

Figure 4. Final analysis table with temperature data combined with potency data. *Other summary statistics could be considered, or one could also explore using the Functional Data Explorer platform which uses all of the data to understand a process instead of summarizing to a single value.

Before we can get into the software, we need to decide which time points in the temperature step are appropriately associated with a particular batch. In other words, when was the batch being processed in the temperature step?

In discussing with other engineers, we determine that 5 minutes is the length of time a batch is processed in the temperature step. Therefore, the temperature data for the period 5 minutes prior to the batch’s finish timestamp should be used to correlate with the potency of a batch.

Now we have the necessary information to start the data preparation process.

Note: This example starts with the datetimes already properly coded with correct data type and modeling type. This is a very important step if you want the joins to work correctly. Please jump to the appendix for this blog post if you need a refresher on using dates, times and datetimes.

Video Demonstration

Join in on the joining! Download the two data sets attached below and follow along. If you don’t have JMP already, you can download a 30-day trial.

Review of Steps

Decide what range of sensor data readings is relevant to the product.
- How does my process impact the end point metrics?
- For each process, what is the time range of sensor readings that is relevant to the final product?
- For each process, what is the time range of sensor readings that I want to correlate to the final product?
Create a column in the data set with the end point metric(s) that defines the first timestamp for each record that is relevant.
- Do this by using a formula column using the finished product’s timestamp (or other reference timestamps if that is not available).
- You may need to define the last relevant timestamp as well.
Join the table with the end point metric(s) to the sensor data using Tables > JMP Query Builder.
- The join criteria need to be edited to include two conditions: one comparing the sensor timestamp to the finished product table’s timestamp, and one comparing the sensor timestamp to the first timestamp calculated in the second step.
Summarize (that is, average) the sensor metric in the joined table using Tables > Summary.
- Select the sensor metric(s) and then the Statistic desired.
- Group by all the other columns.
Use Graph Builder or Fit Y by X to look at the relationship between Potency and Average Temperature.
- Where is potency the highest?
- What can this tell you about where to set the operating range for temperature?
- Consider using modeling techniques so that you can incorporate all of the other process inputs as well.

Please comment on this blog post and let me know if you followed along. I want to know if you found the instructions clear and if not – what I can clarify. Also, what situations do you have that are not being addressed here?

Appendix

Review of Dates, Times and Datetimes

Let’s do a quick review of this type of data because it is important to get the data type and modeling type correct in all the data tables if you want the data to be joined correctly. Analytic software stores dates, times and datetimes in seconds from a reference date. JMP stores dates, times and datetimes as the number of seconds from 1/1/1904. There is another great blog post on Using Dates, Times, Datetimes and Durations if you want to go deeper on this topic.

If you want to use this type of data (to graph, subtract dates, join via dates), you need to make sure the dates/times/datetimes are formatted as Data Type = Numeric and Modeling Type = Continuous. When you import the data, make sure you do so. For special types of date time formats, you may need to apply a custom import format to let JMP know that it is a datetime.

Once you format the data as Numeric and Continuous, you will see that the data is represented in seconds. It may look strange to you, but it is correct.

Applying a format is what makes the data look understandable to yourself and others. You can choose from Date, Time, and Duration formats in Column Info (right click on column header > Column Info > Format). Datetimes (timestamps) are in the Time menu (Figure 5).

Figure 5. Format options for Dates, Times and Durations for a column.

Aside

If you want to join two data tables together via datetimes, make sure you have the level of detail necessary to do so in the datetime columns in both tables. The level of detail is determined by the source data. The format does not change the granularity of the data as it is recognized by JMP; it simply changes the granularity visible to us in the cell.

Let’s consider two scenarios:

If your data only has granularity at the date level (no time), applying a timestamp format (e.g. m/d/y h:m:s) will show the m/d/y and then 12:00:00 AM for the time for all records. There is no time detail in your source values, so you cannot compare times within the same day.
If your data has detail at the datetime level, applying a Date format (e.g. just m/d/y) will show m/d/y in each cell; however, the underlying granularity will remain (Figure 4).

Figure 6. Timestamp columns with different formats but the same granularity.

Contrast the two columns (Timestamp and Date) in Figure 7. The same format is applied to both. However, Timestamp has granularity that Date does not, and you can see that in the header graphs in the data table.

Figure 7. Timestamp and Date have the same format but different levels of granularity.

VSCode and JSL: a different scripting environment

vince_faller — Thu, 02 Sep 2021 16:52:44 GMT

TLDR; There's a VSCode extension that understands JMP. Or inside VSCode extensions, you can just search JMP, and as of this publication, it's the only thing that comes up.

Hello JMP world,

Have you found yourself wanting your JSL scripting environment to mimic more of an IDE? Well, if you haven't checked out Visual Studio Code, I would say now is a great time to look into it, because we, at Predictum, have been writing an extension for it that will let VSCode understand (kind of) JSL.

First, I'm going to go into some of the native features of VSCode that I liked enough to want to do this in the first place.

Workspace specific configs - You can open a folder with code, and it will remember it as a workspace, so you can just Ctrl+R and switch from project to project.
Git built in - It automatically has all the git features you would normally want. And for the stuff that it doesn't have, someone has probably written an extension for it. Which brings us to...
Extensibility - There are TONS of extensions to do all sorts, different languages, SQL Tools, cloud (AWS, Azure) integrations, GitLab/GitHub, you get it.
It's lightweight.

We loved all those things, so we decided to make it understand JSL. Obviously, you need a copy of JMP for all of the features to work right, but you're here, so I feel that's a safe assumption.

What you get

We have:

1. Syntax Highlighting - the most obvious thing we would need

2. Hover Over Descriptions

These are pulled from the scripting index of your JMP if you have it up. So if you use NewCustomFunction(), it will pull in functions from other add-ins as well.

3. Useful Snippets

The function snippet in this example also comes with Natural Docs syntax because all functions are better with good documentation. We've definitely found that our documentation has been a lot better since making this add-in.

4. Running JSL

You can press Shift+Enter to run your selected text, or the whole file if nothing is selected. Or you can press Alt+Enter to run just the single line you're on. We went with the VSCode standard for running instead of Ctrl+R because it felt like it flowed better with other extensions. Also Ctrl+R is already taken. But don't worry, VSCode lets you edit pretty much everything, so you can set anything to any keybinding you like. It will freeze up your current instance of JMP a little, but if you press the "close VSCode server" button, you should be good again.

5. Syntax Error Reporting (recently added)

I love this because it will look through all the files you have open and give them a big red title if any of them are bad. So if you have some nested include()s, you don't have to search through which file is actually the problem. Can't say enough how useful this ends up feeling.

6. Symbol Outline

This is probably personally my most used feature. VSCode knows when you made a function, class, method, variable, whatever and will give you an outline of your current file. This is great because you can just surf the outline to look for what you want if the files are too big. Or you can just press Ctrl+O and type the variable and it will find it. The main reason this is better than Ctrl+F is that it will find ONLY where you assigned it. Not when you called it, or used the same name somewhere else in a different function ( it happens ). And again, you have full control on what types of variables you want in the outline. No functions? No problem.

Cool. You're sold. Now what?

Download VSCode (if you haven't already)
Go to extensions (Ctrl+Shift+X by default)
Search "JMP" and you should see something like
Press install. Done.

We're actively working on it, but updates are a little slow. If you have any issues, feel free to log them in the repo. And as a shameless plug, if you know TypeScript (or just want to learn it) and are interested in contributing, we'd love the help. It's open source. It's just been myself and working on this in our spare time, and lately just Ben.

Statistician Kira Alhorn on how Gore derives business value from providing feedback and new feature ideas to JMP Development

Meg_Eberle — Thu, 19 Aug 2021 13:00:00 GMT

Statistician Kira AlhornKira Alhorn, PhD, works as a statistician at W. L. Gore & Associates GmbH in Germany. With expertise in optimal experimental designs for model averaging estimators, she supports domain experts across R&D and engineering, consulting on exploratory data analysis, design of experiments and statistical process control, among others.

I spoke with @Kira_Alhorn about Gore’s analytics culture and how she and her colleagues have helped influence and shape new features in JMP.

Meg: I know JMP users at Gore have been really active in collaborating with JMP Development to make the software better with each new release – essentially to make sure that new features align with what users actually want.

How do you go about providing feedback? Does this kind of collaboration have an impact on the value you get from the software?

Kira: Having a direct connection to JMP’s developers is a great opportunity to share our wish list and suggestions for future JMP releases. We’ve been meeting developers at Discovery Summits, sharing our feedback in the JMP Community as well as communicating directly via email (or, before COVID, also in face-to-face meetings).

The developers are always open to our comments and – at least within my time at Gore – have solved most of the questions that we’ve had so far. There are quite a few platforms in JMP that have been influenced by statisticians at Gore, e.g., the MSA platform and Custom Designer (not me, though!). Overall, having the possibility to give feedback and seeing it applied in future releases increases the value we are getting from JMP.

Meg: That’s great to hear! Another one of the things I’m curious about is how you’ve utilized all the free learning and networking resources JMP offers – things like the online user Community, for example.

Kira: At Gore, we interact with the JMP organization in many ways. Attending Discovery Summits helps us to discover new trends and features in JMP and to learn about new opportunities to use statistical methods. The interaction with JMP staff and users broadens our horizons and makes us rethink our current approaches to data analysis, design of experiments and statistical process control.

Within the JMP Community, we can help each other and learn about problems other associates are facing. Finally, STIPS has helped us a lot during the pandemic when our statistics trainings have had to be done virtually. We’ve been using parts of the STIPS course as e-learnings for our classes.

Meg: It’s interesting you mention STIPS (our free statistical thinking course). I’ve heard from a lot of JMP users that STIPS has been a great resource for domain experts in particular to hone their statistical skills. In your experience, how has having JMP changed the way domain experts at Gore apply their knowledge and skills?

Kira: Statistics has a long tradition within Gore’s technical functions. Bill Gore, the founder of W.L. Gore & Associates, applied statistical experimentation and analysis himself (in fact, he also published a book titled Statistical Methods for Chemical Experimentation).

So we at Gore already have a strong belief in the application of statistical approaches to complement domain expertise and to enable better decision making. However, applying statistics can be very cumbersome when you don’t have the right software on hand.

Since JMP is intuitive and easy to use, it enables all of our associates to explore and analyze data easily, without needing excessive training to do so. Graph Builder, with all its powerful options, makes sharing results easy and interactive – and supports domain experts in making the right decisions based on both their data and domain expertise.

Meg: What advice would you give to someone looking to cultivate a more mature analytics culture in their organization?

Kira: Establishing a mature analytics culture in any organization is a long and sometimes rocky road. Before being able to do fancy analytics with your data, you first need to have the data available in an accessible and clean format. However, the way to clean and make data accessible might be tough, so keeping a vision and the end goal at the front of your mind will help. You should be able to answer the question "What will change by making data-driven decisions within the company?"

Additionally, know that at times, not everyone will be able to envision the possibilities and opportunities that analytics can bring. Be prepared to just showcase some examples of what can be done using analytics – even if they don’t think they will need it!

Meg: How about advice for someone who’s just starting out with JMP?

Kira: Don’t be afraid and just try it out! Basically, you can click, drag and drop everywhere – there’s not a lot that you can do "wrong." And don’t forget to search behind the red triangle if you can’t find what you are looking for!

Steve Hampton of PCC Structurals on transitioning from Minitab and why it’s so important to take advantage of the JMP resources for engagement and learning he calls an “immense value-add”

Meg_Eberle — Thu, 05 Aug 2021 15:10:56 GMT

Steve Hampton is Process Control Manager for PCC StructuralsProcess Control Manager Steve Hampton has made a career of introducing Lean Six Sigma initiatives in the casting industry. After joining the investment casting manufacturer PCC Structurals fifteen years ago as an engineer, Steve recognized the potential of statistical approaches to better meet the metallurgical and dimensional needs of customers on reduced timelines and at lower costs.

Within his current purview of process control, Steve says it’s critical to think not just of one aspect of a process, but the entire value stream. Early changes in any process can have knock-on effects and by taking a broad view of the entire analytics workflow, he has been able to introduce targeted improvements that reduce rework, defects and costs downstream.

In a recent conversation, @shampton82 shared his thoughts – and some advice – on how he’s gotten more value out of JMP, both for his organization and his own career.

Meg: Introducing any new method or tool can be disruptive at first – particularly to an organization that sees itself as already working with the most cutting-edge tools on the market. But that doesn’t mean there aren’t overwhelming benefits to some of those transitional growing pains! What kinds of ROI or improvements have helped to build momentum for the data initiatives you’ve introduced at PCC?

Steve: This question reminds me of the activation energy curves from my school days. Initially, there is a huge energy barrier for a reaction to take place, but by adding in an appropriate catalyst, that energy is greatly reduced. I see JMP – and the support structure that comes with the software – as that catalyst for our company.

At PCC, we’ve had a long history with NEI and the DMAIC mindset, with Minitab being our main stats package. But we were also pretty stuck in an ANOVA and bivariate regression analysis rut. Even worse, with the limitations of Minitab and Excel, engineers would have to hop between the two multiple times during an analysis, and many just ended up using Excel because that was what they knew and could be efficient with. The result was some OK visualizations of our data and processes but a backslide on statistical rigorousness around decisions.

At the time when I was first introduced to JMP, I was a Minitab superuser and, to be honest, didn’t immediately see the value. Nevertheless, once I made the switch with help from @Jordan_Hiller’s onsite training and a few excellent on-demand webinars, I never looked back. Everyone who has made the transition feels the same way.

The turning point for my plant was a pretty large process excursion where we weren’t getting anywhere using our typical tools – but it was costing us hundreds of thousands of dollars as the problem persisted. The project team went back to basics with process mapping and brainstorming and started using JMP’s powerful visualization tools to quickly test theories in a group setting. Then we could run a statistical test to see if they were worth pursuing further. This led to a rapid understanding of what the main experiment priorities should be and a fundamental shift in what we prioritized in controlling our process.

Out of that, not only did we solve our process excursion (that was initially focused around a few parts), but we also saw improvement across all parts. The speed at which JMP allowed us to get to actionable items and the flexibility in distilling down complex issues with Graph Builder and the Profiler allowed PCC’s upper management to see the value of JMP for our company. Just this year, we are now training all our technical rotation personnel with the STIPS course to ingrain this thinking and the ability to use JMP from the beginning of their careers.

Meg: You’re describing two different kinds of newcomers to JMP. One is a person, maybe fresh out of university, who hasn’t yet used tools like JMP in an industrial workplace. The other is more advanced in their career and already has well-established preferences when it comes to statistical tools. And as a former “Minitab superuser,” you sound like you were previously in that second category!

One of the most common objections we hear – most often from that more advanced-career group – is that some fear statistical approaches in JMP will supplant their domain expertise. How has having JMP changed the way you and your team apply domain knowledge and skills?

Steve: Graph Builder and the Profiler are the Rosetta Stone for domain experts and JMP nerds. The capability for dynamic visuals basically allows domain experts to see what the data is saying across different settings and in different graphical environments. That’s really the “ah-ha” moment because they see what they have in their head – and JMP allows them to explain the links between process elements very clearly to other team members.

So, in project team meetings, I quickly see our domain experts driving the conversation – “can we look at this factor versus that one” or “let’s overlay this and that” – and then start talking to the team about what they are seeing and what can be done.

Meg: Tell us about your relationship with the JMP organization. There are some people who just use the software – and that’s fine – but there are others who really engage with JMP as an organization. What kind of value, in your experience, have you gained from taking advantage of some of the resources JMP offers?

Steve: I’ve been using JMP now for about six years (and JMP Pro for two), but it wasn’t until a couple of years ago that I really got involved in the JMP Community. I have been pretty active on the JMP Community discussion boards (mostly asking for scripting help), attended the 2019 Discovery Summit (which was awesome!) and was lucky enough to get selected to present at least year’s Discovery Summit.

I also regularly watch the on-demand webinars, have gone through the STIPS training, and have taken the online DOE training courses, so at this point you could say I’m all in!

Everything I’ve listed is the immense value-add that most people don’t know about. Or maybe they don’t understand how much value there is outside of the JMP application itself. Having a local technical resource like Jordan who I can bounce ideas off of and who can do group training sessions for us is really a game-changer when it comes to a JMP user group growing and thriving.

Meg: How would you say your advocacy for analytics transformation has impacted your career trajectory?

Steve: I’m not sure I’d be in my current role as Process Control Manager right now if it wasn’t for my love of all things data and stats. I’d held multiple operations and engineering roles before I joined the Process Control team at PCC, but I believe what helped me land the role was my understanding of how data and data-based decisions can lead to step-function changes in the control and improvement of a process. That and my interest in converting others into stat nerds!

It has also help me with exposure to other plants, networking across divisions, and exposure at technical conferences as I help others with their data analytics problems and present on what we’re doing new at my plant.

Meg: What advice would you give to someone looking to cultivate a more mature analytics culture in their organization?

Steve: I think the No. 1 thing is to force people to take the time to learn. Starting out, trying to understand data analytics, and using a new program like JMP requires a manager to say it’s OK to go slower in the short term with analytic problems so that people feel empowered to actually learn rather than resort to their old ways. When the house is on fire, people won’t naturally grab the tool that will slow them down at first, so that’s our job as change agents to let them know it’s OK to go slower at the outset to end up going faster and farther overall.

I also would recommend finding someone with a passion for data analytics and empowering them to have time to share their passion and be a contagious force. They can encourage others to take every available opportunity to try new ways of exploring data and having it lead to actionable items.

Finally, you need to share wins and make sure people are not reinventing the wheel on best analytical practices – so user group meetings, tech portals, and best practice report-outs are critical.

Meg: What advice would you give to someone who is just starting out with JMP?

Steve: Here are my top 5 tips:

Take advantage of the free resources JMP has to offer.
Dedicate yourself to using JMP enough to develop a “button” muscle memory.
Make the platform more efficient by using custom toolbar buttons and scripts for frequently used actions and analysis.
ALWAYS visualize your data before you try to do any analysis.
Get involved in other people’s problems. You’ll be amazed at how much you learn exploring data you’re not used to and questions you’ve never thought of how to answer before.

What's new in JMP Live 16.1

MichaelGoff — Fri, 30 Jul 2021 19:01:42 GMT

We are committed to continually improving JMP Live, and JMP 16.1 is no exception. In our latest release, we have focused on improving the experience around administrating your JMP Live instance. Outlined below is just a sampling of the great changes we have made to JMP Live.
Download JMP Live content as a JMP Project

You can now download JMP Live content as a JMP Project, which allows you to download a report (or all the reports, if downloading a folder) and the data tables associated with the report at one time. When you open the downloaded file using JMP, you will find each report as tab in the Project. The Project also includes a script that allows you to quickly open the post back up on JMP Live. Note: The downloaded Project is only supported by JMP 16.0+.

The Download as JMP Project button is found on the toolbar of a post:

Display name sync

Prior to JMP Live 16.1, you had to set your display name when logging into you account for the first time. Now, JMP Live provides an option to retrieve display names from the authentication provider. To turn on the feature, admins may now set the following option in the ENV file:

AUTH_DISPLAY_NAME_SYNC = 1

The display name sync has different restrictions depending on the authentication provider used with JMP Live. If the instance is authenticated using Direct AD, the display name sync will look for the field displayName as the assumed display name for the user and will use it if present.
Keycloak Authentication is a bit more flexible than Direct AD. Keycloak allows mapping attributes from authentication providers to a set of common attributes for JMP Live. Once the desired attribute is mapped to displayName within Keycloak, JMP Live will use that for display name syncing.

Turning on display name sync disables a user’s ability to manually edit their display name in JMP Live.
Import users from Active Directory
If your JMP Live instance is using Direct AD or Keycloak AD, you will now be able to import users directly from Active Directory using the JMP Live CLI.

C:\Program Files\SAS\JMP Live\JMPLive> bin\jmplive-cli authImport -h
Imports different entities from the configured authentication service for use in JMP Live.

USAGE
  $ jmplive-cli authImport

OPTIONS
  -a, --all Import all entities from the authentication service
  -c, --count Count the number to process without importing
  -t, --type=users|groups (required) The type of entity to import

Admin panel refresh

In JMP Live 16.1, we have refreshed the look and feel of the entire administration panel. We wanted to make accessing information about the JMP Live instance more intuitive and allow for a more optimized experience on both desktop and mobile based formats.

Server status page

The server information page is a new addition to JMP Live 16.1. We created this page to be the default landing area of the administration panel and to provide a new home for the configuration settings. Along the top of the page, we have highlighted the number of enabled users, a list of JMP Sessions currently in use across the pool and the number of reports in the regeneration queue. Other settings of note in the cards below are the SSL Certificate expiration date and the database connection status to help identify problems with the JMP Live instance.

Post access table

The JMP Live admin panel also includes a new page that allows you to browse the view and download activity on every post in your JMP Live instance. Since this table is filterable, the admin can filter by users, access type, IP address and more, allowing an administrator to monitor more closely who is accessing resources on the JMP Live instance.

We also include a more limited version of the access table for individual posts in JMP Live, viewable by the post’s publisher. This table is found in the overflow menu of a post’s page.

Refresh existing admin panel

In addition to adding new pages to the admin panel, we updated the existing pages to match the look and feel of the rest of JMP Live. Here are some of the highlights.

User management
The user management page has gotten a totally new layout in JMP Live 16.1. Now we provide a list of users on the left-hand side of the page and allow you to filter down the list so you can find a specific user quickly.
Clicking on the user in the list brings up the details for that user on the right-hand side of the page. The first section lists basic information about the user and allows an administrator to update the user’s display name if applicable. The next section allows an admin to adjust the user’s permissions for JMP Live.

Group management

Similar in design to the user management page, group management allows an administrator to adjust the settings for each group on JMP Live. After selecting a group in the list, an admin may adjust the basic information about a group or even delete the group if it is no longer needed. The next section allows you to manage the members of the group. You may add new members, delete existing ones, or adjust the permissions for individual members.

JMP Live settings

This page includes the runtime settings from the old admin settings page. The configuration settings are now surfaced on the new JMP Server Info page, which means that every setting listed on this page is editable while JMP Live is running. Each setting is individually editable and includes a pop-up description if necessary.

JMP Pool

The new JMP Pool panel highlights some performance metrics along the top of the page so an admin may quickly assess the state of the pool. Each pool machine is listed in the table with a summary of information about the servers within. However, if you need to manage an individual server, the list may be found in the pool’s details panel.

Audit log

The redesigned audit log panel allows you to explore the activity of the JMP Live instance. The filters are the same as the old audit log design, allowing admins to focus on specific activity, users, timestamps and more.

We have also redesigned the details column to pop up a JSON viewer window so you can explore the full data associated with each entry.

Keycloak CLI

With the update to JMP Live 16.1, we have introduced a new command line tool to help manage a Keycloak installation. This tool is designed to be run on the Keycloak server by system administrators. This CLI makes it easier to manage database settings and HTTPS certificates. The CLI supports a help command and a -h argument to display help information for a given command.

C:\Program Files\SAS\Keycloak\keycloak>keycloak-cli help
keycloak-cli can be used to configure your Keycloak instance using one of the
following commands. For more information about a command, you can use help with
the command name or use --help after the command name.

Usage: keycloak-cli [-h] <command>

Options:
  -h, --help Show this help message and exit.

Commands:
  database   Configures database settings for the Keycloak server.
  http       Configures HTTP settings for the Keycloak server.
  https      Configures HTTPS settings for the Keycloak server.
  help       Displays help information about the specified command

Exit Codes:
  2 Missing required command

Database

The database command helps an admin configure database settings for the Keycloak server. The connection command allows you to get the current connection information, set new connection information, and validate the connection. The create-schema command will create the Keycloak schema in the database.

C:\Program Files\SAS\Keycloak\keycloak>keycloak-cli database help
Configures database settings for the Keycloak server.

Usage: keycloak-cli database [-h] <command>

Options:
  -h, --help Show this help message and exit.

Commands:
  connection      Configures database connection settings for the Keycloak
                    server.
  create-schema   Creates the Keycloak schema in the database.
  help            Displays help information about the specified command

Exit Codes:
  2 Missing required command

HTTP/HTTPS

The https command helps to manage the HTTPS settings for the Keycloak server. The bind-address command allows a user to update the IP address that Keycloak listens on for incoming connections. The certificates command allows a user to update the certificates used by the server and validate the new certificate files before replacing them. The port command allows the user to configure the port number that Keycloak listens on for HTTPS connections.

C:\Program Files\SAS\Keycloak\keycloak>keycloak-cli https help
Configures HTTPS settings for the Keycloak server.

Usage: keycloak-cli https [-h] <command>

Options:
  -h, --help Show this help message and exit.

Commands:
  bind-address              Configures the IP address that Keycloak listens on
                              for incoming connections.
  certificates              Configures the HTTPS certificates for the Keycloak
                              server.
  port                      Configures the port number Keycloak listens on for
                              HTTPS connections.
  proxy-address-forwarding  Configures whether Keycloak accepts X-Forwarded-For
                              headers from proxies.
  help                      Displays help information about the specified
                              command

Exit Codes:
  2 Missing required command

Like the port command on https, the http port command allows you to configure the port number that Keycloak listens on for HTTP connections.

Moving from Minitab to JMP and JMP Pro: A Transition Guide

Larry_LaRusso — Mon, 26 Jul 2021 17:35:51 GMT

Easy to use statistical software has become a essential tool in almost all businesses as more companies seek to have their employees basing decisions on data. Comprehensive statistical analyses, building of sophisticated descriptive and predictive models, and mining data to discover insights are no longer just done by statisticians but by those across many functional areas within an organization. Academia is experiencing a similar need. There has been a substantial increase in the number of programs and courses across a wide range of disciplines exposing students to the tools and techniques of the statistical sciences. Interactive point-and-click software can greatly reduce the barrier for students to develop the necessary skills and enjoy the data exploration process without the need for programming.

This white paper explores two popular software packages – Minitab and JMP. Differences between the two will be illustrated with the goal of easing the transition for experienced Minitab users to move over to JMP. Focus is on the user interface, how tools are accessed, how analyses are performed, and the general workflow that a user follows to analyze data.

Minitab is a graphical user interface-based (GUI) statistical software program first developed in 1978 as a computing tool to support introductory statistics courses. Minitab v19, released in June 2020, is the version featured in this white paper.

JMP and JMP Pro are also GUI-based software, first developed by SAS in 1989 to support engineers, scientists and researchers in applying the tools of the statistical sciences in a nonprogramming, highly interactive environment. JMP and JMP Pro are used across a wide range of industries and academic disciplines. JMP 16 and JMP Pro 16, released in March 2021, are the versions featured in this paper.

As this is a transition guide rather than a comprehensive coverage of their individual capabilities, we are unable to cover all the differences between the two in this white paper. Visit minitab.com and jmp.com to learn more.

Topics covered in this paper include:

The full text of the white paper appears below.
(Note: You can jump directly to a particular section by clicking on the hyperlinks above.)

Introduction

Easy to use statistical software has become a essential tool in almost all businesses as more companies seek to have their employees basing decisions on data. Comprehensive statistical analyses, building of sophisticated descriptive and predictive models, and mining data to discover insights are no longer just done by statisticians but by those across many functional areas within an organization. Academia is experiencing a similar need. There has been a substantial increase in the number of programs and courses across a wide range of disciplines exposing students to the tools and techniques of the statistical sciences. Interactive point-and-click software can greatly reduce the barrior for students to develop the necessary skills and enjoy the data exploration process without the need for programming.

This white paper explores two popular software packages – Minitab and JMP. Differences between the two will be illustrated with the goal of easing the transition for experienced Minitab users to move over to JMP . Focus is on the user interface, how tools are accessed, how analyses are performed, and the general workflow that a user follows to analyze data.

Minitab is a graphical user interface-based (GUI) statistical software program first developed in 1978 as a computing tool to support introductory statistics courses. It gained much popularity in industry during the mid-to-late ‘90s as companies that were engaged in quality initiatives such as Six Sigma began using statistical techniques to improve the quality of their processes and products. Minitab v19, released in June 2020, is the version featured in this white paper.

Overview

While there are some similarities between Minitab and JMP, there are some fundamental differences that are important for one who is transitioning to JMP should know.

1. Interactive analysis output. When performing an analysis in JMP, the output is presented in an interactive window. All the analysis output created, including what’s created across different platforms, are dynamically linked to each other and to the data table. This allows the user to interact directly with the analysis output. Changes such as removing/adding variables to a model, switching the role of a variable and subsetting the data can be done directly from the analysis output without the need to rerun an analysis. Changes can also be made to the data table (e.g., adding/deleting data) and all the output can be set to automatically update.

Minitab output is predominantly static and is not connected across different analysis tools. To remove a variable from a model, for example, the analysis tool would need to be relaunched, those changes specified, and then the analysis rerun. In many of Minitab’s tools, making changes to the data will not automatically update analysis results.

2. Contextual menus. Each analysis output performed in JMP is connected with access to a set of tools related to that analysis. In this way, the approach to analyzing data is more exploratory and interactive in nature, with the results of one analysis guiding the user to a set of tools that are designed as next steps in the analysis workflow.

Minitab runs analyses in a more traditional batch mode, with the user often having to decide upon which analysis to perform entirely separate from seeing features in the data. When a tool is selected, a dialog box is open. Variables are assigned roles and options are chosen. Running the analysis produces a set of output that is typically static. The next step in a typical analysis workflow often requires a new tool to be launched requiring another round of specifying variable roles and options.

3. Defining data to be used for analysis. The typical approach in Minitab is to create a worksheet that contains all the data that a particular analysis will be based on. If it is desired, for example, to perform an analysis on a subset of that data, it is usually required to perform an operation (e.g., Subset, Split Worksheet, etc.) to create a new worksheet. If outliers are discovered an the analyst wishes to see the results without those data points, another new worksheet would typically need to be created and the analysis run again. This requires the need to manage many worksheets in a typical analysis session.

Though one is also able to work with multiple data tables in JMP, it can often be easiest to use Data Filters and Row State operations to define the data to base a specific analysis on. These data filters and row states can be saved and accessed at any time, which allows the analyst to have a single data table upon which all analyses on various subsets of that data can be performed.

In addition, the Local Data Filter tool in JMP allows users to filter the data even after an analysis is performed. For example, a linear regression model can be create from all of the data in a data table. From within the regression analysis output, a local data filter can be launched, which instantly changes all the results for different subsets of the data selected.

4. Simultaneous multiple analyses. A key feature of JMP is its ability to perform a large number of analyses simultaneously. In nearly all analysis platforms within JMP, a By Variable option is available, allowing analyses to be performed for each level of that variable. For example, if one wanted to perform separate hypothesis tests comparing two means for each of 10 different groups of a categorical variable, then that By Variable simply needs to be specified. Minitab would require running that analysis 10 separate times.

JMP platforms also allow you to run a large number of individual analyses together by simply selecting all the response variables and factors. For example, if one wanted to perform every possible one factor analysis for three different response variables and five different explanatory factors, all 3x5=15 different analyses can be performed in one step, including different one factor analysis techniques based on the data type for the response variables and factors. ANOVA, Linear Regression, Logistic Regression, and a Chi-Square Analysis can all be performed simultaneously with the results displayed in the same output window. Minitab would require running each of these analyses separately using different tools.

JMP has unique platforms designed for executing a very large number of analyses together with custom graphs and output created. For example, a multivariable capability analysis can be performed on 50 different process variables. In addition to producing output for 50 different individual capability analyses, unique graphs and numerical output will be created, standardizing all the individual results, to allow for easy comparisons across these 50 variables – a true multivariate analysis.

There are also specialized platforms in JMP custom designed for applications when an extremely large number of analyses are needed. In certain scientific fields, it’s quite typical that one may have, for example, a thousand different response variables and a single explanatory factor. A response screening analysis can be performed resulting in 1,000 separate one-way ANOVAs. The results (statistically adjusted based of the number of analyses) are displayed together using a custom set of graphs and data tables designed for easy comparison and follow-up analyses.

5. Integration. JMP has the capability to integrate with external computational environments such as SAS, MATLAB, R and Python. Through JSL (JMP’s Scripting Language), an analyst can leverage the power of those tools while also being able to work within the interactive GUI of JMP.
Minitab currently has the ability to integrate solely with Python, but not SAS, R or MATLAB.

6. Publishing reports. Both Minitab and JMP have tools to export and save reports outside of the software (e.g., .jpg, Microsoft PowerPoint and Word). JMP can also generate interactive .html reports, allowing others to view analysis reports outside JMP with interactive tools to drill down into the data further. Another product in the JMP suite of products is JMP Live, which provides additional capabilities for automating the generation and sharing of reports updating a dashboard as data and models change.

7. Statistical analysis techniques. Though both software contain many of the more common statistical analysis tools, there are myriad more advanced statistical techniques and data management tools available in JMP and JMP Pro that are not available in Minitab. JMP also typically has more options and capabilities within a particular statistical analysis technique. Visit www.jmp.com to learn more.

Datasets

Various datasets will be used throughout this guide. They can be found here.

File Format

One difference that an experienced Minitab user may first notice when learning JMP is that it is not necessary to work within a Project File structure.

Minitab Project File

Users of Minitab are most familiar with Minitab Project Files denoted with the extension .mpx. Within this Project File, multiple worksheets (i.e., data tables) can be imported or created. Figure 1 shows a sample Minitab Project File containing four worksheets. The display is split into three panes. The bottom-right pane shows the active worksheet (Full 4 Data) along with tabs at the bottom for each of the worksheets. The left pane, known as the Navigator, shows a list of the analyses and operations that have been performed (e.g., Graphical Summary of Yield and Moisture Content, … , Split Worksheet). The top-right pane is the output window displaying the particular analysis output that has been. There are three different views that can be chosen (Data and Output, Data only and Output only). In this figure, Data and Output view was chosen.

Figure 1 Minitab: Project File

Any analysis or operation chosen is performed on the active worksheet. If a user wishes to perform an analysis on a subset of the data, a new worksheet would need to be created from the parent worksheet. Here a set of worksheets were created, one for each of the four GeoLocations (A, B, C and D) using the Data>Split Worksheet command. To perform an analysis on just one of the GeoLocations, that worksheet would first need to be selected to make it the active worksheet before running the analysis.

JMP Data Table File

JMP can also be used from within a Project File structure. First, let’s take a look at how JMP data table files work since many users opt to work from that type of file. JMP data table files are denoted with the extension .jmp.

Figure 2 shows a sample of a JMP data table file. There are four panes to the file. The data table is to the right, the Row Pane is in the lower-left and provides a summary of the rows in the data table, including the number of rows that have been selected, excluded or hidden. The Column Pane in the middle-left displays the variables in the data table, identifying properties for that variable, such as the modeling type and use of a label. The upper-left pane shows the scripts that have been saved to this data table file. The steps to run and save scripts will be shown in a later section.

The red triangles, seen in the different panes, access a variety of tools to perform operations on the data table, columns or rows. Columns can be reordered in the data table by clicking and dragging the columns list in the Columns Pane to the desired location. A more comprehensive list of tools and operations that can be performed on columns can be found under the Cols menu or by right-clicking on a column in the data table.

Figure 2 JMP: Data Table

JMP has a collection of tools, which we will illustrate later, that allows users to perform analyses on specific subsets of the data. In many cases, you will find that working from one master data table in this way allows you to perform all the desired analyses without needing to create and manage multiple data tables, as is required in Minitab.

JMP Project File

JMP Project files are another file structure that can be helpful to work from. Project files provide a single interface to JMP, acting as a container for many different project elements. It can contain (or reference) data tables, analysis scripts, JMP journals, reports, as well as non-JMP elements such as documents, data files, etc.

Figure 3A is an example of a JMP Project file.

In this example, two panes are displayed on the left.

Contents: List of all the files contained within the Project file such as folders, JMP data tables, and other non-JMP files (e.g., .xls, .csv, .doc. .ppt, .jpg, etc.). Here we have a folder, two JMP data tables, a JMP Journal, two excel files, and one word document.

Workspace: Hierarchical list of the project contents that are currently being displayed. Here we have the JMP Journal and two JMP data tables along with an analysis report associated with a script in one of the JMP data tables.

A tabbed collection of windows, one for each of the elements listed in the Workspace pane, is organized to the right. Here we have one of the data tables as the active window. You should recognize this as the JMP data table shown in Figure 2.

Figure 3A JMP: Project File

Each of the panes, including any of the tabs, can be moved to reconfigure the display to a desired view.

Figure 3B shows an example of the Project with the tabs having been arranged so that the two JMP data tables and Journal are placed directly to the right of the Workspace and Content panes with one of them selected to be displayed. On the right side are two analysis reports organized with one placed above the other.

Figure 3B JMP: Project File

This tabbed configurable environment provides the user with the flexibility to reveal, hide, and display the contents of the project file in whatever way is best for the situation.

Project files can be shared and all of the contents, including non-JMP files, will be included.

Visit jmp.com, jmp.com/help or community.jmp.com and search for “Projects” to see a list of resources, including recordings, tutorials, documentation and user discussions, to learn more about the functionality of JMP Projects.

Importing Data

Often, data is produced within other applications, rather than inside statistical software. For example, an engineer may record data from an experiment in an Excel spreadsheet or a central database containing a large collection of customer behavior data from which certain subsets of that data need to be retrieved and analyzed. Both Minitab and JMP can open many different file types, including .csv, .txt, .xls, among others, and each is able to query a database via ODBC (Open Database Connectivity).

JMP

JMP is able to open other file types that Minitab cannot, such as SAS, R, MATLAB, .html, and .pdf files. JMP can also open tables that are built into web pages using the Internet Open functionality.

JMP also has added advantage of being able to import multiple files combining them into one data table. Figure 4 shows a folder containing a collection of different text files related to a survey along with the content of one of those files.

Figure 4 JMP: Importing Multiple Files

Selecting File > Open Multiple, will launch an import wizard that will guide you through the process of importing all of the data across these many files into one JMP data table.

To import data you currently have in a Minitab file into JMP, it is best to first save the Minitab worksheet in a format such as .xls, .csv, or .txt. Then simply choose File > Open in JMP to import that data into a JMP data table. An import wizard will be launched to specify how the data is to be imported (e.g., choose the variables and rows to import, include/exclude variable names, etc.).

Cleaning and Formatting Data

Both Minitab and JMP have a collection of tools that allow the user to clean and format data to prepare it for analysis.

Minitab

Most of Minitab’s tools to format, clean and prepare data are found under the Data and Calc menus. Figure 5 shows the list of tools under those menus.

Figure 5 Minitab: Data and Calc menus

Once a tool is selected, a dialog box will appear, allowing variables to be placed into different roles to describe the specific set of operations to be performed.

Figure 6 is an example of the dialog box for Minitab’s Sort Tool. Here we have chosen to sort all of the data in alphabetical ascending order based on the GeoLocation variable saving the data into the same worksheet (i.e., “In the original columns”).

Figure 6 Minitab: Sort Tool

JMP

Most of JMP’s tools to format, clean and prepare data for analysis are found under three different menus: Tables, Rows and Cols (shown in Figure 7).

Figure 7 JMP: Tables, Rows and Cols Menus

Figure 8 is an example of JMP’s Sort Tool. Though the dialog box has a difference appearance and uses different names for operations, the general procedure is quite similar.

Figure 8 JMP: Sort Tool

All of the data manipulation operations you’ve performed in Minitab will also be able to be performed in JMP. It will just take a little bit of practice to learn how the tools are accessed and how the desired operation is specified in the dialog box.

One JMP tool in the Tables menu (that is not available in Minitab) that can be quite useful in an initial examination of a data set is the Missing Data Pattern platform. This following example uses a small data set, but you’ll find this tool to be most valuable when examining large data sets. Figure 9 displays a data table with five variables, 25 rows and 18 missing values.

Figure 9 JMP: Data Table with Missing Values

The Missing Data Pattern tool under the Tables menu will summarize the “missingness” in the data via a new JMP data table. Figure 10 shows the resulting table along with the accompanying Cell Plot.

Figure 10 JMP: Missing Data Pattern

This new data table describes the Missing Data Pattern in the data. In particular, there are 15 rows with no missing values; one row has a missing value in variable Y5; four rows have one missing value in variable Y3; two rows have a missing value across 3 variables - Y3, Y4, and Y5 and so on. Highlighting a row in this Missing Data Pattern table will highlight all of those rows with that missing data pattern in the original data table. This allows you to hide and/or exclude that data from future analyses, or generate row states such as assigning unique symbols to be displayed in graphs. There is a related platform called Explore Missing Values under the in Analyze>Screening menu that offers additional tools for working with missing data, including statistical techniques to find commonalities between the missing pattern in the variables and rows as well the ability to impute the missing data replacing them with estimated values.

Creating New Variables

Both Minitab and JMP allow the creation of new variables. These new variables can be, for example, simple arithmetic operations using other variables in the data table or a logic conditional argument.

Minitab

New variables are created in Minitab through the Calculator Tool under the Calc menu. Figure 11 illustrates how a new text variable was created based upon the value of the variable ‘Moisture Content’ with the results saved in Column C9.

Figure 11 Minitab: Calculator

JMP

To create a new variable in JMP, access the formula tool for that new column by highlighting a column and choosing Cols>Formula. It can also be accessed by right-clicking on a column and selecting “formula” from the menu. In Figure 12, the formula tool was used to create the same text variable based upon a conditional argument. Note that there is a Preview function available that allows you to see the results of applying the formula as it’s being created.

Figure 12 JMP: Formula

JMP’s Formula Tool is also where new variables can be created using a patterned sequence, a random sample from other columns or a specified probability distribution. In Minitab, other tools would need to be accessed to generate those types of operations.

Data Filter and Row States

Minitab

When performing an analysis in Minitab, a worksheet containing the data for that analysis is made active. If an analysis is to be performed on a subset of a parent data set, most likely a new worksheet will need to be created that contains only that subset of data. If outliers are identified and need to be excluded from future analyses, another new worksheet would need to be created without those data points. As a result, it is typical that one has to manage many worksheets within a project file.

JMP

Multiple data tables can also be set up in a JMP Project file as was shown earlier. Data filters and row state operations can instead be used to define the specific data to base a particular analysis on resulting in many users opting to do all their analyses from one data table. These data filters and row states can be saved and accessed at any time.

We’ll illustrate these analyses using the Grain Yield data.

Figure 13A shows the data table with GeoLocations and C selected. This was done by clicking on the small histogram icon in the upper left of the data table to turn on the Header Graph, and then simply selecting those GeoLocations. The Header Graph is a convenient tool that displays high level summary of a variable at the top of each column (e.g., histograms and min/max values for continuous variables, a bar chart for categorial variables). Figure 13B shows the the result when the tool Rows>Hide and Exclude is selected. Any analysis performed with this Row State active will not include GeoLocations A and C in the calculations nor appear in any graphical display. Other options include using selected data in calcuations but not displayed in a graph (Hide only) or display data in a graph but not used in calculations (Exclude only). This row state can be turned off by again selecting Rows>Hide and Exclude or Rows>Clear Row States.

Figure 13A JMP: Rows selected

Figure 13B JMP: Hide/Exclude

The Global Data Filter is another useful tool for defining a subset of data. This tool is accessed by choosing Rows>Data Filer. Figure 14 shows the data filter open with the GeoLocations B and D selected. Here we turned on the check boxes Show and Include and then selected the GeoLocations we want to use. The data filter can include many variables, and both categorical and continuous variables can be used. This data filter can be saved to the data table, row states can be cleared and then accessed again at any time.

Figure 14 JMP: Global Data Filter

Other characteristics can be set up as row states. For example, you may want all the graphs produced in an analysis to use different colors for the different GeoLocations. This preference can be set up through Rows>Color or Mark by Column. Figure 15 is the resulting data table with the colors shown as row states. Any graphical display created with this row state active will use this color coding. This row can also be saved and accessed at any time.

Figure 15 JMP: Color Row State

The Local Data Filter in JMP provides another means to work with a subset of the data. This approach provides an interactive tool in which to see analysis results change instantly as different subsets of the data are chosen. The Local Data Filter is accessed under the top red triangle in an analysis report window. Figure 16 shows the results of a Regression Analysis using all the data (left), then that same output changing based upon selecting only GeoLocation A from the Local Data Filter. Note that the data points are using the color row state we had set up in Figure 15.

Figure 16 JMP: Local Data Filter

Descriptive Statistics

Both Minitab and JMP are able to calculate summary statistics that describing features in the data.

We’ll illustrate some of these analyses using the Grain Yield data.

Minitab

There are a few ways to generate descriptive statistics in Minitab (e.g, Display Descriptive Statistics, Store Descriptive Statistics and Graphical Summary under the Stat Menu). Descriptive statistics can also be generated using the Tables tool, which we’ll illustrate here. Figure 17 show the steps for calculating the sample mean and sample standard deviation for for Yield for each GeoLocation/Year combination and the resulting table.

Figure 17 Minitab: Descriptive statistics via Table

If different summary statistics are desired, or if different categorical and/or associated variables are to be used, the Tables>Descriptive Statistics tool is relaunched, the dialog box updated, and the analysis rerun.

JMP

JMP can also calculate descriptive statistics in a variety of ways, including the Distribution platform, Graph Builder, Column Viewer, and through column formulas. Similar to what was shown using Minitab, we illustrate how to calculate summary statistics through the Analyze>Tabulate platform.

Launching the Tabulate platform opens the window shown in Figure 18A. Variables are simply placed into different roles to create the table. Those variables can be moved to change the way the table is displayed. Variables can be removed, added, new statistics calculated, among other capabilities with the table updating the instant those changes are made. Think of it as a blank canvas that you create the desired table.

Figure 18A JMP: Descriptive Statistics via Tabulate

Figure 18B shows thes the steps to create the table displaying the mean and standard deviation of Yield for the 6 different combinations of Year and GeoLocation.

We begin by placing Yield into the Resulting Cells Drop zone. The default is to display the sum (the default statistic) of that variable as a single cell (upper-left image in Figure 18B). Note: You can choose to also display the mean and standard deviation by default via preferences (JMP > Preferences > Platforms > Tabulate).

In this case we will replace the sum with the mean and standard deviation by dragging those two statistics from the list on top of the cell where the word sum is shown. The table will updated, replacing the sum statistic with the mean and standard deviation (upper-right image in Figure 18B).

The categorical variables (Year and GeoLocation) are simply dropped into the desired locations. Year is placed as the rows (lower-left image in Figure 18B) and GeoLocation as the columns (lower-right image).

Figure 18B JMP: Descriptive Statistics via Tabulate

This table can be easily changed to create different views by moving variables into different drop zones. Figure 18C shows the resulting table when the variable GeoLocation is moved to be another row variable inside of Year.

Figure 18C JMP: Descriptive Statistics via Tabulate

This table of summary statistics can be made into a JMP data table for further analysis by selecting Make into Data Table under the red triangle. Figure 18D shows the resulting data table.

Figure 18D JMP: Descriptive statistics via Tabulate – Creating a Data Table

Graphing

Minitab and JMP approach graphing data very differently; it is one of the more significant differences between the two software that a new JMP user will experience.

We’ll illustrate some of these analyses using the Grain Yield data.

Minitab

To create a graph in Minitab, one must first decide which graph is needed and then select it from the Graph menu. Figure 19 shows a list of the graphs available under that menu.

Figure 19 Minitab: Graph menu

We’ll illustrate the process of creating a graph in Minitab by creating a scatterplot. When the scatterplot tool is launched, a screen appears (left image in Figure 20A). At that point, the user must pick one of the scatterplot options. In this example, Simple was chosen. A dialog box then appears to enter the variables to be plotted on the Y and X axes (right image in Figure 20A).

Figure 20A Minitab: Scatterplot

The buttons Scale, Labels, Data View, Multiple Graphs and Data Options provide additional options. Figure 20B shows the resulting graph if no additional options are chosen.

Figure 20B Minitab: Scatterplot

A few elements of the graph can be edited (e.g., changing the color of the points) by launching the Edit Graph tool under the drop-down menu in the upper right.

To make the scatterplot shown in Figure 20C that has a separate regression line for each of the 4 GeoLocations would require creating an entirely new graph. This would be done by relaunching the Scatterplot tool and selecting the “With Regression and Groups” from the dialog box shown in Figure 20A.

Figure 20C Minitab: Scatterplot

To create an entirely different type of graph with the same data (e.g., comparative box plots, etc.), then the user would return to the Graph menu, launch that specific graphing tools and specify the appropriate options. Each iteration of exploring the data graphically typically requires this approach until the desired graph is produced.

JMP

JMP approaches creating graphs very differently than Minitab. Though there are tools to create graph across every analysis platform in JMP, we’ll illustrate creating visualizations via the powerful Graph Builder platform. This tool has been designed to match the typical way in which one would explore and analyze data in practice. Upon launching the Graph Builder platform (Graph > Graph Builder), the display starts with what can be thought of as a blank canvas and a graph palette (Figure 21A).

Figure 21A JMP: Graph Builder

A graph is created by placing variables into different roles and selecting the desired graphical display. For example, if the variable Moisture Content is dropped into the Y zone, JMP will begin by displaying a dot plot (Figure 21B).

Figure 21B JMP: Graph Builder – Dotplot

If instead of a dot plot, a different graph is desired (e.g., box plot), simply select that option from the graph palette at the top and the graph instantly changes. Figure 21C shows a graph where both a dot plot and box plot were selected.

Figure 21C JMP: Graph Builder – Dotplot and Boxplot

If another continuous variable is dropped in the X zone, JMP will automatically change the graph to a scatterplot. Figure 21D shows an example of such a graph with a regression line included.

Figure 21D JMP: Graph Builder – Scatterplot with regression line

For each type of graph selected, tools and options are displayed on the left to alter or augment the graph even more. For example, the equation, 95% Confidence Interval and RSquared value was added to the graph.

Each one of these graphs created during this process can be saved to the data table if desired. Alternatively, the analyst can continue visually exploring the data. Figure 21E shows how the graph would change if the variable GeoLocation were dropped into the Wrap drop zone.

Figure 21E JMP: Graph Builder – Individual scatterplots with regression lines

If GeoLocation were moved from the Wrap drop zone to the Overlay drop zone, and the option to display the equation and RSquared were turned off, the graph would change to that shown in Figure 21F.

Figure 21F JMP: Graph Builder – Scatterplot with overlaid regression lines

Floating the cursor over one of the axes, the pointer changes to a hand symbol, allowing a simple way to move and/or rescale the axes. Double-clicking on an axis brings up a set of tools to alter the graph even further such as changing from a linear scale to a log scale for example.

Right-clicking on the graph brings up a menu of additional tools to alter the display, such as changing the colors and symbols used.

This process illustrates the fundamental difference in how each software approaches graphical analyses. In Minitab, graph types are chosen and options selected without seeing what those choices will do. In JMP, you build graphs interactively seeing how each choice would change and enhance the graph. This results in a much more efficient and richer exploration of data allowing you to quickly generate myriad graphs to uncover features in the data and to create the visualizations that best communicate your findings.

Two very powerful tools that can greatly enhance visually exploring data is JMP’s Local Data Filter and Column Switcher. Illustrated with the Advertising Experiment data.

With JMP’s dynamically linked data, new variables can be switched and/or the data filtered, and the graph will instantly update to reflect those changes. Figure 22 shows a comparative dot plot for the variable WS1with 95% confidence intervals for four different categories of the variable Ad Style. The Column Switcher, seen in the upper-left, was set up so that the analyst can simply select (or play through) 9 other possible outcome variables to display on the Y axis. The Local Data Filter was also set up using the explanatory factors Promotion (three levels) and Time Period (10 levels). The analyst can subset the data by these variables. For each selection, the graph will instantly update to display the selected variable for the Y axis and to include only the data as chosen with the Local Data Filter. In Minitab, a new graph would have to be created separately for each possible combination of these variables.

Figure 22 JMP: Column Switcher and Local Data Filter

JMP contains many graphical tools not currently available in Minitab. These greatly expand your ability to explore and present data. Figure 23A-J show some of these visualizations.

Figure 23A JMP: Hover Graphlet

Figure 23B JMP: Image Labels

Figure 23C JMP: Geographic Maps

Figure 23D JMP: Motion Bubble Plot

Figure 23E JMP: Correlation Color Map

Figure 23F JMP: Shape Based Heat Maps

Figure 23G JMP: Density Contour Plot

Figure 23H JMP: Word Cloud

Figure 23I JMP: Comparative Histograms

Figure 23J JMP: Scaled Parallel Plots

Figure 23K JMP: Mosaic Plot

Figure 23L JMP: Nonlinear Functional Fit

Univariate Analyses

Minitab and JMP approach running statistical analyses quite differently. We’ll step through the analysis of a single continuous variable for each software to demonstrate these differences.

We’ll illustrate these analyses using the Yogurt Production data.

Minitab

In Minitab, the Graphical Summary tool is a useful way to analyze a single variable. To run this analysis, select this tool in the Stat menu. A dialog box appears to select the variable upon which to perform the analysis. Figure 24A show the steps to run the analysis and 24B displays the resulting output. The normal distribution curve that is shown on the graph is the only distribution that can be displayed in this output. Using other probability distributions would require a different tool in Minitab.

Figure 24A Minitab: Graphical Summary

Figure 24B Minitab: Graphical Summary

If different statistics other than those displayed were desired, a different tool would need to be launched (Stat > Basic Statistics > Display Descriptive Statistics) and those options selected. Steps and output now shown.

The Graphical Summary displays the results of the Anderson-Darling normality test. If you wanted to examine a probability plot and/or an empirical Cumulative Distribution Function graph to accompany the test, it would need to be done using two different additional tools (Stat>Basic Stat>Normality Test and Graph>Empirical CDF). The procedure is similar to running any other analysis. A dialog box appears upon launching the tool for the user to specify the variable to run the analysis on and to select options. Figure 25A shows the steps to run the normality test analysis and 25B shows the resulting output.

Figure 25A Minitab: Normality test

Figure 25B Minitab: Normality test

To create the CDF plot, a different tool (Graph>Empirical CDF) would be launched and the analysis run (steps and output now shown).

If, for example, the normal distribution was not a good model to use for a variable, other distributions may need to be evaluated. This evaluation is performed in a different tool (Stat>Quality Tools>Individual Distribution Identification) (steps and output not shown).

Let’s say the next step in the analysis is to perform a hypothesis test for the population mean. To do so would require accessing yet another tool (Stat > Basic Stat > 1-sample t) (steps and output not shown). To perform a hypothesis test regarding the variance, this same procedure of using a new tool (Stat > Basic Stat > 1 variance)

Other analyses common to analyzing a single continuous variable are: Equivalence Test, Tolerance Interval, Prediction Interval, Capability Analysis, among others. Each would require a different tool in Minitab and each producing a separate output. Current, Minitab does not have the capability to produce a prediction interval so this would actually require manually creating a formula.

JMP

JMP approaches univariate analyses very differently and is more aligned to the natural work flow of exploring and analyzing data. JMP has a single platform (Analyze > Distribution) that can perform all the analyses mentioned in the previous section. The platform is accessed by selecting it from a menu and completing a dialog box, as demonstrated in Figure 26A. Figure 26B displays the initial output.

Figure 26A JMP: Distribution

Figure 26B JMP: Distribution

The initial output provides both a graphical and numerical display of the data, including a histogram, box plot, summary statistics and a confidence interval for the mean. Clicking on the red triangle at the top of the summary statistics table provides access to a set of tools to change the statistics that are displayed.

All the analyses mentioned among many others can be conducted from within this one report window. These are all accessed via the red triangle next to the variable name. These contextual menus provide you with a set of analyses that would be next to consider in your analysis workflow. Figure 27 shows that menu along with the Continuous Fit>Normal analysis selected.

Figure 27 JMP: Distribution

Figure 28A through 28K show results from the Continuous Fit and all the other univariate analyses performed on these data with each new set of output added to the one report window allowing for easy comparison.

Figure 28A JMP: Quantile Plot and Distribution Fit

Figure 28B JMP: Goodness of Fit

Figure 28C JMP: CDF Plot

Figure 28D JMP: Interactive Distribution Profiler

Figure 28E JMP: Interactive Quantile Profiler

Figure 28F JMP: Test Mean

Figure 28G JMP: Test Standard Deviation

Figure 28H JMP: Prediction Interval

Figure 28I JMP: Tolerance Interval

Figure 28J JMP: Equivalence Test

Figure 28K JMP: Capability Analysis

Additional interactive tools are available bringing an extra level of analysis capabilities via “what-if” scenarios. Figure 29A – 29C show three of these (p-value animation, power animation, and capability animation). In addition to helping better understand the process being studied, these interactive tools provide a great way to understand or teach statistical concepts.

Figure 29A JMP: p-value Animation

Figure 29B JMP: Power Animation

Figure 29C JMP: Interactive Capability Plot

The Local Data Filter and Column Switcher is also available within this interactive report. Figure 30 shows the Column Switcher set up for the 7 production lines and the Local Data Filter set up for Batch Number. You can now choose the production line and the batches to include in the analysis and all of the numerical and graphical output is instantly updated. With Minitab, each outcome variable and/or subset of the data one wishes to analyze would require rerunning each and every analysis across all the various tools creating a substantial amount of reports to manage.

Figure 30 JMP: Column Switcher and Local Data Filter

Saving and Sharing Results

Minitab

Each time an analysis is performed in Minitab, the output will be placed in the Navigator pane as a separate element. Saving the Project File will save all those output elements along with all the worksheets into the Project File. Individual output can also be saved as an image file (e.g., .png, .jpg) and placed into other applications.

JMP

There are many ways to save your analysis results in JMP. These include saving as an interactive .html file, to a JMP Journal file, as an image file placed into other applications, and even published to a private secure dashboard via. JMP Live providing the ability to view and interact with reports outside of JMP. We will illustrate saving reports as a script to the data table as that is the most common way you will save your analysis work.

From within a report, choose Save Script > To Data Table under the Red Triangle in the top left of the report window (Figure 31A).

Figure 31A JMP: Saving a script to data table

The saved scripts will appear in the top-left pane of the data file window (Figure 31B).

Figure 31B JMP: Saved Scripts

Clicking the green triangle will rerun the saved script to perform all the analyses associated with that report. This allows you to rerun a full set of analyses on a modified data set (e.g., removing/adding data, recoding values, subsetting via the global data filter, new row states). Scripts can be reorded in the pane or grouped together to aid in management of your analysis work.

One-Factor Analysis

Both Minitab and JMP have tools to perform one-factor analyses (e.g., simple linear
regression, simple logistic regression, one-way ANOVA, and Chi-square test for
comparing proportions). Similar to univariate analyses, they approach how the
analyses are performed very differently.

We’ll illustrate some of these analyses using the Tablet Production data.

Minitab

When performing a one-factor analysis in Minitab, the appropriate tool is launched (Stat>Regression>Fitted Line Plot, for example). The appropriate variables are specified within the dialog box and other options chosen. Output is generated and placed in the output window. Consider upon examining the the output of a simple linear regression for example, it is desired to add a confidence interval to the graph, or to examine residual plots, or perhaps it’s desired to change the scale of one of the variables from linear to log or perhaps use a different response or predictor variable all together. The fitted line plot tool would need to be relaunched, the options changed, and then a new set of output created. Each set of output in Minitab is primarily static, allowing only a few alterations.

Different one-factor analyses (e.g., logistic regression, ANOVA, chi-square test) require the use of other tools.

JMP

In JMP, all one-factor analyses are performed from one platform, Analyze>Fit Y by X. Figure 32A shows the dialog box for this platform. When the Y, Response Variable and X, Factor is chosen, JMP automatically determines the correct analysis.

To illustrate how JMP can perform multiple analyses simultaneously, we have chosen two variables for the Y, Response (Dissolution, a continuous variable; Lot Acceptance, a binary variable) and two variables for the Factor (API Particle Size, a categorical variable with three levels; Mill Time, a continuous variable).

JMP will create four different analyses:

One-way ANOVA for Dissolution vs API Particle Size.
Simple linear regression for Dissolution vs Mill Time.
Contingency table analysis for Lot Acceptance vs. API Particle Size.
Simple logistic regression for Lot Acceptance vs. Mill Time

Figure 32A JMP: Fit Y by X

As in many other platforms in JMP, Fit Y by X has the capability for the analysis to include a categorical By Variable, running a separate analysis for each level of that variable. The models can also be fit assigning weighted values for the observations. The platform can also be used on data that is in summarized Frequency/Table format.

The initial output for these four analyses is shown in Figure 32B. Note: Row states were set up so that lots meeting the specification of dissolution ≥70 are color coded green and those < 70 are color coded red.

Next to each analysis is a red triangle hot spot that reveals a contextual menu with the tools JMP recommends as possible next steps in the overall analysis. As there are four different statistical analysis techniques performed, there are four different sets of tools under each menu. Figure 33 shows each of these contextual menus. You are now ready to choose the next step in the analysis workflow. Each subsequent analysis produces another set of output added to the report with a new red triangle hot spot providing additional options.

Figure 32B JMP: Fit Y by X Report

Figure 33 JMP: Fit Y by X Contextual Menus

To illustrate how an analysis might unfold in the Fit Y by X platform, the Fit Line option under the red triangle for the Bivariate Fit of Dissolution by Mill Time was selected. Figure 34 shows the output that now gets added to the report based on that selection.

Figure 34 JMP: Linear Regression Fit

Notice that a new red triangle hot spot has appeared next to the chosen linear fit. Figure 35 shows the options available under that contextual menu. Here you see the option to add confidence or prediction intervals to the graph, create residual plots, save values to the data table, among others.

Figure 35 JMP: Linear Fit Contextual Menus

As shown in the Bivariate Fit menu in Figure 33, other models can be fit and compared. For example: polynomials, splines, orthogonal, non-parametric densities, and robust regression. Transformation are available under Fit special. Figure 36 shows those options.

Figure 36 JMP: Transformation

Each model fit being considered can be added or removed from the report as you explore and compare different model choices finding the one best for the data. Figure 37 shows the display with a quadratic fit and non-parametric density added.

Figure 37 JMP: Comparing Models

The Local Data Filter and Column Switcher can be accessed if you wish to change the variables used or select a subset of the data to base the analyses on. All the output will instantly update to reflect those choices. It’s not necessary to subset the data into a new data table and rerun the analysis as it is with Minitab.

As this example again illustrates, analyzing data in JMP is much more than simply performing a particular statistical technique on a dataset and observing a static collection of output. The platforms are designed to follow a natural analysis workflow aligned with how you would analyze data in practice. You explore, interact with the data and results, change variables, filter the data, try different model options, with each result leading to the next logical step in a sequence of analysis steps.

Multivariable Statistical Models

Building statistical models that involve many variables is probably the most powerful and frequently needed type of analysis. There is usally four purposes for building these models:

For testing specific hypotheses.
Example: Does Drug Treatment A reduce the recovery rate by at least 15% for males over 70 years of age as compared to a placebo? Is the effect the same for females?
Finding the few variables out of many that are most correlated with an outcome.
Example: What are the key socio-economic conditions most correlated with poor health outcomes for a community?
Quantitatively describing the relationships between variables.
Example: How much does viscosity of a solution decrease when the processing temperature increases by 5 ⁰C and the processing time increases by 10 minutes?
Create a method on which to predict an outcome.
Example: What is the likelihood a customer would be interested in learning about a new mortgage refinance program based upon characteristics of the customer and their current mortgage?

The primary goal for a model may be just one of these, some or even all of them. The modeling technique chosen is often driven by which of these goals are most important.

Minitab

Minitab v19 has three multivariable modeling techniques:

· Generalized Linear Models (e.g., ANOVA, Linear Regression, Logistic Regression)

· Classification and Regression Trees

· Discriminant Analysis

Similar to performing a one-factor analysis, building a multivariable statistical model requires first choosing a modeling type appropriate for the situation (Stat>ANOVA>General Linear Model, for example). Within the dialog box, the user then specifies the variables to be included and any other relevant options.

The output is generated upon running the analysis and placed in the output window. In the process of building a statistical model, an analyst typically needs to fit many different models, with each including a different set of variables and model terms until a final model is determined. If it is decided to remove a variable or term from the model, the dialog box is relaunched, changes made, and the analysis is rerun. Each possible model considered must be run separately each producing a different set of output. As is most analysis results in Minitab, the numerical and graphical output is mostly static with just a few options to make changes. If one wishes to remove some observations from the data, for example, a new worksheet must be created and all of the models run again.

JMP and JMP Pro

JMP and JMP Pro v16 have the following multivariable modeling techniques along with many variants within each.

· Generalized Linear Models (e.g., ANOVA, Linear Regression, Logistic Regression)

· Partition (Decision Tree)

· Neural Network

· Discriminant Analysis

· Bootstrap Forest *

· Boosted Trees *

· K Nearest Neighbors *

· Naïve Bayes *

· Support Vector Machines *

* JMP Pro Only

There are other modeling techniques in JMP and JMP Pro beyond these. They are typically used for more specialized applications vs. for general modeling purposes and thus not listed above (e.g., Structural Equation Models in social sciences; Uplift, Choice, and MaxDiff for consumer research; Functional Data Explorer to model data that are functions, signals, or series).

Each of JMP’s modeling platforms also has many options regarding estimation methods and model fitting specifications. These provide the ability to generate even more models to consider and allow for greater capabilities in handling complex situations (e.g., Weighted Least Squares, Penalized Regression, Informative Missing, K-Fold Cross Validation to name a few).

We’ll illustrate the model building process in JMP Pro by fitting a Logistic Regression Model using the Mortgage Refinance Outreach data.

Logistic Regression Model:

In JMP, all linear models are created using the Fit Model platform under the Analyze menu. This one platform allows you to choose from a variety of model forms and model fitting techniques by selecting the desired option under the Personality drop down menu. Figure 38 shows the options available in JMP (left) and JMP Pro (right).

Figure 38 JMP and JMP Pro: Fit Model – Personalities

Figure 39A shows the dialog box for the Fit Model platform with the outcome variable and model terms added. Here we are fitting a main effects only model. The option to include interaction terms, quadratic effects, defining terms as: (crossed or nested) and (fixed or random) are available. We have also included a validation column. This column has random allocated the data into: 75% training and 25% validation. JMP will use only the training data to fit the model. The validation data will be used to evaluate the predictive ability of the fitted model. This approach can help avoid the undesired result of a model overfitting the data.

Figure 39A JMP Pro: Fit Mode

Upon running the analysis, JMP will produce the nominal logistic regression report.

As in all of JMP’s analysis output, it’s best to think of a report as an interactive platform vs. static output. For example, model terms can be removed from the fitted model without the need to return to the Fit Model dialog box, making the changes, and rerunning the analysis. Figure 39B shows the Effect Summary table from the report with the variable Age selected.

Figure 39B JMP Pro: Fit Model – Effect Summary for Nominal Logistic Regression

Figure 40 shows options that are available under the top red triangle.

Figure 40 JMP Pro: Fit Model – Contextual Menu in Nominal Logistic Regression report

Figure 41A – 41D shows some of the common types of output that is examined when building a logistic regression model added to the report.

Figure 41A JMP Pro: Fit Model – Confusion Matrix in Nominal Logistic Regression report

Figure 41B JMP Pro: Fit Model – ROC Curve in Nominal Logistic Regression report

Figure 41C JMP Pro: Fit Model – Lift Curve in Nominal Logistic Regression report

Figure 41D JMP Pro: Fit Model – Profiler in Nominal Logistic Regression report

When the primary objective of a statistical model is for prediction only and it’s not necessary that the equation has a simple form with interpretable coefficents, other more complex models should be explored. JMP Pro has a very powerful platform called Model Screening that will fit many different models using various techniques presenting the result together in one report. Figure 42 shows the dialog box for that platform with the variables assigned to their roles and a set of model types selected.

Figure 42 JMP Pro: Model Screening

Figure 43 shows some of the summary results produced in the report allowing for you to easily see and compare model performance. Notice that the Boosted Neural Network model has the lowest overall error rate on the validation data (16.0%) with the Nominal Logistic Regression at 19.2%.

Figure 44 displays the Decision Threshold tool which can be important to evaluate when predicting a categorical outcome as we are doing here. This tool provides an interactive means to evaluate how the error rates change for each model as you consider different probability threshold values.

Each of the model platforms can be launched from this report providing access to the full set of tools available for each.

Figure 43 JMP Pro: Model Screening – Model Summary

Figure 44 JMP Pro: Model Screening – Decision Threshold

Conclusion

Both Minitab and JMP offer users an easy to use point-and-click environment to analyze data. Though they both contain many of the same statistical techniques, you access and use them in different ways. For example: most univariate analyses in JMP are performed within the Distribution platform where in Minitab, many different tools are used depending upon the specific data type being analyzed and the statistical technique to be chosen. In JMP, the output from each univariate analysis can be added to the same report window.

As was illustrated in this transition guide, the data tables, graphs, and reports are dynamically linked in JMP providing you with a more interactive environment to explore and analyze data. This capability allows you to conduct an entire set of analyses on various subsets of data using only one data table and one set of analysis reports. In contrast, a typical analysis session in Minitab requires you to manage many different worksheets and a large collection of analysis output.

JMP’s approach to visually exploring data via the Graph Builder is a very different paradigm than how Minitab approaches graphing data. As we demonstrated in the guide, JMP’s Graph Builder allows you to see how each option you are considering changes the visualization providing you with a dynamic way to discover features and patterns in the data.

Visit jmp.com to learn more about JMP’s capabilities and to find learning resources such as books, webinars, white papers, courses, among others.

About the Author

Self-described “analytics evangelist” Craig DePorter of Omya discusses why analytics enablement means learning more with less

Meg_Eberle — Tue, 13 Jul 2021 20:23:50 GMT

Craig DePorter is Head of Advanced Mineral Development at Omya, a leading global producer of industrial calcium carbonate and dolomite derivatives. A polymer chemist with more than 20 years of industry experience, Craig has throughout his career promoted data-driven approaches in the pursuit of both business and scientific objectives.

A Lean Six Sigma Black Belt, he has long applied industrial statistics to challenges spanning the entire product pipeline from new product development to commercialization to quality assurance in Europe, the Middle East and North America. Craig is also a prolific inventor – he holds 19 US patents – and is deeply engaged with talent recruitment and forging industry-academia partnerships.

I asked .@CraigD about his evangelism for analytics, not just at Omya but throughout his career.

Craig DePorter, Head of Advanced Mineral Development at Omya, says analytics increases an organization's efficiency.Meg: How has your advocacy for analytics transformation impacted your career trajectory? Have you encountered career opportunities that might not have been available to you before?

Craig: I am an analytics evangelist, specifically because the technique (which is not software-specific) has been proven countless times to allow more efficient research. People are able to learn more by doing less, even though they get scared of design sizes when they start. For example, I’ve seen very, very few people make a real effort to use design of experiments (DOE) and decide to go back to “the old way.”

Have I been offered any new job opportunities because of it? Not explicitly, but perhaps indirectly since I’ve increased the efficiency of the organizations I worked for.

Meg: And surely that efficiency can only be scaled up as you bring colleagues on board with data-driven approaches. That being said, introducing statistical methods to a previously statistics naïve organization can be disruptive. What, in your experience, has been the best way to grow support for data initiatives? Were there setbacks?

Craig: It’s all about learning more with less effort – and being able to back up those discoveries with sound statistics instead of techno-jargon and scientific supposition. My efforts at my current company, Omya, have been very successful in engaging a growing community of users who see the benefits of analytics. Though I am on the heels of at least two (if not three) prior attempts to do so that were not really as successful.

Meg: One of the most common objections we hear from newcomers to JMP is that some fear statistical approaches will supplant domain expertise. From your perspective as a polymer chemist, how has having JMP changed the way you apply domain knowledge?

Craig: This is something that I make sure I cement into my user group: that domain expertise is even more important when using DOE, since the planning aspects are the most critical steps. What combinations won’t work? Can you expand the ranges? Does this encompass what you “think” will be the best setting? And so on.

Meg: Tell me about your relationship with JMP as an organization. What are the benefits to connecting with users outside of Omya, for example at a Discovery Summit?

Craig: The JMP organization is top notch, even back to the 1990s when I started using the software and did training at SAS headquarters in Cary. And expanding JMP to other companies that I have worked for has always been with great support from the commercial and technical teams.

Discovery Summits are superb at growing the community of statistically-minded people. No hard-sell of JMP. It certainly does not hurt that JMP is the best overall tool I have found for the sorts of things that I do, but the feel of the Summit is about espousing the concept of using statistics in your day-to-day work. Creation of this mindset prevails ... and it just so happens that everyone uses JMP. It’s just a tool.

Meg: What advice would you give to someone who is just starting out with JMP?

Craig: Try it, and have a mentor help you get going. Start small.

Add() vs. + vs. Sum(): What’s the difference?

Wendy_Murphrey — Wed, 07 Jul 2021 13:00:00 GMT

Ever been confused by the result when trying to add numeric values using a plus (+) operator? Maybe you discovered the Sum() function to get around the problem but never really understood why it worked. In this blog post, I explain the Add() function, plus (+) operator, and the Sum() function to help you get the results you need from your script or column formula.

Let’s start with the Add() function. Did you know that plus (+) is the symbolic representation of the Add() function? It is! That means that the following two expressions are exactly the same:

1 + 2;  //Returns 3
Add( 1, 2 );  //Returns 3

The Add() function and the plus operator add values to calculate a total value. The arguments can be numbers, matrices or lists of numbers.

But what if one of the values is a numeric missing (.)? The Add() function will return missing if any of the arguments is a missing value.

In the previous example, the result was 3 for either expression. Building on that, if a missing value was one of the arguments, then it would return missing, rather than 3.

1 + . + 2;  //Returns missing
Add( 1, ., 2 );  //Returns missing

So how do you still obtain a total value excluding the missing values? This is where the Sum() function comes in. The Sum() function will also add values to calculate a total value, however, it ignores missing values.

Below is the syntax for the Sum() function. Notice the arguments are separated by commas and the total value of all non-missing arguments is calculated.

Sum( 1, ., 2 );  //Returns 3

Can Sum() ever return missing? Well, yes, it can. When the argument(s) supplied to the Sum() function are all missing values, the result will also be missing. In this example, the value of both x and y is missing.

x = y = .;
Sum( x, y );  //Returns missing

And don’t get tripped up thinking that you can use the plus operator to specify the arguments.

Sum( 1 + . + 2 ); );  //Returns missing

When you do that, JMP sees this as one argument that uses the Add() function. And remember that the Add() function returns missing when any of the arguments is a missing value:

Sum( Add( 1, ., 2 ) );  //Returns missing

Which is essentially the same as this:

Sum( . );  //Returns missing

If you changed that example to include two arguments, such as:

Sum( 100, 1 + . + 2 );  //Returns 100

The result is 100 because the arguments are 100 and missing, as in:

Sum( 100, . );  //Returns 100

I hope this helps to clarify the uses of Add(), plus (+), and Sum().

John Szarka on JMP Pro, reproducibility and why being active in the JMP Community and Early Adopter program helps users at W. L. Gore get more from their investment in the software

Meg_Eberle — Tue, 06 Jul 2021 14:48:49 GMT

W. L. Gore's John Szarka says the Early Adopter program lets users have a voice in the development process.Statistician John Szarka supports regional and global new product development and process improvement for W. L. Gore & Associates’ Core Technology group. His work integrates many areas of industrial statistics, including design and analysis of experiments, statistical process control, measurement systems analysis and sampling plan developments. He is a champion and core team member for Gore’s analytics initiatives establishing best practices and training curriculums across the enterprise.

I spoke with @jszarka about how W. L. Gore has cultivated a dynamic analytics culture, standardizing around JMP Pro and engaging with all the available resources to keep learning and improving.

Meg: Many people think of W. L. Gore as something of a pioneer in analytics transformation, given the company’s long history with data-driven approaches. But keeping that analytics culture in a state of evolution is no doubt an ongoing process. What kinds of ROI or improvements have helped you and your colleagues to sustain momentum – and keep innovating – when it comes to analytics at Gore?

John: I feel very fortunate that statistics and analytics are rooted in the founding of our enterprise. Our co-founder, Bill Gore, wrote the book Statistical Methods for Chemical Experimentation back in 1952 while still at DuPont, before the company was started in 1958. Our investment in having a global statistics team has been highly valued by our technical leaders across the organization.

With such an explosion of "big data" and modeling advances, one thing I feel we’ve recently taken very seriously at Gore is valuing our data as an asset. Having the right people and systems in place to enable us to extract the most value from our data is essential for us now and moving forward.

We want to ensure that we are using the right data to answer the right questions. That remains at the front end of our conversations when we’re exploring projects using big data for new product development or process improvement studies. In many instances, we can use JMP Pro to meet our needs on the data and analysis side to provide insights and help in making decisions. Our investment in JMP Pro – up from the base version of JMP – has made this type of modeling much easier for our associates.

Meg: One of the most common objections we hear from newcomers to JMP is that some fear statistical approaches will supplant domain expertise. How has having JMP Pro changed the way domain experts at Gore apply their knowledge and skills?

John: It’s never an either-or argument when it comes to using data from the process versus domain expertise. We make the best combined inferences when we bring this information together. There is a strong culture of experimentation at Gore. We may have information from the past around how we think a certain experiment may work out. In some instances, the actual data agree with those prior beliefs, and in other instances they don’t.

One of the key points is to remind the group around the reproducibility of the experiment. We are likely to build on the work of one experiment with a follow-up experiment. And if we continue to see a particular signal, we can be more certain that it’s true. If it falls off, then perhaps not. Over time we build robust knowledge from the science and the data together.

Meg: Tell us about your relationship with the JMP organization. What value do you get from participating in Discovery Summits, users’ groups, training, STIPS, the online JMP Community, etc.?

John: The online JMP Community is a vibrant space that I try to visit at least once a week. It has been a great place to learn new tips, read discussions on various issues and get a sense for how others are leveraging JMP for their organizations. I’ve probably contributed the most myself in the "wish list" category – mostly based on observations that could improve the JMP experience for our users at Gore even more.

This past year, we as a statistics group at Gore looked at modifying our in-person trainings for the virtual environment. Utilizing STIPS has been great for us because it gives rich context in the type of problems our technical associates are trying to solve. The content is framed with the user in mind – how they can answer their questions using statistical techniques. Many software vendors would have a series of "click here" steps that you follow, but I was really impressed with the production quality of many of the STIPS modules.

The online Discovery Summit conferences in the Americas 2020 and Europe 2021 were the first I attended. I gravitated to the presentations from industry, as the problems they were working on were interesting. One of my favorite presentations was Statistical Process Control for Process Variables that have a Functional Form. Profile monitoring in the SPC literature has existed for some time, but I really liked how easily we can use platforms like Functional Data Explorer and the Model Driven Multivariate Control Chart to easily implement them in practice.

Meg: Tell us about your relationship to the JMP software developers. How do you go about providing feedback, and do you feel that feedback helps influence new JMP releases? Does your ability to maintain a collaborative relationship with JMP Development have an impact ultimately on the value you get from JMP?

John: The JMP software developers are a fantastic group of people to interact with! I’ve made connections with JMP associates over the years through external conferences along with JMP Discovery Summit. That led me to use the Early Adopter program to get a sense of the direction of future releases and have a voice into that process.

The developers are great listeners and really seek to understand our problems and the solutions that will help us the most in future JMP releases. This has also led to joint meetings between members of Gore and JMP to discuss targeted areas. I’ve seen multiple instances where specific requests I’ve made have been implemented, which is proof that this process works.

Meg: What advice would you give to someone who is just starting out with JMP?

John: Partner with those in your organization already using JMP or reach out to your peers in quality organizations you’re a part of to see the areas they get the most value out of JMP. I would personally start in the graphing menus, especially Graph Builder. There is tremendous power in storytelling with graphs and I find that I can tell a lot of stories by utilizing the many features within Graph Builder. As many users start learning more about the various models you can build in JMP, they sometimes get away from being as good at graphing, but I would get into the habit of centering your focus on graphing from the onset.

New features in Process Screening for JMP 16

tonya_mauldin — Mon, 28 Jun 2021 22:14:04 GMT

Process Screening enables you to explore a large number of processes across time. Several new features were added to this platform in JMP 16. This blog post details the new features for Process Screening that were introduced in JMP 16.

XBar and IMR charts in a single launch

Companies often have many ongoing processes. Some processes have more than one observation per subgroup and require an XBar chart, while other processes may have a single observation per subgroup and require an IMR chart. Beginning with JMP 16, you can get both chart types with a single Process Screening launch.

Consider the following data table.

dt=New Table( "Process Data",
	Add Rows( 24 ),
	New Column(
		"wafer", 
		Numeric( 1 ),
		"Nominal",
		Set Values(
			[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
			11, 12]
		),
		Set Display Width( 0 )
	),
	New Column(
		"Y1", 
		Numeric,
		"Continuous",
		Format( "Best", 10 ),
		Set Property(
			"Spec Limits",
			{LSL( 104.41 ), USL( 131.89 ), Target( 118.15 )}
		),
		Set Values(
			[114.555760105678, 113.52359591958, 114.946183872216, 116.25374611217,
			110.287898508744, 119.847995090729, 113.877878795723, 113.106755626713,
			114.832071507276, 115.737338527253, 120.193341974807, 113.425287275443,
			., ., ., ., ., ., ., ., ., ., ., .]
		),
		Set Display Width( 0 )
	),
	New Column(
		"Y2",
		Numeric,
		"Continuous",
		Format( "Best", 10 ),
		Set Property(
			"Spec Limits",
			{LSL( 164.39 ), USL( 429.65 ), Target( 297.02 )}
		),
		Set Values(
			[322.616751775796, 323.833298568833, 301.607366273833, 278.646245766939,
			469.298298488357, 332.715779503856, 346.462173291699, 213.59945858556,
			291.464226219377, 345.449129206092, 242.233144838257, 423.564714196848,
			291.370833295888, 333.921504947116, 288.644927205998, 258.806219486679,
			263.935377860945, 276.312880747942, 322.144968498633, 341.978235063698,
			316.143895248667, 302.164492808713, 319.751938178979, 269.802897971304]
		),
		Set Display Width( 0 )
	),
	Group Columns( "Processes", :Y1, 2 )
);

This table contains data from two processes. The first process (Y1) has one observation per wafer. The second process (Y2) has two observations per wafer. Therefore, the control chart type should be XBar for Y2 and IMR (Individual and Moving Range) for Y1.

To create this analysis, select Analyze->Screening->Process Screening. Assign Y1 and Y2 to Process Variables. Assign wafer to Subgroup. Change the chart type to XBar and R.

Alternatively, you can run the following JSL:

obj=dt<< Process Screening(
	Process Variables( :Y1, :Y2 ),
	Control Chart Type( "XBar and R" ),
	Subgroup( :wafer )
);

After clicking OK, you will find the following analysis:

An XBar and R chart type was applied to Y2. An Individual Measurement and Moving Range chart type was applied to Y1.

Color Out of Spec Values

Process Capability and Manage Spec Limits already provided methods to color values that are out of the specification limits. This functionality was added to Process Screening JMP 16.

Run the following code to open Semiconductor Capability.jmp and run the Process Screening script attached to the table.

dt=Open("$SAMPLE_DATA/Semiconductor Capability.jmp");
obj=dt<<Run Script("Process Screening");

To color the out of spec values in your data table, select the red triangle next to Process Screening and choose Show Capability->Color Out of Spec Values.

In the data table, cells with observations that fall below the lower spec limit are colored red. Cells with observations above the upper specification limit are colored blue.

Ability to specify K for K Sigma

Process monitoring is often plagued by too many false alarms. This can lead to the alarms being ignored. The K Sigma option, introduced in JMP 16, gives users the ability to specify K. Recall the limits are calculated using +/- K * Sigma, and increasing K will widen the limits. This gives you a way of making alarms less sensitive, producing fewer false alarms.

As an example, run the following:

dt=Open("$SAMPLE_DATA/Semiconductor Capability.jmp");
obj=dt<<Process Screening(
	Process Variables( :A2P1 ),
	Control Chart Type( "Indiv and MR" ),
	Subgroup( :Wafer ID in lot ID )
);

Note that there are 9 Test 1 violations. After investigations, it was determined that these were all false alarms. To make the alarms less sensitive, K Sigma can be adjusted. Select Analyze->Screening->Process Screening. Select A2P1 as Process Variables. Select Wafer ID in lot ID as Subgrouping. In the bottom left of the dialog, change KSigma to 4.

Click OK.

A KSigma of 4 causes the control limits to be wider (center line +/- 4*sigma) as compared to a KSigma of 3 (center line +/- 3*sigma). This example now has no test 1 violations as all observations fall within the 4 sigma limits.

Three key new features were added to Process Screening in JMP 16.

The ability to have XBar and IMR control charts in a single launch.
The ability to color and select out of spec values.
The ability to alter KSigma to make alarm tests more or less sensitive.

Procter & Gamble's Zhiwu Liang explains how nurturing a dynamic relationship with JMP Development helps the company get more from its investment in a software that is constantly being improved

Meg_Eberle — Wed, 23 Jun 2021 19:50:08 GMT

Zhiwu Liang of P&G says JMP makes it so easy to show results to managers.Zhiwu Liang is Principal Statistician at Procter & Gamble, based in Brussels. With nearly 20 years of experience working in the consumer goods industry, Liang is skilled in design of experiments, non-linear model and prediction, statistical modeling, decision trees, time series analysis and prediction. He holds a PhD in Mathematical Demography from the University of Groningen in the Netherlands.

In a recent conversation, and I discussed the course of his career, communicating data with non-statisticians and the many reasons he sees to nurture a collaborative relationship with JMP software developers.

Meg: In your role at P&G, you do a lot to promote analytics enablement through teaching and consulting as well as doing your own work with more complex modeling and experimental design. How has your advocacy for analytics transformation within P&G affected your career trajectory?

Zhiwu: Advocating for analytical methods is my job! But it has also definitely helped me to gain recognition within P&G. And a huge part of that is having a suitable tool with which I can easily communicate with non-statisticians. By teaching JMP skills at P&G, I’ve come to be known not only as a statistician but also as a JMP expert. Whenever my colleagues have questions about JMP – no matter whether it is stat-related or not – they come to me. And I can often find some issues related to their design and data analysis, figure out the problem and help them to use the correct tool for design of experiments (DOE) or modeling (data analysis). Often before they even realize they’ve used the incorrect method.

Meg: Tell me more about using JMP on a day-to-day basis. Are there ways in which the software makes your standard processes – or even your job as a whole – easier?

Zhiwu: For me, JMP is not only a tool for data analysis; it is also a communication method with which to transfer difficult statistical thinking to a simple graph or profiler and enable others – especially non-statisticians – to understand the data and models. The tool I use most often in JMP is DOE, and it’s much better than other software. At P&G, we have many statistical courses which require participants to use JMP. You can see people’s excitement and satisfaction when they get the correct results so easily for even the most complex models. In fact, my colleagues are also now using JMP on a daily basis. They like the visibility function in JMP because it makes it so easy to show the results to their manager.

Meg: One thing that is relatively unique about the JMP organization is that we see relationship-building as a way for our customers to optimize the benefits they get from their investment in JMP. Can you talk a bit about how you engage with the JMP organization and the broader network of users?

Zhiwu: There are a lot of forums in community.jmp.com that we utilize for sharing information and asking questions about JMP. But I personally prefer Discovery Summits because at Discovery, you can always learn new things – whether it’s a new method, new tool, new application. Also you can easily connect to other users and the JMP software developers. By talking to the architects of the software, you can develop an even better understanding of the methods you have access to in JMP. And you can suggest they add new features for a specific model in upcoming version releases.

Meg: That’s interesting. Why is it so valuable to be able to provide this kind of feedback to JMP software developers?

Zhiwu: It is very important for P&G to cultivate this kind of collaborative relationship. We have thousands of people using JMP on a daily basis. Different functions and categories have different needs. Sometimes we even have to combine Excel tools with JMP to solve some particularly tough questions. So we need to provide timely feedback to JMP about any existing features that do not work in specific situations. We then ask JMP developers to improve the software or provide suggestions on how P&G might build a JMP add-in tool (which, I have to say, might only work for the current version). But that’s the way we can let JMP make people’s lives better at P&G.

Meg: What is the future of JMP at P&G?

Zhiwu: With data science and machine learning booming at P&G, we expect JMP can embed more machine learning techniques, such as convolutional neural networks, LightGBM, Catboost and NPL tools – just like you did for XGboost – to make our lives easier and solve most of our problems within just one tool.

New features in CUSUM control charts for JMP 16

tonya_mauldin — Tue, 08 Jun 2021 21:01:16 GMT

Cumulative Sum (CUSUM) control charts enable you to detect small shifts in a process. They are useful in detecting shifts that occur over time, such as a gradual drift, and that are not necessarily accompanied by a sudden shift. This post details the new features for CUSUM Control Charts that are available in JMP 16.

Test Beyond Limits and Alarm Scripts

The existing CUSUM Control Chart made it easy to visualize when points fell beyond limits. However, there was not a straightforward way to take additional action with this information. JMP 16 introduces two new options: Test Beyond Limits and Alarm Scripts.

Run the following for an example:

dt=Open("$SAMPLE_DATA/Quality Control/Engine Temperature Sensor.jmp");
obj=dt<<Run Script("CUSUM Control Chart");

Notice there are points that fall beyond the control limits. From the red triangle next to CUSUM Control Chart – Y, select “Alarm Script”. Alarm scripts enable you to write and run a script that indicates when the data fail special causes tests. Results can be written to the log or spoken aloud, and there is an option to include an explanation of why the test failed. You can also send results to an email or file.

The default option is “Write with explanation”. This will write the test failures and an explanation of the test failures to the log. Click OK to choose this option.

From the red triangle next to CUSUM Control Chart – Y, select “Test Beyond Limits”. This causes the points that fall beyond the control limits to be flagged.

Additionally, the following information will be written to the log since we turned on the Alarm Script option.

“Out of Control for test 15 in column Y in sample 39.

Out of Control for test 15 in column Y in sample 40.

Out of Control for test 15 in column Y in sample 43.

Out of Control for test 15 in column Y in sample 44.

Out of Control for test 15 in column Y in sample 45.

Out of Control for test 15 in column Y in sample 46.

Out of Control for test 15 in column Y in sample 47.

Out of Control for test 15 in column Y in sample 48.

Out of Control for test 15 in column Y in sample 49.

Out of Control for test 15 in column Y in sample 50.”

Show Excluded Region

Show Excluded Region specifies that subgroups that are entirely excluded are shown on the horizontal axis in the CUSUM control chart. This option is applicable only when an X variable is specified.

To see an example, run the following JSL to open the Oil1 Cusum sample data table and exclude rows 33-40.

dt=Open("$SAMPLE_DATA/Quality Control/Oil1 Cusum.jmp");
dt<<Select Rows({33,34,35,36,37,38,39,40})<<exclude;

Select Analyze->Quality and Process->Control Chart->CUSUM Control Chart. Notice the new option in the bottom left for “Show Excluded Region”. This option is turned off by default. In previous versions of JMP, excluded points would always show on CUSUM Control Charts (unless they were also hidden).

Specify weight as Y. Specify hour as X. Click OK.

Notice that subgroups 9 and 10 (Excluded) are not shown on the graph.

Tune Chart

The Tune Chart option shows a control that enables you to set an acceptable range for the Y variable (practical significance). This is particularly helpful when the testing interval is more frequent, which can result in a much shorter practical average run length.

For an example, run the following:

dt=Open("$SAMPLE_DATA/Quality Control/Oil2 Cusum.jmp");
obj=dt<<CUSUM Control Chart( Y( :weight ), X( :hour ) );

From the red triangle next to CUSUM Control Chart – weight, select Tune Chart. You can use the slider or the number edit boxes to set known acceptable values. Set the minimum to 8.05 and the maximum to 8.15.

Click Done. The value for k changes from .5 to 2.1756. Because we are not using the data units option, this value is calculated as (Maximum-Minimum)/2/sigma. If we were using the data units option, this value would be calculated as (Maximum-Minimum)/2.

Version 16 introduced several new features for CUSUM Control Charts. The new features are

Test Beyond Limits and Alarm Script
Show Excluded Region
Tune Chart

JMP 16 new features in Control Chart Builder – part 2

tonya_mauldin — Fri, 14 May 2021 17:13:14 GMT

Control Chart Builder is an interactive workspace which allows you monitor process variation. This blog post details the new features introduced for Control Chart Builder in JMP 16 that specifically deal with tests and alarms. To see other new Control Chart Builder features that do not deal with tests or alarms, please see my previous blog post .

One-sided charts with one-sided alarms

One-sided charts are frequently used with attribute control charts. If the number of defects is high, you want an alarm to display. Conversely if the number of defects is low, you do not want an alarm to display. Low defects are good. The “Show Limits” option has been replaced with “Show Upper Limit” and “Show Lower Limit.” If one limit is turned off (either upper or lower), the tests associated with the hidden limit which apply solely to the limits (Test Beyond Limits and Test 1) will not alarm.

Run the following for an example:

dt=New Table( "C Chart Data",
	Add Rows( 20 ),
	New Column( "Subgroup",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Set Values(
			[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
		)
	),
	New Column( "Subgroup Size",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Set Values(
			[59, 57, 43, 54, 58, 52, 51, 62, 48, 46, 55, 52, 58, 55, 55, 53, 51, 61,
			55, 57]
		)
	),
	New Column( "Defects",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Set Values(
			[110, 101, 94, 101, 53, 93, 132, 104, 121, 118, 112, 91, 98, 92, 126,
			122, 112, 97, 115, 90]
		)
	)
);
obj=dt<<Control Chart Builder(
	Class( Shewhart Attribute ),
	Variables( Y( :Defects ), n Trials( :Subgroup Size ) ),
	Chart(
		Points( Statistic( "Count" ) ),
		Limits( Sigma( "Poisson" ) ),
		Warnings(
			Test 1( 1 ),
			Test 2( 1 ),
			Test 3( 1 ),
			Test 4( 1 ),
			Test 5( 1 ),
			Test 6( 1 ),
			Test 7( 1 ),
			Test 8( 1 )
		)
	)
);

This code creates a data table and the associated C Chart with the Western Electric Tests turned on.

Notice that there is a point that falls below the lower control limit that is flagged. The company is not concerned with a low defect count. They do not want to be notified when the defect count falls below the lower control limit. To solve, this issue, uncheck the box in the control panel (lower left) for “Show Lower Limit”.

The lower control limit is removed from the graph. The point that falls below the lower limit is no longer flagged.

Zones, Shade Zones and Alarms for Individual on Group Standard Deviation Charts

In previous versions of JMP, Control Chart Builder did not provide the Zones, Shade Zones, or Alarm test options for Individual on Group Standard Deviation charts. These have been added to JMP 16.

To see an example, run the following:

dt=Open("$SAMPLE_DATA/Quality Control/Coating.jmp");
obj=dt<<Control Chart Builder(
	Show Control Panel( 0 ),
	Variables( Subgroup( :Sample ), Y( :Weight ) ),
	Chart(
		Position( 1 ),
		Points( Statistic( "Standard Deviation" ) ),
		Limits( Sigma( "Moving Range" ) )
	),
	Chart(
		Position( 2 ),
		Points( Statistic( "Moving Range on Std Dev" ) ),
		Limits( Sigma( "Moving Range" ) )
	)
);

The code above, opens the Coating sample data table and creates an Individual on Group Std Dev & Moving Range on Group Std Devs chart. Right-clicking inside this graph will yield several new options. The Zones and Shade Zones options have been added to the Limits menu. Customize Tests, Tests, and Westgard Rules have been added to the Warnings menu.

Choose the Zones, Shade Zones, and Tests->All Tests options.

Note that no points were flagged because there were no Western Electric test violations for this data.

Test Excluded Subgroups

Sometimes you may not be interested in flagging excluded observations as test violations. A new option has been added called Test Excluded Subgroups. By default, this option is turned on so that it matches previous versions of JMP.

Run the following JSL to see an example:

dt=Open("$SAMPLE_DATA/Quality Control/Washers.jmp");
dt<<Select Rows(9)<<exclude;
obj=dt<<Control Chart Builder(
	Show Control Panel( 0 ),
	Class( Shewhart Attribute ),
	Variables( Subgroup( :Lot ), Y( :"# defective"n ), n Trials( :Lot Size ) ),
	Chart(
		Points( Statistic( "Proportion" ) ),
		Limits( Sigma( "Binomial" ) ),
		Warnings(
			Test 1( 1 ),
			Test 2( 1 ),
			Test 3( 1 ),
			Test 4( 1 ),
			Test 5( 1 ),
			Test 6( 1 ),
			Test 7( 1 ),
			Test 8( 1 )
		)
	)
);

This code opens the Washers sample data table and excludes the 9^th row of the data table. A P Chart is then created with Western Electric Tests turned on. Notice that there are two Test 1 violations. The 2^nd Test 1 violation is for the excluded point.

From the red triangle next to Control Chart Builder, deselect the option Test Excluded Subgroups.

Notice that only one point is flagged as a Test 1 violation. The second point is no longer flagged because this observation has been excluded and we turned off Test Excluded Subgroups. This option has also been added to Control Chart Builder preferences.

Alarm Script Dialog Improvements

The Alarm Script dialog has been overhauled for JMP 16. You can find this option by clicking on the red triangle next to Control Chart Builder (after creating a report) and select Alarm Script.

Text has been added to the top of the dialog that describes how to use these features. The default selection has been altered from Write to Log to Write with explanation. Three new selections have been added: “Write to File”, “Email”, and “None.” The example script for Write to File will write the test failures to a file in your TEMP folder called test log.txt. The example script for Email will send an email for each test failure encountered. The None option, essentially turns the alarm scripts off.

Additionally, a new reserved variable was created ^QCPHASE. This variable can be used to obtained phase information for your test failures. This new reserved variable can be seen in the Write to Log and Email sample scripts.

New features for tests and alarms that were introduced in version 16 for Control Chart Builder are as follows:

One-sided charts with one-sided alarms
Zones, Shade Zones and Alarms for Individual on Group Standard Deviation Charts
Test Excluded Subgroups
Alarm Script dialog improvements

Other new features for Control Chart Builder in JMP 16 that were detailed in my previous blog post are as follows:

Show Limit Labels
KSigma
Remove Location Chart and Remove Dispersion Chart right-click options
Graph Spacing
Subgroup role for Levey Jennings charts

BIOTRONIK's Gerd Kopp talks advocating for analytics, overcoming internal hurdles and building a relationship with the JMP organization

Meg_Eberle — Wed, 12 May 2021 19:55:22 GMT

"Advocating for analytics was an expectation of my role from the very start," says Gerd Kopp of Berlin-based BIOTRONIK.Gerd Kopp is Quality Management and Reliability Statistician at multinational biomedical technology innovator BIOTRONIK, based in Berlin. In addition to providing technical consulting in reliability analysis, design of experiments, measurement systems analysis, predictive modeling, process control, sample size determination and statistical SOPs for quality management, he also heads the company’s JMP training program.

I recently sat down with to discuss BIOTRONIK's analytics transformation and his work to promote widespread analytics capability.

Meg: We often hear that having access to JMP saves time and reduces costs – both important benefits obviously, but benefits that are framed in terms of the company and its business goals. On a personal level, how does having access to JMP affect your job satisfaction on a day-to-day basis? Have you noticed any changes in morale among your colleagues after they’ve switched to JMP?

Gerd: When you work with any kind of software on a daily basis, it’s important that it’s not just a means to an end. You want the way to produce results to be effortless or even fun. With JMP that’s absolutely the case for me, and for my colleagues. In the ‘Using JMP’ courses [I run at BIOTRONIK], I often notice that new users are surprised how easily graphs can be created and rearranged in a drag-and-drop manner. This not only saves time, but it also helps with the perception of statistics as being an asset rather than an obstacle.

Meg: Speaking of obstacles, what were some of the key hurdles you’ve had to overcome to convince stakeholders to expand access to JMP? How did you demonstrate to your leadership that the value of JMP could be scaled up with an expanded license structure?

Gerd: For stakeholders who don’t anticipate using JMP as often as I do, cost can be an issue. In such cases, it’s important to demonstrate that necessary tasks can be handled more efficiently and more nimbly in JMP. When we introduced JMP almost 10 years ago, I held a course on DOE and statistical modeling with two software packages side by side – JMP and another tool we were using at the time. JMP was not only faster to use for every task, but when I showed how nuisance factors such as component measurements could easily be included in a DOE as covariates, this boosted our already existing efforts to make processes even more robust. So JMP was not only more efficient, but it added new possibilities for analysis.

Meg: What do you see as the key benefit of getting involved in the JMP community outside of your organization, whether through regional users groups or Discovery Summits?

Gerd: There are several aspects to this. The user community at community.jmp.com is the gateway to JMP experts all over the world. When I’m looking for help on JMP JSL programming, for example, I usually find my answers there. The regional [Germany-Austria-Switzerland] users group puts me in touch with people in my position at other companies. Seeing how certain challenges are handled elsewhere can help a great deal when I’m looking to find a way to master my own similar challenges. Discovery Summits extend this to the international stage, but most importantly, these conferences present an opportunity to meet JMP developers. They are an excellent source to explain what happens in JMP platforms behind the scenes, and they’re always interested in users’ ideas for new functionality in JMP.

Meg: An excellent point. Our developers are very focused on introducing new JMP features that users actually want, and we rely on your feedback to understand how to improve future versions. You could say that listening is a big part of our development process. What do you see as the main advantages to cultivating this kind of collaborative relationship with JMP?

Gerd: Having a collaborative relationship with JMP means having better access to JMP related information and JMP staff. With technical issues, it’s always an option to contact the very helpful JMP support team, but when I’m looking for specific advice on a JMP platform, it’s also great to know the respective developers. I’ve asked [our regional] system engineers for hints on occasion. They’re always happy to help if they can – JMP staff in general are very nice people.

Meg: How has your advocacy for analytics transformation within BIOTRONIK impacted your career trajectory? Do you feel that becoming an advocate for analytics – or change agent – has helped you to gain recognition?

Gerd: Advocating for analytics was an expectation of my role from the very start. After holding courses on Statistics 101 and design of experiments, colleagues started contacting me about all kinds of statistical applications in engineering. This led me to develop trainings on special topic areas like measurement systems analysis, process capability and reliability. With a growing number of JMP users in the organization, serving as a JMP administrator then became a more substantial part of my job. Last year we implemented our first BIOTRONIK JMP add-ins to standardize and accelerate recurring analyses, which broadened my role to include statistical programming. So over the years I think it’s fair to say that my job description has grown to be multifaceted, and all of it is tied closely to BIOTRONIK’s emphasis on analytics and our usage of JMP.

Data import, reporting and scheduled publication in JMP Public or JMP Live

eurvan1 — Fri, 07 May 2021 14:20:27 GMT

I like when my JMP reports are updated daily and published automatically without having to press any keys. So, to demonstrate what I mean, I will use a case study that should be of interest to most people. Starting in 2020, I wanted to look at the evolution of the pandemic to predict whether or not future lockdowns will happen in specific countries.

First, I had to look for a good and reliable data source. After researching the different publicly available databases, I chose the Johns Hopkins Coronavirus Resource Center. My next step was to import three different data sources into JMP to analyze daily cases, daily deaths and daily recoveries. I selected Internet Open in the file menu, which made it easy and straightforward to import the data. Below is an example of confirmed COVID-19 cases.

To get this into a script, I had to convert this manual import step for the three data sets, as shown below. Here an example of the data set for the confirmed COVID-19 cases:

Open(
	"https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv",
	HTML Table( 1, Column Names( 1 ), Data Starts( 2 ) ));

The main syntax that I used is the “Open()” function, which opens tables and scripts, as well as data from the internet! This statement is much simpler than the one automatically written to the table.

Next, the data is imported into JMP:

Now we are ready to take the next steps to get the data ready for analysis.

Stack the dates into one column with generic referenced columns names (since the data get updated daily). The Stack option is a fantastic platform in the Tables menu that enables this task to be performed manually.
Rename the Columns in Cumulative Cases, Cumulative Deaths and Cumulative Recovery.
Join the three data sets: daily cases, daily deaths and daily recovery.
Change the Date Format into Numeric and Continuous and set Date Format.
Create three new columns using formulas for daily new cases, daily new deaths and daily new recoveries without negative values by using the Dif formula function for each column.

The negative values appeared when the location was switched in the data set (see below), which, of course, does not make any sense. Therefore, I had to create three new columns to replace the negative values with a 0 (by using the Max formula function for each column).

The final data set is now ready for analysis.

To get this automated at this stage, all the different steps for the data preparation have been scripted. And thanks to Action Recording in the log, which is new in JMP 16, my life is now so much easier. I can now easily extract the different scripted data manipulation steps from the Action Recording itself and insert them in the final script.

Now that I have the script for all the data import and preparation steps, I can use Graph Builder to visualize and analyze the data in a useful manner, as in the example below:

The top half of the graph contains the worldwide cumulative cases, deaths and recoveries. The bottom half has the worldwide daily new cases, deaths and recoveries. I added a Local Data Filter, which allows me to filter on a specific country. I added the graph script to the final script.

As last step, I want to publish this to JMP Live or JMP Public. Below is the published web report on JMP Public, the public version of JMP Live:

Since publishing in JMP 16 has changed quite a bit, here’s how you can use a script to publish from JMP 16 into JMP Public. First a JMP Live connection using desktop JMP needs to be created. In JMP 16, go to File -> Publish -> Manage Connections. Add a new connection. Next, specify the URL and API Key. To get the API Key, generate it from the JMP Public or JMP Live account settings.

In the JSL, use that connection to publish to JMP Live. Here an example of the script to publish the analysis to JMP Public (where graph1 is the report to be published). I set it public so that everybody can consume the report).

webreport = New Web Report();
webreport << Add Report( graph1, title( "Graph1 Title" ) );
use_data = "true";

jmplive = New JMP Live( Connection( "JMP Public" ), Prompt( If Needed ));

jmpliveresult = jmplive << Publish ( webreport, Public (1));
report = jmpliveresult << As Scriptable;

Since this report is updated daily with new data, the existing reports need to be replaced.

jmpliveresult = jmplive << Replace( webreport, ID( "website ID" ) );

The website ID is the code after public.jmp.com/packages/ (see below), which enables you to replace the content of the existing report rather than having to create a new report with updated content.

So far so good! To get this published on a daily basis, I use the Windows Task Scheduler. In the Task Scheduler, I select a Basic Task, create a name for the task, and then have it triggered it so that it publishes the report every day at 8:00 a.m.

The last step is to tell the Task Scheduler which program to open and which action to take. Therefore, I defined the JMP16.exe path and the JSL script to be used.

Et voilà, now my report will be published and updated every day without pressing any keys. You will find the published report at JMP Public.

JMP 16 new features in Control Chart Builder – part 1

tonya_mauldin — Thu, 13 May 2021 19:41:54 GMT

Control Chart Builder is an interactive workspace which allows you to monitor process variation. This blog post details the new features introduced for Control Chart Builder in JMP 16.

Control Limits Labeled in the Graph

We had requests from users for the ability to label the control limits in the graph. This can be useful when the Limits Summary table is not displayed. “Show Limit Labels” has been added to JMP 16. To see an example of this, run the following JSL:

dt=Open("$SAMPLE_DATA/Quality Control/Shirts.jmp");
obj=dt<<Run Script("Control Chart Builder");

This code opens the Shirts sample data table and creates a C Chart. From the red triangle menu next to Control Chart Builder, select Show Limit Labels.

The control limits are now labeled on the right side of the graph.

K Sigma option

Process monitoring is often plagued by too many false alarms. This can lead to the alarms being ignored. The K Sigma option, introduced in JMP 16, gives users the ability to specify K. You can find this option in preferences. Choose File->Preferences. From Preference Group, select Platforms. From Platforms, select Control Chart Builder. In the Options section, near the bottom on the left side you will find K Sigma. Check the box and change 3 to 5.

As an example, run the following JSL:

dt=Open("$SAMPLE_DATA/Quality Control/Diameter.jmp");
obj=dt<<Run Script("Control Chart Builder");

This script opens the Diameter sample data table and creates an XBar and R chart.

The control limits have been calculated using +/- 5*sigma rather than +/- 3*sigma. This makes the control limits wider and can reduce the number of false alarms.

Remove Location Chart and Remove Dispersion Chart have been added to the right-click menu

In previous versions of JMP, you could remove location or dispersion charts by right-clicking on the Y axis and choosing the appropriate remove option. Additionally, you could simply grab the Y axis label for the frame of interest and drag it out of the graph. A third method has been added to JMP 16. Now you can right-click in the graph and choose Remove Location Chart or Remove Dispersion Chart. We are committed to providing our customers with options so they can find the method that works best with their workflow.

As an example, run the following:

dt=Open("$SAMPLE_DATA/Quality Control/Pickles.jmp");
obj=dt<<Control Chart Builder(
	Variables( Y( :Acid ) ),
	Chart( Position( 1 ), Limits( Sigma( "Levey Jennings" ) ) )
);

This code opens the Pickles data table and creates a Levey Jennings & Moving Range chart. Typically, a dispersion chart is not shown with a Levey Jennings chart. To remove the dispersion chart, right-click in the Moving Range chart and select Remove Dispersion Chart.

You are left with only the Levey Jennings chart.

Graph Spacing option to control the spacing between charts

Users had differing opinions about how much spacing they wanted to see between charts. A Graph Spacing option has been added to JMP 16.

For an example, run the following JSL:

dt=Open("$SAMPLE_DATA/Quality Control/Diameter.jmp");
obj=dt<<Run Script("Control Chart Builder");

This JSL opens the Diameter sample data table and creates an XBar and R chart. From the red triangle next to Control Chart Builder, select Graph Spacing. In the dialog, change the value from 2 to 5 and click OK.

Notice that the line separating the location chart from the dispersion chart is thicker. This option is also available in Control Chart Builder preferences.

Ability to create a Levey Jennings chart with more than 1 observation per subgroup

Previous versions of JMP did not allow a Levey Jennings chart if there was more than one observation per subgroup in Control Chart Builder. This functionality has been added in JMP 16.

As an example, run the following JSL to open the Diameter sample data table.

dt=Open("$SAMPLE_DATA/Quality Control/Diameter.jmp");

Select Analyze->Quality and Process->Control Chart->Levey Jennings Control Chart. Notice that a subgroup role has been added to this dialog. Specify DIAMETER as Y. Specify DAY as Subgroup.

Click OK. A Levey Jennings Chart is created.

This blog post details the following new features for Control Chart Builder in JMP 16:

Show Limit Labels
K Sigma
Remove Location Chart and Remove Dispersion Chart right-click options
Graph Spacing
Subgroup role for Levey Jennings charts

Part 2 of this blog series will be coming soon in which I cover JMP 16 new features for tests and alarms in Control Chart Builder.

New features for Distribution in JMP 16

tonya_mauldin — Thu, 13 May 2021 19:43:54 GMT

The Distribution platform can be used to explore the distribution of a single variable. This is a rich platform that allows many different visuals, statistics, and tests on variables. This blog post details the new features to the Distribution platform in JMP 16.

Student's t Distribution

The Student’s t distribution has been added to the list of available continuous distributions that can be fit to variables. The Student’s t distribution is a robust option that spans the space between a normal distribution and a Cauchy distribution. As the degrees of freedom in the Student’s t distribution approach infinity, the distribution is equivalent to the normal. When the degrees of freedom in the Student’s t distribution equals 1, the distribution is equivalent to the Cauchy. The default behavior is to estimate the degrees of freedom for the Student's t distribution. As an alternative, you can choose to use the Fix Parameters option to specify a particular DF (degrees of freedom).

As an example, run the following JSL:

dt=Open("$SAMPLE_DATA/Body Fat.jmp");
obj=dt<<Distribution( Continuous Distribution( Column( :"Neck circumference (cm)"n ) ) );

This code opens the Body Fat sample data table and performs a distribution analysis on Neck circumference (cm). From the red triangle next to Neck circumference (cm), select Continuous Fit->Fit Student’s t.

Better broadcast functionality for prediction intervals, tolerance intervals, and custom quantiles

In previous versions when prediction intervals, tolerance intervals, and custom quantiles were broadcast across responses and by groups, you would get multiple dialogs for every response and grouping. Beginning with JMP 16, you get a single dialog that applies to all groups.

As an example, run the following code:

dt=Open("$SAMPLE_DATA/Cities.jmp");
obj=dt<<Distribution(
	Continuous Distribution( Column( :POP ) ),
	Continuous Distribution( Column( :OZONE ) ),
	By( :Region )
);

This code opens Cities.jmp. It performs a distribution analysis for POP and OZONE by Region. Hold down the Ctrl key while clicking on the red triangle next to POP and select Prediction Interval. Click OK. In previous versions of JMP, you would have been presented with 12 dialogs (one for each response and by level combination). You would have had to click OK on each dialog. Now, the single dialog applies to all 12 combinations.

In addition to prediction intervals, this functionality was added to tolerance intervals and custom quantiles.

Show Within Capability option and preference

Users requested the ability to obtain a capability analysis in Distribution that by default does not show the Within Sigma Capability report. In JMP 16, a Show Within Capability check box was added to the Process Capability dialog and a Show Within Capability option was added to Distribution preferences.

To see an example, run the following code:

dt=Open("$SAMPLE_DATA/Cities.jmp");
obj=dt<<Distribution(
	Continuous Distribution(
		Column( :OZONE ),
		Fit Normal
	)
);

This code opens the Cities sample data table and fits a normal distribution to OZONE. From the red triangle next to Fitted Normal Distribution, choose Process Capability.

Enter 0.075 for LSL. Enter 0.15 for Target. Enter 0.25 for USL. Click on the open/closure icon next to Process Capability Options. Uncheck the box for Show Within Capability. Click OK.

Capability analysis is added to the report window, but the Within Sigma Capability section of the report is not shown.

JMP 16 introduced three main new features for the Distribution platform:

The Student’s t distribution.
Better broadcast functionality for prediction intervals, tolerance intervals, and custom quantiles.
Options to get a capability analysis with no within sigma capability statistics by default.

Deprecating the Name() parser directive in JMP 16

nickholmes13 — Tue, 06 Apr 2021 19:48:17 GMT

Beginning in JMP 16, JMP ceases to promote the Name() parser directive commonly used to reference variable names that break typical syntax rules for Names in JMP.

This change comes in response to many customer complaints that the syntax, use cases, and functionality for the Name() parser directive was confusing.

Instead, JMP will advocate for the use of a new syntax form with the name quoted as a string and followed by the character “n”.

Here are a couple of examples of variable names and column names with illegal characters and how they are referenced with the old Name() parser directive compared to the new syntax:

Old:

Name( "taxable income( 2011 )" ) = 456000;
tax = .25;
Print( tax * Name( "taxable income( 2011 )") );
//114000

New:

"taxable income( 2011 )"n = 456000;
tax = .25;
Print( tax * "taxable income( 2011 )"n) ;
//114000

Old:

dt = Open("$SAMPLE_DATA/Body Fat.jmp");
dt:Name("Age(years)") << Set Modeling Type(Ordinal);

New:

dt = Open("$SAMPLE_DATA/Body Fat.jmp");
dt:"Age(years)"n << Set Modeling Type(Ordinal);

To clarify any confusion, this new syntax was added to JMP a few releases ago and, if you prefer, you can still use the classic Name() parser directive for the time being. However, since Name() is being deprecated, it will eventually be removed in a future version of JMP and no longer recognized. In JMP 16, the new syntax is documented and used in scripts generated by JMP.

Keep in mind that, in most cases, it may likely be easier to use functions like Column() and As Column() to reference data table columns as strings, but the "colName"n syntax is a great option for variable names that break the typical syntax rules.

Examples using Column() and As Column() to reference “Age(years)”:

dt = Open("$SAMPLE_DATA/Body Fat.jmp");

//Change the modeling type of "Age(years)" column
Column(dt,"Age(years)") << Set Modeling Type(Ordinal);

//Add 10 years to each age
For Each Row(dt,
	age10 = As Column("Age(years)") + 10;
	Show(age10);
);

Please reach out to JMP Technical Support if you have any questions or concerns!

Hover labels: Understanding the extensions framework

LisaG — Tue, 06 Apr 2021 18:44:11 GMT

In a previous post, you learned about graphlet presets and the Paste Graphlet command. We showed you how they support context-aware, clickable visualization thumbnails in hover labels. These commands implement the "porcelain" aspect of the JMP 15 hover labels extensions feature, providing a user-friendly interface that doesn’t require any coding skills.

In this post, you’ll get a better look at the underlying "plumbing" layer that powers the porcelain layer. This layer is designed as a framework that can also be accessed directly by the user, allowing the implementation of requirements that aren’t covered by the porcelain features.

The hover label extensions framework

A framework is a combination of basic services and extension points. From the point of view of the framework, presets and custom graphlets are identical, so the only difference is that JMP provides the preset definitions as a convenience to users.

At the framework level, all graphlets – including presets – use the Set Graphlet command, but each preset initializes the framework with a different set of arguments. These arguments include JSL script fragments, later used as callbacks. The framework evaluates these callbacks in response to user interactions, such as hovering over a visual element or the clicking on a hover label thumbnail.

Callback evaluation happens inside an Hover Label Execution Context (HLEC), which captures contextual information about the underlying visual element. Callbacks can take advantage of this contextual information to implement dynamic behavior. Take, for example, the Histogram preset. After looking at the context to find the baseline visualization Y role assignment, it then reuses that column for its own Y role assignment.

Other entry points to the hover label extensions framework include Paste Graphlet (a wrapper for Set Graphlet), Set Textlet, and Set Gridlet. In this post, we’ll use the Set Graphlet command as an example, but the same principles apply to the others as well.

The best way to create custom graphlets is to use the Hover Label Editor dialogue. You can also use the Hover Label Editor to inspect how a graphlet preset is defined. Preset definitions provide great examples of how to use the framework capabilities.

Graphlet callbacks

The Hover Label Editor Graphlet panel exposes a few script arguments, the most important ones being Picture and Click. The framework will call them at runtime in response to the following events:

Hovering: When the user positions the mouse pointer over a visual element for a short period of time, like when the user pauses the mouse over a bar in a bar chart, the Picture script fragment is executed. Doing so results in producing either a PictureBox (a display box that contains a graphics picture object) or a display tree that contains a PictureBox. If the script calls for the execution of a Platform, it will be launched inside an invisible window. The framework will then traverse the display tree of the result and then capture and scale down a screenshot of the first PictureBox encountered. That image becomes the thumbnail displayed inside the hover label.
Clicking: When the user left-clicks on the hover label graphlet thumbnail, the optional Click argument is triggered. If it’s not present then the Picture script is reused, but the execution of a contained Platform results in a visible window. This is how presets launch their visualizations.

The Click callback can execute arbitrary JSL scripts. For example, it can open web pages, make REST calls, load files from disk and update tables. It allows integration with external data, systems, and other advanced scenarios.

As mentioned before, the evaluation of callbacks happens inside execution contexts, which have two components:

A Local Data Filter that subsets the rows to match the visual element data.
A Local Namespace that captures the visual element attributes.

Local Data Filter

Each visual element in a graph corresponds to a subset of the original data. For example, let’s look at a bar chart showing the average of students’ height by sex. The first bar corresponds to the female students; the second one is for the males. This implicit grouping happens during the definition of a visualization.

The framework makes grouping explicit through an equivalent Local Data Filter (LDF), which is used to restrict the data table rows visible during the execution of the callback. The LDF definition is based on:

The baseline visualization graph type.
The columns assigned to its grouping roles (X, Group X/Y, Overlay, etc.).
The values associated with these columns.

In the example below, the baseline graph is a small multiples bar chart. The framework maps it to an LDF based on its two grouping variables:

The one assigned to the X role (age).
The one assigned to Group Y (sex).

Note how the hovered bar and the panel that contains the chart generate the filter values.

Additional control over column inclusion is possible through the Skip Filters attributes. While it is rarely necessary, it can be useful if you want to display more data than is usually associated with the underlying visual element.

The core mechanism behind drill-downs is the ability to launch Platforms with context-based LDFs in response to user interactions.

Local namespace

The second component of the HLEC is a local namespace. Its main purpose is to expose context information to the graphlet, which allows the graphlet to exhibit dynamic, context-aware behavior. The framework wraps each graphlet callback in a local namespace before executing it.

The graphlet local namespace is populated by the framework with local variables. These context variables follow a naming convention, meaning their names must always start with an underscore. They capture information from different sources associated with the current execution context:

Hovered visual element type (bar, marker, etc.).
Index of the first row of the associated data.
Number of observations in the associated data.
Hover label grid entries (one variable for each entry).
Type of callback (Picture, Click, etc.).
LDF where clause.
Drill-down levels count.

An easy way to check a new graphlet’s values when debugging it is to use the Textlet context preset, which adds the values as text to the hover label itself, as seen below:

Putting it all together

The following diagram shows how the different parts of the framework come together in the execution of a graphlet preset. In a custom graphlet, the user provides the graph template using the Picture script argument.

First, the framework captures a graph template and other arguments at configuration time. Later, in response to user interactions, it creates the filter and namespace that are part of the HLEC. It instantiates the template inside the HLEC to generate the hover label thumbnail image. If the user clicks on that image, it launches the associated platform on its own window.

A custom graphlet example

Now that you understand the mechanics, here’s an example of how accessing the underlying graphlet framework helped a user solve a real problem.

Not too long ago, a user asked a question here in the JMP User Community. She had a folder with image files (specifically, high-resolution pictures) of her experiments. She wanted to display the image associated with an experiment in the hover label, but the images were too large to store in her JMP tables. Was it possible to load them from the disk on demand?

There were a few obstacles to making this happen. It didn't align with any of the existing presets, it wasn't a good fit for Paste Graphlet since it required dynamic behavior, and the filename for the displayed image had to calculated on demand.

The solution was to create a custom graphlet by extending the framework with callbacks based on JSL file primitives.

The principles of that solution are shown below in an example based on the Iris dataset. The critical part of this example is showing how the image can be loaded directly from the disk. The filename is based on one of the HLEC local namespace variables. As a bonus, you can see how to open a related web page in response to the user clicking on the displayed image.

Set Graphlet(
	// Value from hover label grid entry (labeled column)
	Picture(
		Picture Box( Open( img_dir || local:_Species || ".jpg", "jpg" ) )                                                                                       
	), 
	// Value from data table using column reference for visualized row
	Click(
		Web( "https://en.wikipedia.org/wiki/Iris_" || Column( "Species" )[local:_firstRow] )
	)
)

Conclusion

While we hope that you are always able to use presets and paste graphlets to implement the majority of your requirements, we realize that’s not always possible. By using the framework, you can create your own custom hover label extensions whenever you have a special need.

If you have questions, want to share your creations, or just see what other users are coming up with, please join us here in the JMP User Community. We look forward to seeing your next cool graphlet!

New features in the Column Switcher for JMP 16

paul_vezzetti — Fri, 19 Mar 2021 19:31:29 GMT

The Column Switcher lets you quickly analyze different variables without having to re-create your analysis. JMP 16 includes several usability and functional improvements to the Column Switcher. Below are some of the highlights.

Improved Launcher Dialog

There is now a single dialog for configuring the Column Switcher. The dialog includes two lists.

The top list contains the columns used in the platform which are available for switching. Only one column in this list can be selected at a time. This is called the initial column.

The bottom list contains the columns from the data table which can be switched into the platform. These are the replacement columns and this list supports multiple selections. The bottom list is dynamic as the contents might change depending on the initial column selected. Both lists support various filtering and sorting operations under the red triangle menu.

There are additional options available under the Options outline. The Title allows you to set the title for the outline around the Column Switcher. The Column Switcher can also be launched in the closed position by checking the Close Outline checkbox.

Column Switcher Improvements

If you forgot a column when creating the Column Switcher or added a new column to the data table after creating the Column Switcher, you can now add (or remove) columns from the column switcher list after creation. There are add (+) and remove (-) buttons next to the Column Switcher list which launch separate dialogs to manipulate the column list. The add dialog will only allow new columns which are suitable for the current platform and switch column.

Animation Looping

There is a new button shown below in the animation controls which controls whether the animation continuously loops over the available replacement column or if it stops when it gets to the bottom of the list. By default, looping is enabled. However, toggling the button will disable the looping. This can be useful if you are recording the animation and you only want to capture one pass through the columns.

Retaining Axis Settings

By default, most platforms reset the axes when switching columns. This makes sense if the data on the axis is significantly different between columns and you want to see the full extent of the data for each column. However, there are times when you want the axis range (and potentially other options like reference lines) to remain so that you can compare the columns in the same context.There is a new setting under the red triangle menu called Retain Axis Settings. This is off by default. If turned on and there is some customization of the axis such as setting the scale or adding a reference line, then those settings will not be lost when switching between columns.

Here is an example where the axis range and reference line are locked while the Column Switcher is running. This allows the user to see how the data changes over time without losing the context on the y-axis.

Linking Multiple Platforms to the same Column Switcher

This is a more advanced feature that is only supported currently through JSL and in the Dashboard Builder. It allows a single Column Switcher to control multiple platforms. This creates some complexity as all of the platforms must contain the same initial switching column and must support the list of replacement columns.

The process of linking a platform to an existing Column Switcher is relatively easy. You just need a reference to a Column Switcher and a reference to a platform and then use the Link Platform command. The platforms don’t need to be in the same window although it is often easier to track the interaction if they are.

Example Linking Two Platforms in Separate Windows:

// Open the data table
dt = Open( "$SAMPLE_DATA/Process Measurements.jmp" );
// Create a distribution platform
dist = dt << Distribution(
	Continuous Distribution(
		Column( :Process 1 ),
		Always use column properties( 1 ),
		Process Capability( 0 )
	),
	Histograms Only
);
// Add a column switcher to the distribution
cs = dist << Column Switcher(
	:Process 1,
	{:Process 1, :Process 2, :Process 3, :Process 4, :Process 5, :Process 6,
	:Process 7}
);
// Create a Fit Y by X platform
biv = Bivariate( Y( :Process 1 ), X( :Process 2 ) );
// Link the platform to the column switcher
cs << Link Platform(biv);

Example Combining the Platforms into a Single Window:

// Open the data table
dt = Open( "$SAMPLE_DATA/Process Measurements.jmp" );
// Create a new window to hold the platforms and column switcher
New Window( "Multiple Platforms with one Column Switcher",
	H List Box(
		V List Box (
			dist = dt << Distribution( 
				Continuous Distribution(
					Column( :Process 1 ), 
					Always use column properties( 1 ),
					Process Capability( 0 )
				),
				Histograms Only),
			biv = Bivariate( Y( :Process 1 ), X( :Process 2 ) )
		),
		cs = dt << Column Switcher( 
		:Process 1,
		{:Process 1, :Process 2, :Process 3, :Process 4, :Process 5, :Process 6,
		:Process 7} )
	)
);
cs << Link Platform( dist );
cs << Link Platform( biv );

I've attached an example of a linked Column Switcher in a dashboard to this article. This example uses the Process Measurements.jmp data file from the Samples directory.

Preferences

There are now Preferences that can be set for the Column Switcher. These are intended to make it easier to create new Column Switchers.

Always include switch column in replacement list
A common mistake when creating a Column Switcher is to forget to include the initial switch column in the list of replacement columns. As a result, once you switch away from the initial configuration you can’t get back because that column is not in the list. With the improvements in JMP 16, you could use the Add button to add the initial column to the list. However, this is extra work. Also, the launch dialog now prompts you if you don’t include the switch column in the replacement list so this is less likely to happen. This preference makes it even easier since it will automatically include the initial column in the replacement list even if you don’t include it and it won’t prompt you to add it during the launch.

Retain axis settings when switching
This preference changes the default for Retain Axis Settings from OFF to ON.

Run Animation in loop
This changes the default for the animation loop button.

Default Outline Title

By default, the outline title for the Column Switcher is “Column Switcher”. This can be changed during the launch in the Options section. The preference can be used to change it for all new switchers.

These are some of the highlights of the new features in Column Switcher. There have also been numerous bug fixes and other improvements.

The Enhanced Log in JMP 16: This changes everything

Jordan_Hiller — Thu, 18 Mar 2021 19:23:34 GMT

The JMP log has received a major upgrade in JMP 16. If you’ve never had a reason to look at the JMP log before, you do now: The new Enhanced Log functions as a living record of your current JMP session.

Here’s how it works. Launch JMP and go about your point-and-click business as usual: Import data, clean it up, examine distributions, make graphs, do some analysis. When you’re done, open the Enhanced Log from the View menu. You’ll find a replayable, archivable record of your work. Let’s look at an example:

The Enhanced Log window is divided into two panels. The top panel has a list of actions that were performed; the bottom panel contains the corresponding JSL code.
Each action captured has information about the origin (usually what data table the action was performed on) and the result. There’s also a timestamp.
Different data tables and report windows are color-coded to aid with navigation. Click on one of the color-coded blocks to bring forward the corresponding window.
Search for specific steps using the Filter box at the top of the Enhanced Log window.
Any superfluous actions can be deleted if desired: Click on the row to select it, then choose “Clear” from the red triangle menu or right-click context menu.
Many JMP platforms will generate two recorded actions: one capturing the platform launch dialog choices, and a second snapshot generated automatically when the resulting report is closed. The snapshot helpfully captures any details that were added to the report later from red-triangle options.
To save the actions as a JSL script, choose “Save Script > To Script Window” in the red triangle menu.

The Enhanced Log is something we'll be showing everyone who starts scripting in JMP .

Why is this such a game changer? Two main reasons:

Documentation: By capturing the code corresponding to the most important point-and-click actions, the log serves as a record of your work. Save the log code to document your work.

Repeatability: When your data is updated with new values, you can replay the previous data cleaning and analysis steps quickly and accurately.

We hope you’ll explore the Enhanced Log in JMP 16, and let us know what you think!

Discovery Summit Online: The most interactive conference you'll find

Jeff_Perkinson — Tue, 02 Mar 2021 14:32:03 GMT

We've built a beautiful virtual conference space so that we can all be together.This time last year, only a couple of weeks before Discovery Summit Europe, a conference we hold annually for JMP users, we had to cancel the in-person event. A global pandemic was here, and we didn’t want to be part of spreading COVID-19.

We quickly pivoted – scheduling a couple of our plenary speakers and a handful of papers from our user program to present at our first virtual conference. It was not nearly the equivalent of the in-person conference we had planned, but we felt pretty good about doing something in such a short time.

As soon as we were done with the frenzy of reworking that conference, we had six months to plan for the next one. And now this is our third one.

We have successfully designed a conference that is different from every other virtual conference.

The Discovery Summit difference

I should give you a little background on Discovery Summit. We like to think of it as the embodiment of the values represented in our software – interactive, highly visual, full of statistics know-how and fun. We bring together the most curious scientists and engineers; the best analytical minds; and JMP staff, developers, testers, and regional experts to spend three or four days immersed in conversations and explorations about how to build a culture of analytics within organizations and methods to solve engineering challenges.

Of course, like other conferences, there are keynote talks from experts and thought leaders. And we recruit a robust program of presentations from our users and staff. But it doesn’t end there.

Our attendees regularly call Discovery Summit the best conference they’ve ever attended. What sets Discovery Summit apart? The emphasis on interaction between attendees and with our staff. We want the value from the conference to come not only from the stages, but also from the audience. We plan for plenty of time for questions from the audience in our breakout talks, and we encourage the audience to challenge the speakers with tough problems.

We place demo pedestals prominently, where we station our staff of experts to engage with our users. We give advice on how to use JMP effectively and solicit feedback on how to improve the software to make it even more valuable. These conversations happen in the expo hall where users from various companies and industries can listen in and help each other.
During lunch and break times, we designate tables with special interest topics where like-minded individuals can meet each other and share stories and advice.
We plan social hours at the end of each day to give people a chance to relax and get to know each other as people outside of the analytic challenges we’re all trying to solve.

This adds up to an experience that is unique in the world of conferences. It’s exciting, exhilarating and exhausting, but so worth it.

What is a virtual Discovery Summit?

That brings me to what we’ve done to recreate this experience online. You see, we take pride in the atmosphere of our conference. So, we’ve thought hard about how to bring the things we value to a virtual conference.

We know that you are tired of virtual meetings and watching videos on your computer, so we don’t simply want to be another source of “content” to be consumed. We want the time you spend with us to be interactive and exhilarating. We want you to be an active participant in the conference, not a passive viewer.

For this reason, we use a conferencing platform that allows us to see and hear each other. We encourage – but don’t require – attendees to turn on their cameras, so you can see who’s “there” and interact with them as if you were physically together. We want you to know that we’re real people. We want to have a human connection with you.

However, we recognize that some of the content from the conference, specifically the user presentation program, benefits from asynchronous consumption. That is, it is good reference material and meant to be watched at your leisure and when you need it. As such, we have pre-recorded those presentations and made them available two weeks before the conference and beyond. These are stellar presentations that deserve to be watched, re-watched and shared.

During the four scheduled days of the conference, we put the emphasis on the content that benefits from an audience engaged with each other:

Keynote presentations that are meant to inspire and put you in the right frame of mind to tackle the challenges you face. For that reason, we deliver these in real-time. You’ve got to show up to hear these speakers. We will open the floor for questions, giving you the chance to challenge the speakers – just like you would in person.
"Semi-live" presentations from users and JMP experts, where the recorded presentation is played and the author stands by to answer questions (either in chat or verbally) during and after the presentation. Again, if you want the chance to hear from the speaker directly, you’ve got to show up to participate.
Meet the Developers with live video and screen sharing gives you the opportunity to get to know our expert staff. Take this time to present them with the challenges you face and get advice from the people who know analytics and JMP the best.
Birds of a Feather sessions facilitate the happenstance networking that occurs when you bump into a fellow attendee. We host breakout rooms with live video and audio for the most curious, brightest engineers and scientists from many industries and companies to gather based on common interests. Again, this interaction is only available during the conference.
Social hours are designed to help you unwind after a full day of interaction and learning. Again, the networking opportunities here give you a chance to put a human face with other users and experts in the JMP community.

Finally, we’ve built a beautiful, virtual conference center to help put everyone in the same “place” at the same time. Easily navigate to the plenary sessions or the breakout talks. Find the Birds of a Feather and Meet the Developers sessions just as you would in person. It’s a small touch, but it helps to ground you and let you know you’re not the only person at this virtual conference.

We know that Discovery Summit is a different kind of conference – and we want you to see that too. Join us and experience it for yourself.

Sharing is caring: A few reasons to fit structural equation models in JMP Pro 16

LauraCS — Thu, 04 Mar 2021 18:16:44 GMT

“There is no delight in owning anything unshared”

Lucius Annaeus Seneca the Younger, Roman Stoic Philosopher

When you’re involved in a really fun project or know a really cool trick, you want to share it with the world…especially when you believe this knowledge will help others. My really fun project and collection of exciting tricks are embodied in the structural equation modeling platform in JMP Pro. Sharing the unique features of our structural equation models (SEM) platform brings me absolute joy! Why? Because I spent countless hours, spanning more than 10 years, fitting SEMs in difficult-to-use software; believe me when I say that I know the many pain points alleviated by the SEM platform in JMP Pro.

As a statistical software developer, I’ve been privileged with the opportunity to revamp SEM software, to rethink the approach of fitting SEMs and to facilitate the process of doing so. After a little more than two years of development work with an outstanding team, I couldn’t be happier with the outcome! And best of all, we are just getting started.

I’ve written before about the unique advantages of fitting SEMs in JMP Pro 15, but with the release of JMP Pro 16, there’s a lot more to celebrate:

What’s the buzz about?

Efficiency is everything. There are major gains in workflow when models are collected in one place. Statistical comparisons, tweaks in specification, and side-by-side comparison of results are just some of the things you can do much more efficiently in JMP Pro, especially when paired with interactive shortcuts. Say goodbye to folders with dozens of individual model output files. Preview some of the time savings you can enjoy in this video.

Path Diagram functionality is unparalleled. If you’re like me, the insights your models unveil are more important than the path diagrams that illustrate them. But JMP’s outstanding path diagrams are likely to make you cherish them just as much because they convey insights like no other! Indeed, I felt compelled to write a whole other blog post (with cool demo videos) just on this topic: Can your SEM path diagrams do this?

Visualizations speak louder than words. Whether you’re seeking to explain SEM results to students, to stakeholders in your company, or to a funding agency, good visualizations will always help you make your case. The SEM platform in JMP Pro has powerful visualizations peppered around multiple statistics – from residuals to effect profiling to measures of reliability and validity for surveys and tests.

Alerting you to trouble…before it happens. After fitting a model, nobody likes reading obscure error messages that require difficult (and time-consuming!) detective work to resolve. The SEM platform in JMP Pro has a dynamic status indicator to alert you – during model specification – to potential trouble down the road, saving you lots of time and headaches. Check out this video for a taste of how we help you avoid common mistakes.

So now that I’ve revealed our great collection of tricks, don’t hold back on the delight of sharing them with others! And if you’ve never used our software before, a free introductory webinar will ease the learning curve so you can get started with SEM right away. You can also read a SEM-specific tutorial published in the Structural Equation Modeling journal (coming March 15, 2021), or read about how a professor and CEO enjoys using our SEM platform in his research.

Can your SEM path diagrams do this?

LauraCS — Mon, 01 Mar 2021 21:33:49 GMT

A central feature of structural equation modeling (SEM) is the availability of path diagrams that convey statistical models intuitively. For example, a model can be expressed as a system of linear equations or a path diagram that even a young child can understand:

Factors that predict the taste of candy expressed as a system of structural equations and as a path diagram.

If you’re an SEM user, we both know the countless hours you’ve spent drawing and editing path diagrams so you can convey your results to your audiences. If you’re not a current SEM user, then you can avoid the pain of using secondary software to draw those diagrams just right.

The SEM platform of JMP Pro 16 has amazing features, including presentation-ready path diagrams that can be copied as vector images to be used in presentations and publications (yes, even this pink candy diagram was created in the SEM platform!). We don’t want you to spend too much time doing customizations; after all, you’re already spending time doing high-level stats! So we’ll get you as far as we can, leaving enough room for you to express your artistry, but only if you so desire.

Interactive model specification

Path diagrams in the SEM platform are fully interactive, enabling users to specify models directly on the canvas or make changes to previously specified models. All elements are selectable, and context menus unveil options to add latent variables, regressions, covariances, and even set constraints. And if your models are too complex for drawing directly, then you can always check out our list UI to get the best of both worlds: efficiency of specification and visual appeal.

It’s simple to convey results

Dashed lines, bold lines, transparency…you name it, we take advantage of it all. Arrow characteristics are mapped to effect sizes and statistical significance, making it easy to process model results visually. And if that weren’t enough, dependent variable nodes are shaded according to the proportion of variance explained by their predictors, because a picture is worth a thousand words!

Statistical model = path diagram (really!)

There are numerous conventions for drawing path diagrams, and not everyone agrees on which convention is best. RAM notation (stands for reticular action model, McArdle & McDonald, 1984) is my favorite because diagrams depict statistical models with full precision: If you see it, it’s there; if you don’t, it’s not. Alas, it’s not without its disadvantages, as diagrams can easily become cluttered, sometimes to the point where they stop being useful – especially if visualizing the mean structure.

We tackled this issue in the SEM platform by using RAM notation as default while allowing users to hide elements of the diagrams at their discretion (but we’ll let you know if you’ve hidden something important to avoid specification mistakes). This can be useful, for example, when focusing on structural relations between latent variables, or when many covariances obstruct the visualization of regression effects. The cluttered path diagrams above can be improved by hiding the means and loadings.

This video shows how simple it is to clean up your path diagrams to convey results effectively.

Appearance

Often, less is more when it comes to path diagrams, so our default path diagram is clean and simple. However, you may have your own reasons for changing the appearance of path diagrams, and our customize menu allows you to do just that – from node sizes and arrow thickness, to colors and fonts, the choice is yours!

Interactive Layout

We don’t want you to feel like you are stuck with our layout choices. You can interact with the path diagram elements and rearrange them to your liking. But we wouldn’t want you to spend too much time on this, so we have options to speed up the process. Check out a few of these in this video.

Creating great-looking path diagrams that convey information efficiently is really easy in JMP Pro!

References

McArdle, J. J., & McDonald, R. P. (1984). Some algebraic properties of the reticular action model for moment structures. British Journal of Mathematical and Statistical Psychology, 37(2), 234-251.

New in JMP 16: Improved covariate handling in DOE

Ryan_Lekivetz — Fri, 26 Feb 2021 20:20:20 GMT

If you happened to catch my previous blog post, you will know that I am a big proponent of the use of covariates in the Custom Designer when the need arises. For JMP 16, we have made some improvements to make it easier to use covariates.

New Look

The first thing you may notice is a new place for covariate specification. Under the Factors outline, there is a special outline labeled “Covariate/Candidate Runs.” The hope is that by bringing it more out into the open, more users will become aware of the ability to use covariates.

We still need a data table to load covariates, so we will use the “Runners Covariates.jmp” data table from sample data here. This table has a list of 100 runners who are candidates for an experiment on running shoe wear (first 10 shown here):

Say we wanted to use a subset of 24 of these runners with an additional controllable factor for shoe type (A/B). Clicking the Select Covariate Factors button, we can choose all three factors as covariates:

We now get an embedded table within the Custom Designer:

We can now add a factor for Shoe Type:

and modify the model, however we see fit.

Additional options

Note that the rows of the table with the candidate runs for the covariates in the Custom Designer are selectable. The reason for this is that there are situations where we want to ensure certain runs are in the final design. For instance, materials may have been used in the past or the experimenter has a commitment to use some certain candidate runs. Another possible case is to use a candidate set by selecting the “Include all selected covariate rows in the design” so the designer will make sure to all the selected runs in creating the design. This is the look of the design generation outline with covariates:

Another possibility is to use a candidate set for adding additional runs to an existing design, effectively an Augment Design using covariates. In this Discovery Summit Europe talk, one of the ways to create a design was finding a subset of the full factorial where the predicted value at a given point was above a specified value. The next batch of 48 runs was chosen from this set while accounting for the points in previous iterations of the design.

The “Allow covariate rows to be repeated” allows the Custom Designer to choose any of the rows more than once, which is useful when you just want to supply runs in a certain form. While you can have the same effect by replicating the design several times, it is not always clear how many replications of a design are sufficient.

One more thing

Much of the work for handling covariates is going on behind the scenes. Under the covers, previous versions of JMP only focused on D-optimality for the covariates. In JMP 16, the covariate rows are chosen according to the specified optimality criteria. The integration between covariates and controllable factors is also better handled to search through more possibilities.

Thanks for reading, and I hope you find the ability to use covariates handy in your own experiments!

New in Graph Builder for JMP 16

XanGregg — Tue, 02 Mar 2021 16:44:01 GMT

Here's a quick tour of many of the enhancements made to Graph Builder in JMP 16.

	Density Random is a new jitter option in the Points element, and it's the default whenever the number of dots is too many for the Centered Grid option. It's like Random Uniform, but the points are constrained to an envelope corresponding to a kernel density function (like a violin plot).
	A new Ruler tool is available for all graphs in JMP, not just Graph Builder. It's useful to measure the size of or distance between features, such as the low points in this example. As you make measurements, the results are written to the log.
	The Heatmap element now supports hexagonal binning. Hexagons are better than rectangles in many cases since they approximate a circle better for binning nearness and represent diagonal patterns with less pronounced aliasing.
	With a little scripting, you can create your own custom marker shapes for use in the Points element, dependent on any number of other variables in the data table. In this example, the arrow is a custom polygon which incorporates the ratio of two crime statistics to determine its direction.
	The Smoother element has many more choices for smoothing functions. In addition to the existing spline smoothers, there are now loess, Savitzky-Golay and moving average smoothers with many options themselves. Having moving average as a smoother, instead of a separate formula column allows it to follow the Local Data Filter and the Graph Builder grouping variables.
	Categorical axes now have a Size By property, which allows another variable to control the spacing of the labels. In this example, the bar widths are based on country population for a result sometimes called "mekkobars" since they resemble a mosaic or "marimekko" chart.
	Lines can now be labeled with key information: the first, last, minimum and maximum values, plus the overlay name. Labeling with the name can eliminate the need for a legend, and labeling key values can highlight important anchoring information without overload.
	The size of the filled area in heatmaps can now be scaled by a separate variable. The same applies to hexagonal heatmaps, and it's a useful way of showing counts of a cell when the color represents some other variable.
	The legends for size variables are explicit now, showing the actual marker sizes and the range of values.

And there are lots more new features that don't fit into a thumbnail image:

A new Summary Statistic option provides cumulative sums, which saves on adding a separate formula column and works better with filtering.
The legend can be dragged to anywhere in the graph, not just the predefined corners.
Run charts are easier to create. Instead of manually adding the row number to the X axis, you can leave it empty and choose the Row Order option on the Line element.
The box plot element has more drawing options to match the Distribution platform.
There's quick access to custom axis label reordering, right from the axis.
Custom shape files are easier – specify them right on the shape variable in Graph Builder instead of as a column property in the data table.
Bar labels can be moved with the mouse.
Date/time variables can have their granularity adjusted for aggregation and wrapping directly from the graph.

Finally, I'm excited about one new JMP graphics feature that only affects platforms outside of Graph Builder. Analyses like Control Charts that have semantic colors for bad-to-good now rely a preference for those colors instead of hard-coding red and green, so you can choose something more accessible if you like.

Moving from SPSS to JMP: A Transition Guide

Larry_LaRusso — Thu, 25 Feb 2021 01:28:40 GMT

Because each analytical software package handles, processes, and outputs analysis differently, transitioning from one software program to another can be intimidating. Once users become comfortable with a program it can be difficult to make transitions, either to another program or to a more recent version of the same program. This can especially be true for new JMP® users who learned SPSS either at another company or as a student.

This guide, written by Dr. Joshua Lambert of the University of Cincinnati, is meant to ease the user’s transition from SPSS Version 27 to JMP 16 and JMP Pro 16. In this white paper, we'll illustrate the differences between SPSS and JMP and show you how easy this transition can be.

To demonstrate these differences, we'll use data from one of the most infamous shipwrecks in history: the sinking of the RMS Titanic.

Topics covered in this paper include:

The full text of the white paper appears below.
(Note: You can jump directly to a particular section by clicking on the hyperlinks above.)

Introduction

The year was 1989, and Poison’s new song “Every Rose Has Its Thorn” topped the charts. The personal computer landscape was changing rapidly and was seeking to shed some of its thorny past through the newly introduced graphical user interface (GUI). The Apple Macintosh popularized the desktop GUI, and many software companies were aiming to create transformative products to leverage the GUI’s user friendliness and accessibility. At SAS Institute, John Sall and his team of developers released John’s Macintosh Project (JMP) Version 1 in October 1989. JMP Version 1 gave users a graphical front-end that allowed them to specify which analyses and options they would like the software to perform without the need to write code. In 1994, JMP 3.1 was released for Windows desktop environment. In 2020, the JMP team continues to introduce newly developed statistical methods while maintaining its long list of high-quality analytic tools in JMP 16 and JMP 16 Pro for both Mac and Windows. JMP’s point-and-click user-friendly environment makes it a popular tool for a variety of data science activities.

Statistical Package for the Social Sciences (SPSS) is a well-known and highly utilized program for the statistical analysis of data. It has a rich history in both academia and industry, and is the standard in certain disciplines. Originally developed in the pre-GUI era, SPSS found a niche early on by being among the first to switch to the GUI format. Like JMP and many other companies, SPSS developed a graphical front-end that allowed users to easily process and analyze data, thus delivering analytics to the masses.

JMP and SPSS data analysis workflows are fundamentally different. JMP is designed so that users have a dynamic and fluid link between their data and analysis. This link leads to a different perspective for data analysis, one which we will explore in this guide. SPSS requires a static workflow that makes users specify which analysis they would like to perform prior to seeing output. JMP walks users through an information cascade, from defining response and explanatory variables they would like to process to the more specific comparisons they would like to investigate. JMP’s user experience is designed to take users through interactive and dynamic data analysis.

In 2020, there are a multitude of software options for processing and analyzing data. Each software handles, processes, and outputs analysis quite differently, which can make transitioning from one software program to another quite difficult. This guide is meant to ease the user’s transition from SPSS Version 27 to JMP 16 and JMP Pro 16. The following example best illustrates the differences between SPSS and JMP and how to ease the transition from one to the other. This example is independent of the software under consideration, so the background and data set come from alternate sources.

Example

The sinking of the RMS Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.

The titanic.csv file contains data for 887 of Titanic’s passengers. Each row represents one person with the columns describing different attributes about the person: Survived (0=No, 1=Yes), Pclass (1=1st class, 2=2^nd class, 3=3^rd class), Name, Sex (Male, Female), Age, Siblings/Spouses Aboard, Parents/Children Aboard, and Fare.

Importing and Cleaning Data

SPSS

Upon opening SPSS, we can open many file types (CSV, Stata, SAS, Excel, etc.) using the options under the File menu. After a series of steps in which the software checks for things like delimiters and menus, we import the following file (be sure to uncheck Space under “Which delimiters appear between variables?” for the titanic.csv data):

A quick glance at the spreadsheet reveals eight variables and 887 observations. We see data on Survival, Passenger Class, Name, Sex, Age, whether a sibling or spouse was aboard, whether there were parents or children aboard, and the Fare (measured in £s).

SPSS has two methods of looking at each data set, the Data View and Variable View options, which can be seen via tabs on the bottom of each data set. When we open the data in Variable View, we see the data type of each variable and can modify that information by adding labels and changing formats.

There are already some things we need to take care of to clean this data and get it ready for analysis. Fortunately, most of the variables were read in their correct measure type. Age and Fare are measured as Scale or continuous variables, while the other variables are measured as Nominal. PClass, SiblingsSpouse and Parents Children variables should be changed to Ordinal types, which can be done in the Variable View by clicking under the Measure column. There is also no missing data, so there is no need to assure that the software is treating missing observations correctly.

In SPSS, we need to label categorical variables which are String (text) type, which is also done in Variable View using the Values option. There are two variables (Name and Sex) that are of the String type. For this example, we will not be using the Name variable. We do plan to use the Sex variable, so we need to add Values to the categories of Sex. This can be done by clicking the … button on the Sex row and Values column in Variable View.

The basic idea of presenting data in SPSS is to provide two viewing options that show the data in terms of the individual observations, as well as an overview of the type of each variable.

JMP

Switching over to JMP, we will open the same data and do the same manipulations. Let’s begin with what happens when starting the program. When JMP starts, the first thing we receive is a Tip of the Day and Home Window screen. The Home Window manages different windows for data, log and output. Recently used files are available on the left side so that we can conveniently reopen files that were used in a previous session.

Another feature of JMP is the JMP Starter window, which is found under the View menu. The JMP Starter is convenient for new users since it organizes commands for analysis into categories and provides summaries for each data and analytic option. We can open the Titanic data for this project by clicking on the Open Data Table command in the File category and navigating to the folder that contains the file, selecting it, and clicking Open. You can also open data directly from the File menu.

Like SPSS, JMP can open a variety of files including CSV files, Excel spreadsheets, SAS data sets, SPSS documents and text files.

Unlike SPSS’ Data and Variable View, JMP does not switch between multiple views in the data. Instead, there are table panels along the left side that do similar things.

The middle-left panel displays the column information and shows each of the eight variables and the data type. There are modeling-type icons for each data type: blue for continuous variables, red for nominal data and green for ordinal data. Note that the variables Survived needs to be changed to Nominal type and Pclass needs to be changed to Ordinal. Sibling/Spouses Aboard and Parents/Children Aboard can also be changed to Ordinal. To change the type, simply left- or right-click on the variable column header and select Column Info. The type can also be changed by right- or left-clicking on the icon (blue triangle or red histogram) on the left panel. The bottom-left panel summarizes the rows and shows additional information.

Let us change the Survived and Pclass variables. By right- or left-clicking on the blue icon next to the Survived variable on the left-middle panel, we can select Nominal, which turns the icon red. By right- or left-clicking on the blue icon next to the Pclass variable on the left-middle panel, we can select Ordinal, which turns the icon green. We could have also made these changes by right-clicking on the column headers and selecting Column Info.

There are other data types in JMP 16. (For a full description of the data types please refer to this link.) For instance, Name is technically Unstructured Text and not Nominal. This change can only be made by right-clicking on the column header and selecting Column Info and then selecting Unstructured Text under Modeling Type. Unlike JMP, SPSS does not have an option to treat variables as text type variables nor a platform to analyze text responses.

Like SPSS, we can also add value labels in JMP. To do this, right-click on the column header we would like to apply labels to and select Column Info. Then under Column Properties, select Value Labels. Unlike SPSS, JMP gives the us the option to use the Value Labels or the original values. When columns are given properties, JMP signals these changes with different symbols in front of the column names of interest. Here we see that asterisks have been created for the variable Sex. It should be noted that JMP has many other Column Properties that can help us better understand variability within our data such as Value Labels, Value Order and Units.

Visualization

Visualization is a powerful tool for analyzing and understanding data. The ability to easily make graphs via statistical software is one of the principal reasons why analytic software is so popular. Visualization is most easily handled in SPSS via its Chart Builder platform and in JMP via its Graph Builder platform. In this guide, we will discuss three simple yet powerful visualization tools: bar charts, histograms and scatter plots.

SPSS

Suppose we want to limit the discussion to the variables Age, Fare and Sex to visualize the distribution of each and see all three on one graph. We start with forming a bar chart of Sex. One quick and easy way to do this is with the Legacy Dialogues under the Graph menu.

For this data, we see that slightly more than half of the data points are male. Turning to Age and Fare, we can visualize this data in a histogram, which can also be found in the Legacy Dialogues.

In addition to the histograms, we also get some simple statistics for each variable (Mean, Standard Deviation and the sample size N). We can double-click on each graph in the output and customize as we see fit. Examining the distribution of Age and Fare, we see observations that had a much higher Fare than the rest.

For more complicated graphs, we use the Chart Builder option under the Graph menu. Using Chart Builder, we can build a scatter plot of Age versus Fare that is color-coded by Sex.

The Chart Builder dialog box is useful because it’s easy to use and can be customized. We simply click on the type of graph desired and drag it into the preview box. From there, click and drag the variables we want to use on the x and y axes, and drag the color variable onto the Set color? box. Once the graph is set, we can view our results by clicking OK.

In viewing this scatter plot, several points about the data can be made. First, there appears to be two unusual observations (one Male and one Female) for Fare which happen to be similar Ages. Perhaps they were traveling together? Those data points certainly look out of place with respect to the rest of the data. From the scatter plot, it is difficult to see whether there is a linear relationship between Age and Fare. Also under Fare, there appear to be many points of 50£ or so.

JMP

Before we look at data visualizations in JMP, recall from the introduction that JMP is designed to be interactive software. Most of this interactivity comes in the form of little red triangle icons in various places on the data and output that enables the user to explore additional analysis. SPSS output is mostly static, and further exploration requires reopening the graph of analysis platforms and reselecting options. We start again with a basic bar chart. Users can easily access graphing features via the Graph menu item from the top menu bar. For most graphs and charts, JMP’s dynamic and user-friendly Graph Builder is the logical choice. To make a bar chart, we select the Graph Builder, which also allows a variety of other graphs, such as pie charts, bubble plots and line graphs.

Graph Builder is an easy-to-use, drag-and-drop platform that allows us to explore and visualize data in engaging and interesting ways. Creating a bar chart for a categorical variable is first done by selecting the bar graph button on the ribbon at the top of the graph. Next, drag Sex into the x box at the bottom of the graph. As we drag the Sex variable around the graph, we see that JMP’s Graph Builder dynamically changes and gives us an actual view of what the graph will look like if we decide to drop the variable there. This is different from SPSS, which requires us to select OK before displaying the actual graph. Undoing that change is as simple as dragging the variable off the graph entirely or onto another section of the graph platform. Once Sex is dropped onto the x box at the bottom of the graph, a Bar options panel appears where we can select options for the new bar plot. For instance, including the % of Total over the bar is as simple as selecting that option under the Label drop-down. This option shows the proportion of the total sample that is Female and Male on top of the bar rather than nothing at all (default option).

Changing the axis to the percentage of the total is as easy as selecting % of Total under the Bar options in the Summary Statistic drop-down. All plot titles, axis labels and legend settings can be changed by either double-clicking or right-clicking and going to the setting to be changed. Graph Builder is very interactive, so changing a bar plot to a pie chart is as simple as clicking the pie button in the ribbon at the top. Clicking on Done removes the Graph Builder Control Panel and leaves only the graph that was created. This is useful when we would like to export the graph as an image file, which can be done by clicking File>Export and selecting one of JMP’s many file type options (e.g., Image, PDF, Microsoft Word). We can reopen the Control Panel by clicking the red down-pointing triangle and selecting Show Control Panel.

Another useful option under the red down-pointing triangle is Include Missing Categories if the variable has missing data. It shows the missing data on the graph alongside the Female and Male categories and includes it in all percentages shown. Also useful is the Recall button. Let’s say we spent a long time making the perfect graph only to realize that one data point was mislabeled. With JMP, that is no problem. Exit out of Graph Builder and go back to the data set and change the data as needed. Go back to Graph>Graph Builder and a magical Recall button appears. Click it and it will recreate the graph we just closed out – with the corrections we made to our data.

Moving on to histograms, we find a histogram icon in the Graph Builder ribbon. Starting with Age, we can obtain simple and straightforward output by dragging it from the variable list to the x box. We could have also dragged it to the y box to get a vertical histogram instead. As with the bar plot, useful histogram options can be selected on the left panel, including histogram style, the mean and standard deviation, or the counts displayed at the top of the histogram boxes.

Let us recreate the scatter plot that we created in SPSS that showed the male and female passengers who had an expensive fare and similar ages. Because JMP is dynamic and interactive, we can start with the Age histogram we already created. To transform this into a scatter plot all we need to do is drag the variable Fare onto the y axis and click the scatter plot button in the ribbon above the graph. Next drag the variable Sex onto the Color box, which is located to the right of the graph between Overlay and Size. And voilà! In just a few clicks, we have transformed our histogram into a scatter plot with points colored by Sex.

Again, we see what appears to be two unique individuals whose Fare is much higher than the other individuals. We list them as possible outliers, but this is where the differences between SPSS and JMP truly begin. JMP allows us to dynamically interact between the data and reports. When all the points are selected (not grayed out), we can hover over the points and get information quickly about the variable values used in the graph, as well as the specific rows in the data table those points correspond to. We can also select individual points to highlight a certain data subset of interest.

Returning to the data table, the selected rows show up as highlights in the data table, which gives us the option to quickly inspect the rows to determine exactly which observations are causing the controversy. If the data needs to be checked for consistency, then having the data and the output dynamically linked makes it easy to go back and inspect individual observations. There are many ways to clear data selections, but one of the easiest is to double-click on Selected in the Rows panel of the data table (likewise double-clicking on All Rows selects all the data).

From Graph Builder, we begin to see exactly how dynamic linkage separates working with JMP from working with SPSS. We can click on individual points in these graphs and select them for dynamic analysis. Doing this for our observations with high Fare shows that what we thought were just two observations was actually three! Miss Anna Ward, Mr. Thomas Drake Martinez Cardeza, and Mr. Gustave J Lesurer all had Fare of 512£. Miss Ward and Mr. Lesurer were both 35 years old, while Mr. Cardeza was 36. Interestingly all three passengers survived.

In SPSS we observed the possible outliers in the graph, but JMP takes it a step further and allows us to highlight those individual points and see them in the data and/or other report. In addition, we can exclude certain data points from analysis without having to delete them. Excluded data stays within the data table, but is not considered in analysis, thus making it easy to do analysis with and without potential outliers. There is also the Hide option if we wanted to remove individual points from visualizations but not analysis. This exclusion or hiding can be done by going back to the data table, right-clicking on the row that to be excluded or hidden and clicking the corresponding button (Exclude or Hide).

Another distinguishing visualization feature of JMP 16 is the header graph. Within the data table, there is a small histogram button on the left-most column (see figure below with blue circle). Clicking this icon will show graphs in the column headers for each variable, which is extremely useful and dynamic. By clicking on the various parts of the header graph, observations in the table that correspond to our selection are highlighted. Below, I selected those that survived (1), which gives me a quick view of the variable distribution across the other variables in my data set! Neat!

Descriptives

Descriptive Statistics

Descriptive statistics are at the heart of any good statistical analysis and, when performed properly, provide real insight beyond what is seen in visualizations. This section will focus on univariate descriptions and creating custom tables. Assuming we want to continue to study the variables Age and Fare, we will leave in the potential outliers and consider that data because it is still unclear whether the potential outliers should be removed from analysis.

SPSS

Many SPSS users know that the Frequencies dialog box can be a one-stop option for analysis of many different variables simultaneously. While there is a Descriptives dialog box, the Frequencies dialog box is nice because we can request descriptive statistics and graphs in one place. Also, we can put several different types of data into this option and SPSS will produce the appropriate graphs and statistics for each type of variable.

The options selected here give the standard univariate report, where we see the typical measures of center and spread that are discussed in basic statistics courses. Frequency tables, histograms, bar charts and many other useful descriptive outputs are also available under the Charts option.

SPSS Analyze: Descriptive Statistics: Frequencies MenuSPSS Frequencies Dialog BoxSPSS Frequencies Statistics OptionsSPSS Frequencies Statistics Output

JMP

Like the Frequencies dialog box in SPSS, JMP has the Distribution platform to obtain simple descriptive output. The Distribution platform is found under the Analyze menu. We start by selecting Survived, Sex, Age and Fare.

After clicking OK, a distribution report is displayed. By default, quantiles, mean, standard deviation and other summary statistics are produced for continuous variables, as well as a 95 percent confidence interval for the mean. Also, by default the variable outputs are unstacked and will be arranged horizontally. By clicking the red triangle and selecting Stack, the output is switched to a vertical view. Frequency tables are produced for categorical variables, and the Male subgroup (Sex=1) has been selected to highlight the interactivity between different portions of the same report.

By clicking on the red triangle, we can customize report display, obtain more descriptive measures or fit different distributions to the data. Add-ons include box plots, normal quantile plots, CDF plots and distribution fits. Being able to work with the output interactively allows us to fit many different theoretical distributions to the data (e.g., normal, exponential, Poisson, binomial, normal mixtures, etc.) and assess their relative fit to the data.

Another nice feature in JMP is the ability to save distribution reports as interactive html files that can be viewed and worked with interactively without the use of JMP software. This can be done through the File>Save As…>Save as type:>Interactive HTML with Data. These files make for good demos to students or can be worked into reports so that the results can be easily discussed between those who have JMP and those who don’t.

Descriptives: Custom Tables

SPSS

In many cases, specific output is necessary, like in our example where the visuals showed differences in Age and Fare by Sex. SPSS also has options to create custom tables for specific output. We can click and drag the variables into the proper locations and specify which descriptive output we want to see. To see the table, with numbers filled in, we must click OK first and then navigate to the results section to view the table. An example of this output is seen below

JMP

JMP makes custom tables via the Tabulate platform under the Tables menu. The Tabulate platform can build a table interactively, similar in spirit to Graph Builder. Again, SPSS requires us to drag the variables we wish to have in the table and then click OK before viewing the results. JMP gives us these results immediately after dragging and dropping the variable. We can see computations and output as we begin customizing our table. In addition, the interactive table can be turned off in favor of a traditional dialog box if we want to make a very specific output.

Descriptives: Correlation

SPSS

Using the Bivariate dialog box inside Correlate under the Analyze menu, we can obtain sample correlation estimates between Age and Fare. Note that there is significant positive correlation between the two variables.

JMP

JMP groups sets of analyses into one platform that allows us to do many different things (all somewhat related). For example, the easiest option for finding Pearson correlation coefficients is to use the Multivariate platform under the Multivariate Methods of the Analyze menu.

The Multivariate platform is much more than a simple vehicle for finding correlations; however, the basic report shown here delivers both a Pearson correlation estimate and a scatter plot matrix. By clicking on the red triangle, the histograms can be added along the diagonal so that distributions can be quickly visualized. The Multivariate platform also allows us to test correlations, find alternative measures of correlation (e.g., Spearman and Kendall), confidence intervals, partial correlations, and visuals such as Color Maps. In addition, correlation-based measurements such as principal components analysis, outlier analysis and item reliability are all available in the Multivariate platform.

Inference: Two-Sample t-Test

SPSS

Suppose we want to test whether Age differs significantly between those who survived and those who did not. The standard independent samples t-test is a popular option in SPSS, which is found under the Compare Means menu. Here we can specify one or more continuous measurements and determine if the mean value of that measurement significantly differs between two groups. In SPSS, both the equal variance and unequal variance tests are calculated, and we can rely on a test for variances to determine which test is the most appropriate.

JMP

While there is a button in the JMP Starter menu for t-tests, users should be aware that the t-test and one-way ANOVA analyses all go into the same platform. In fact, JMP combines all two-variable inferences into one Fit Y by X platform. Using the Fit Y by X platform, we can make inferences on bivariate comparisons for any type of data.

The grid in the bottom-left corner of the launch window lists different analysis types. We can choose which x and y variables we want to study and JMP chooses the appropriate method based on the type of data being used.

For now, we want to examine the differences between age based on survival status. In the report, we have simple descriptives, along with a t-test that assumes unequal variances (found by selecting Means and Std Dev and t Test from the red triangle icon). We may add graphics (such as box plots) and additional tests looking at the variances, or nonparametric tests more appropriate for skewed data.

Again, we see an important distinction in JMP, as it combines many similar tests into one common platform. We can start with a t-test and look at graphics and summary statistics to determine if the assumptions of a t-test are met. If they are not, then we may add on an alternate test that is more appropriate for the data considered.

Inference: Crosstabs/Contingency Tables and Chi-Square Test

SPSS

Creating a contingency table and performing a chi-square test of categorical data is an important component to many analyses. Examination of table data is one area where SPSS has always focused, and we can find many options for analysis under the Crosstabs listing in the Descriptive Statistics menu. The Crosstabs menus allow us to ask for several different sets of analysis based on what type of comparisons we want to make.

From this example, we can see that the rate of survival among women(sex=0) aboard the Titanic is more than three times that of males(sex=1): 74.2 percent versus 19 percent. From the Chi-Square Tests table we can also see that there is enough statistical evidence to conclude that there is significant association between survival status and sex (chi-square p-value =0.000). It should be noted that SPSS does not display (by default) p-values less than 0.001 but instead just displays 0.000. This may confuse some users if they believe that the p-value is in fact 0. In reality the p-value is just very small (less than 0.001). JMP displays p-values by default to more decimal places (four instead of SPSS’ three) and gives a clearer representation of p-values less than 0.0001 with the notation of <0.0001. When the expected counts are small, SPSS shows a warning message and provides a Fisher’s exact test of the difference among samples with small cell counts. If interested in relative risk and odds ratio estimates, we can check the Risk option in the statistics box to gain those estimates.

JMP

We can access the Contingency Analysis options in JMP via the Fit Y by X platform. Simply specifying categorical variables for both x and y will lead us to contingency analysis. We can specify the same table as we did in SPSS to look at the relationship between sex and survival status. The JMP report gives table information, a chi-square test and Fisher’s exact test (2x2 table only, same as SPSS).

We can then add on additional summaries and tests such as relative risk, odds ratios and measures of association (for items such as Kendall’s tau). The Contingency Analysis is considered a standard report, but we can also see the list of options that are available via the red triangle icon. An interesting aspect to JMP is that the software tries to help users in areas where they may be unfamiliar. By selecting the odds ratio option and simply hovering the mouse over that area, an explanation box appears that indicates which kind of analysis this option produces. For users having trouble interpreting statistics, p-values or other output, this option is available by simply circling or hovering over an area with the mouse.

Inference: Linear Regression

SPSS

Linear regression can be performed using the Linear Regression option under the Regression menu. Suppose we want to use Age as a predictor of Fare. The Linear Regression option in SPSS is a powerful tool with many different options for model-based estimates, measures of fit, diagnostics, predicted value output and visual plots to assess fit. We can incorporate many potential predictors and look at both simple linear regression and multiple linear regression in the same set of menus.

Without using additional options, we obtain standard output with R-square, ANOVA table and a table for regression estimates. We see a significant relationship between Age and Fare, and that as Age increases, the Fare of those individuals also tends to increase (p-value=0.001). It is important to note that the linear option does not allow string variables to be used as potential predictors. In fact there are many different regression options in SPSS, depending on what type of model needs to be fit. The linear option is limited to numeric covariates only; for multiple regression with both continuous and categorical covariates, refer to the General Linear Model platform.

JMP

We have two options for regression models in JMP: the Fit Y by X platform or the Fit Model platform, both under the Analyze menu. Simple bivariate fits go along the Fit Y by X platform, so let’s start there.

The Fit Y by X platform can again be employed to look at the relationship between two different variables. Here we have listed Age as X and Fare as Y. It is important to note that the Fit Y by X platform ONLY looks at bivariate relationships. It is possible to select more than one candidate for both the x and y roles. In such cases, output consists of multiple bivariate fits with each output displaying the appropriate measures for each set of variables.

The Bivariate Fit report starts with a simple scatter plot and we must assess exactly which type of bivariate fit we would like to make. Here we see a linear fit that models a simple linear regression line to the data. A standard report includes the same basic information as found in SPSS: R-square, ANOVA and parameter estimates with hypothesis tests for the intercept and slope parameters. In JMP, however, we can interact with this output to look at multiple fits simultaneously. For example, we can choose a simple mean, straight line and quadratic fits to the data and compare the fits by either R-square or looking at the parameter estimates. Other fits include density ellipses and smoothing splines, from which comparisons between fits can be made and estimated values can be output to the original data for further study. Under each fit, we can specify diagnostic measurements and plots to assess whether the assumptions for using each particular model are satisfied.

JMP puts many different types of regression models all under the Fit Model platform. In the example shown here, we are fitting a multiple linear regression model that predicts Fare via a combination of Sex, the ordinal variable Pclass, and the categorical variable Sex (no need to separate categorical and continuous covariates). Clicking Run displays the Fit Least Squares output, which includes an actual versus predicted plot, leverage plots, residual by predicted plot and more. More options, such as profile plots, are available under the red triangle.

Inference: Logistic Regression

SPSS

Logistic regression can be performed using the Binary Logistic option under the Regression menu. Suppose we want to use Age as a predictor of the binary variable survival. The Binary Logistic option in SPSS has many different options for model-based estimates, measures of fit, diagnostics, conversion to odds ratios and more. We can incorporate many potential predictors and look at both simple logistic regression and multiple logistic regression in the same set of menus.

JMP

Like linear regression, we have two options for logistic regression models in JMP: the Fit Y by X platform or the Fit Model platform, both under the Analyze menu. Bivariate fits go along the Fit Y by X platform, and multiple logistic fits go in the Fit Model platform. Once OK is clicked, we have many pieces of useful information about our model at our fingertips. More options, such as Odds Ratios, are available under the red triangle.

Data Manipulation

Both SPSS and JMP are excellent for manipulating data since they each provide a number of different ways to organize and manipulate the data. We can sort data, sort variables, merge files, recode and run many other manipulations. Listed below is a side-by-side comparison illustrating a number of the data manipulation features in SPSS and JMP. New JMP users should know that since the data file and the output are dynamically linked, there are occasions where output needs to be closed before the data is manipulated. When sorting or manipulating data, JMP will sometimes default to creating a new data table that has the necessary manipulation; new users should specify that the manipulated file should replace the current data file. Data can be manipulated in JMP through a variety of menu options. The Tables menu item offers many common data table transformations and the Cols>Utilities menu items offer many tools for completing common data column manipulation. The JMP Starter view also shows many of these options in one unified location.

SPSS

JMP

Data Manipulation: Creating New Variables

SPSS

Suppose that we want to create a new variable that is the Age as months (instead of years). We can use the Compute Variable option to define a new variable (call it Age_Std) and attach a formulaic calculation to that variable (Age * 12). SPSS has a number of different functions for aggregating and transforming data, which are all grouped by function type. After filling in the numeric expression, SPSS creates a new variable of the specified name and populates it with the computation from the numeric expression.

JMP

JMP creates new variables directly in the data table. Suppose we want to create the same new variable (Age_Std) in JMP. We start by double-clicking on the unused column at the end of data table to make a new column. Then we double-click on the column name to bring up the Column Info window where we give the column a name, and click on Column Properties and choose Formula to bring up the Edit Formula window. Now, we can edit the formula to create the numeric expression we used in SPSS. Note that JMP also has a wide array of functions that can be applied to manipulate and transform our existing data. Also, we can also create a new column using the menu item Cols>New Columns. Common data transformations can also be done on the fly in many of JMP’s platforms such as Graph Builder, Tabulate, Fit Y by X, and Fit Model. This on-the-fly option is as simple as right-clicking on the variable we would like to transform in the Columns box and going to Transform.

Data Manipulation: File Splitting

SPSS

Suppose we want to look at different output for men versus women aboard the Titanic. We can use the Split File option to tell SPSS that further output should always be stratified by a particular variable. We have the option of creating new files for just men and just women, but the splitting option is a unique way to indicate to SPSS that subsequent output should be separated between women and men.

If we want to return to a non-stratified analysis, simply go to the Split File option and choose “Analyze all cases, do not create groups” to turn the stratification off.

JMP

Splitting a file may not be necessary in JMP since all analysis platforms feature a By option to stratify output by a particular variable. So, if we want stratified descriptive statistics, we use the By statement in the Distribution platform as seen below.

Saving and Reproducing Output

Manipulating data and producing great output and visuals are possible only if there is a mechanism to save and reproduce them. It doesn’t seem obvious but JMP is also fueled by a scripting language, which works mostly behind the scenes for casual or new users. Many advanced users find that coding produces quicker results for repetitive tasks. Let’s discuss how to save and reproduce output in SPSS and JMP.

SPSS

When we save an SPSS data file in SPSS format (.sav file), the data file maintains all manipulations, transformations and labels. In order to retain output, we save as a separate file (.spv format). Since the point-and-click interface for SPSS is really a mechanism for code generation, code precedes all output in the output file. For instance, if we want to look at the descriptive statistics for Age and Fare, the following code could be generated in the output from the point-and-click interface:

JMP

JMP takes a different approach to saving and reproducing output. When we save data as a JMP data file (.jmp file), the data retains manipulations, transformations, labels AND script saved from analysis. Let’s look at an example. Suppose we want to save the distribution analysis of Age and Fare information for later use. In all JMP reports, we can find an option for saving the code (referred to as JMP script) to the data table. There are other options to save JMP script but saving to the JMP data table is one option. When we save the data in JMP format, all scripts saved to the data table are saved as well. We can edit the name of the saved script and reproduce the output with the red triangle located by the saved script’s name in the data table. In addition, we have one file that contains both the data information and analysis related to a specific project. JMP data files can be shared among many JMP users, and one file can be a single platform for all project analysis. If we’d like a written set of instructions similar to the code listed above for SPSS, we simply select Save Script>To Journal and the necessary .jsl (JMP scripting language) code will be displayed in a window of editable text. Like SPSS, JMP’s scripting language is quite complex and can be useful in completing repetitive tasks.

Other Features

Users transitioning from SPSS to JMP may be interested in some of JMP’s unique features.

Machine Learning and Text Mining

While both JMP 16 and SPSS 27 come standard with decision trees and neural networks, JMP 16 has a number of advanced Predictive Modeling methods that SPSS 27 does not, including Bootstrap Forest, Boosted Tree, Naïve Bayes, and Support Vector Machines. Also, JMP 16 has many built-in tools useful for creating a predictive model. One such tool is the Make Validation Column which allows the user to create a column based on stratification, grouping, and cut point columns.

Text mining is another data mining approach is fun and engaging in JMP. SPSS does not have any built-in text mining features. As an example, recall that the Name column is unstructured text. With JMP 16, we can explore this text data for patterns of words or phrases. Under the Analyze menu, the Text Explorer module allows us to select many unstructured text columns and tell JMP how it should tease out the words or phrases within the text selected. Once we click OK, Text Explorer shows common terms and phrases within the columns selected. As we can see from the figure above, the word “mr” appears 513 times; “miss” appears 182 times while “mrs” appears 125 times. This insight is something our Titanic data does not currently account for as a variable. Using JMP, we can easily create indicator columns (0 or 1) for rows that have the words “miss” or “mrs” in their Name column. Just right clicking on the words and selecting Save Indicators creates these new indicator variables in our data set. That is so cool!

Sample Data Sets

One difficulty in learning and teaching statistics is finding nicely formatted example data sets. JMP knows this and has included hundreds of data sets. Categorized by type of analysis, field, and teaching resources, JMP’s built-in datasets are easily accessible via the Help>Sample Data menu item. Opening these data sets is as easy as selecting the type of data or statistical method we want to explore or learn about and clicking the blue highlighted link. Want to learn how to make a map in JMP? Under the Geographic Maps data type, find the US Election 2008 data. Once the data set opens, we find a number of JMP scripts ready to be run in the top left. Clicking the green play arrow next to Graph Builder Map Sized by Number of Electors launches Graph Builder with a map of the United States color-coded with blue or red depending on whether Obama or McCain won that state. Neat! Want to learn about survival analysis? Find the Reliability/Survival category and select Appliance data set. Running the Life Distribution script in the upper left of the data set fits and compares distributions of time-to-event data. It is so easy to learn and explore with JMP!

Conclusion

Both SPSS and JMP are terrific software tools for manipulating, visualizing and analyzing data. In this guide, we have highlighted how to complete many common data-oriented tasks in SPSS and JMP. Throughout, we made sure to point out many of the differences between the two software packages and how JMP may elevate the insights you get from your data. It is easy to see that JMP and SPSS approach data analysis workflows differently. The point-and-click mechanism in SPSS generates SPSS code, so users decide which analysis and options they want to perform and then submit the generated code to obtain the output. The code and results are listed in the output file, which can be saved, copied or manipulated. By contrast, JMP dynamically links the data and reports to create an interaction between the data and the user. The user starts with a general area of analysis and can then customize output to add different features or analytics. The dynamic link between data and output makes exploring unusual observations very simple and intuitive. The interactivity of features such as Graph Builder allows users to create and update visuals in real time.

Acknowledgements

I would like to thank Ruth Hummel, Larry LaRusso, Anne Miley, and Amy Stuart at JMP for all their assistance, edits and feedback during the making of this guide. I would also like to thank Jason Brinkley, the author of the first version of this guide, for providing a clear and precise starting point for creating the second version.

Getting high-quality graphics out of JMP

Ross_Metusalem — Tue, 02 Feb 2021 19:31:43 GMT

Welcome to the second of two posts on creating publication-quality data visualizations in JMP. This post focuses on getting graphs out of JMP as high-quality images. If you’re interested in how to customize your graph’s appearance to get it looking just right, check out the previous post.

As in that post, I’ll work with a customized line graph constructed from the Corn Wheat Soybean Production sample data in the Sample Data Library. This graph was made in Graph Builder, but everything I discuss here applies equally well to graphs produced by JMP’s analysis platforms. Getting graphs out of analysis reports does introduce one special consideration with respect to exporting image files, which I’ll call out when relevant.

Avoid screenshots

Imagine you’ve got your line graph looking just how you like it, and it’s time to put it into a slide deck you’re working on. The first instinct is to take a screenshot, mainly because it’s easy and familiar. This produces an image file (let’s say a PNG file) that you drop into your deck. After resizing the image to fill the slide, this is what you get:

Just say no to low-resolution graphics like this one

Eeesh. We’ve all seen this kind of blurry image fail before and maybe even have perpetrated one ourselves. (I’m certainly not innocent here.) This image is pixelated due to the combination of the screenshot’s image format and the resizing. Screenshots are typically captured as raster graphics, which include the common PNG, JPG and BMP formats. Raster graphics represent images as grids of pixels, and therein lies the problem. When an image is resized to occupy more pixels than the file specifies, the result is a blurry image like the one above.

To avoid pixelation, the graph needs to be captured as a vector graphic. Vector graphics represent images as collections of objects like points, lines and curves instead of as pixels. Because of this, they can be resized infinitely without pixelation; the objects are simply drawn to fill however many pixels they need to. JMP can produce vector graphics in a number of common formats listed below, and vector graphics should be used whenever possible.

Check out this video that compares resizing raster versus vector graphics. The difference is clear. (Bad pun, I know. You’re welcome.)

Getting vector graphics out of JMP

Let’s look at two ways to get vector graphics out of JMP. For a standalone image file, use JMP’s export functionality to export a vector graphic. To put a graph directly into a document or slide, use copy and “paste special” to paste a vector graphic.

Exporting to an image file

Going to File > Export reveals the option to export a stand-alone image file (among other export types) of the current JMP window, with a drop-down menu of the file type options. Vector formats available in JMP include PDF, SVG, EPS and EMF (Windows only).

When exporting a Graph Builder graph, click the Done button before exporting to ensure the element ribbon isn’t included in the image. After that, it’s as simple as going to File > Export and choosing a vector format.

When exporting graphs in analysis reports with multiple sections, File > Export will create an image containing all open sections in the report, along with their section titles. One option to get only the desired graph is to close any unwanted report sections using the little gray triangles and to delete the section titles of any open sections, and then export the window. Take a look at the Oneway Analysis report below. If I’d like to export just the graph at the top, but I leave all the other report sections open, the resulting image includes the entire report:

Exporting a report window will send all open sections to a single imageBy closing the Oneway Anova section before exporting and then double-clicking and deleting the top section title (“Oneway Analysis of Commodity Acres Planted by Commodity”), you get a clean graph like the one below. Of course, you should save your analysis, perhaps as a script in the data table, before deleting that title, since the red triangle menu will also be deleted.

There are multiple ways to export just one graph from an analysis report

Ultimately, it might be preferable not to muck with an analysis report just to export one of its graphs. As an alternative, you can use the selection tool to highlight whatever you want to export, send the selection to a journal, and then export the journal. It’s efficient and foolproof. Here’s a quick demo.

Alongside the image export options under File > Export, there are also options to export to PowerPoint or Word. These options will place all open report sections as vector graphics (and editable tables, for any report tables) directly into a PowerPoint deck or Word document. This feature provides a nice option for getting all report content directly into a slide or document. But if you’re trying to get just one graph out of JMP and into an existing slide or document, copy-paste is a more straightforward way to go. Speaking of which...

Copy and ‘paste special’

To drop a graph directly into a document or slide – no image file needed – you can copy and “paste special” in a vector format. Standard paste options typically use raster graphics, so using the paste special command is a must. Copying a graph and its legend is as simple as right-clicking in the graph area and selecting Edit > Copy Graph. From there, select the paste special option in whatever word processor or slide editor you’re using to view a list of available graphic formats in which to paste the image. Simply select an available vector format and the image will be dropped into your document or slide.

The formats that JMP makes available when copying a graph are set in JMP’s Windows Specific or Mac Specific preferences groups. When copying a graph in JMP, JMP places all the formats specified in those preferences onto the operating system’s clipboard; using paste special tells the application on the receiving end to allow you to select among the formats that are both available on the clipboard and compatible with the receiving application. If a format you’ve chosen in JMP for inclusion on the clipboard is not included in the receiving app’s paste special options’ format list, it’s a safe bet that the format isn’t compatible with the receiving app.

On Mac, PDF is a good option and is always included on the clipboard. On Windows, EMF is a good option; it is enabled by default, but if it doesn’t appear as an option under paste special, make sure that enhanced metafile is checked in the Windows Specific preferences (see below).

Choosing which image formats are included on the clipboard

But what if I have to use raster graphics?

Occasionally, using a vector graphic may not be possible. A journal may accept image submissions only in PNG or JPG formats, for example. There are a couple options here. First, Windows users can set the DPI (dots per inch) for exported graphics. Larger DPI values produce higher-resolution graphics. Mac users might consider another route: export a vector graphic, open it in Preview, and then export the graphic in a raster format while upping the pixels per inch setting as needed. It’s possible to take a similar approach in Windows, albeit with something other than Mac’s Preview application. This approach has the benefit of producing a vector graphic as the “main” graphic file for possible future use, while also producing the high-resolution raster graphic needed at the moment.

One more thing: Transparent backgrounds

Whether exporting or copy-pasting, you may not want JMP’s default cream background color included in your graph. To get a transparent background, go to the Preferences, select the Reports group on the left, and then check the transparent background option at the bottom of the list. Now all the graphs you get out of JMP will not include a filled background surrounding the graph area. Check out the graphs in the next section to see the result.

Setting transparent backgrounds

An added benefit of vector graphics: Infinite customization

As discussed in the previous post, JMP’s graph customization options are extensive. But sometimes there may be a fine-grained change that can’t be achieved in JMP. For example, maybe you’d like to move the “RFS authorized” reference line label a little further down in the graph area but the option isn’t available in JMP. Luckily, vector graphics can be edited outside of JMP using any vector graphics editor. I’ve used the freely available Inkscape to edit my graph, but many other vector graphics editors are out there. (This is not an official Inkscape endorsement.) In the two graphs below, the first one was produced exclusively in JMP and the second graph was produced after further editing in Inkscape. See if you can spot all the differences between them. There are a handful in addition to the position of the “RFS authorized” label.

A customized line graph made in Graph Builder

The same graph as above after further customization with a vector graphics editor

Give up? Or maybe you just never liked playing Spot the Difference. I also moved both axis titles slightly, lightened the box around “27.5m acres,” darkened the left and bottom borders, and rounded off the ends of the colored lines in both the graph and the legend. These differences might seem minute, and that’s the point: extremely fine-grained customization is possible when working with a vector graphic editor, so you can get your graph looking exactly how you like.

If using Windows (sorry, Mac users), you can even edit your vector graphic right inside PowerPoint or Word. It’s as simple as pasting a vector graphic, selecting the Edit Picture and Ungroup options to break out all the individual objects in the graphic, and then using the built-in shape and text editing tools. The video below presents a quick example.

Getting high-quality graphics out of JMP

It’s easy to ensure your JMP graphs look their best in your presentations and publications: set the transparent backgrounds preference; export or copy-paste in vector graphics formats like EMF or PDF; if necessary, fine-tune your graphs outside of JMP as needed with a vector graphics editor. By combining these best practices with JMP’s outstanding graphing capabilities and myriad graph customization options, your graphs are bound to impress.

Thanks for reading. Now get to it.

Hover Labels: Codeless customizations

LisaG — Mon, 08 Feb 2021 15:51:23 GMT

As described in the Git documentation, the term “porcelain” refers to a user interface that is designed to be easy to learn and navigate. This post demonstrates how the high-level commands that implement the porcelain part of the Hover Label extensions feature can support drilling and further exploration of visualizations with just a few clicks (no JSL knowledge required!). The following examples use graphlet presets, Paste Graphlet and platforms with built-in graphlets.

Graphlet presets

With the addition of hover labels’ extended features, such as graphlets, you can drill down into visualizations for further analysis. To add a graphlet in a Hover Label, select the predefined graphlet preset, accessible via the right-click menu under Hover Label. These presets have dynamic graph role assignments, which they derive from the variables used within the graph. Presets are also preconfigured to be recursive to support drilling down.

From the list, there are nine different preset options to choose from to help answer the question of what is behind an aggregated visual element. The Marker preset shows the exact values, whereas the Histogram, Box Plot, and Heat Map presets show a distribution of values. You can also break down and look at a graph with another category with the Bar, Pie, Treemap and Line presets. The Tabulate preset makes it possible to examine a table’s raw data.

For preset graphlets that use categorical columns, select the Next in Hierarchy column property, found in the data table, to specify which columns will be used in the next level of a drill-down.

In the following animated gif, the Distribution graphlet preset is used to look at the distribution of bill lengths per species in the Penguins.jmp data table. After applying the preset and hovering over the bars, the distribution graphlet appears in the hover label. In the top-right corner of the hover label, there is a pin option that keeps the hover label present on the graph. Notice below that there are two pinned graphlet labels side by side, allowing for an easy comparison of the distribution of the data all in one chart.

To take it a step further, click on the graphlet thumbnail, as shown in the animated gif below, to launch the graphlet in its own Graph Builder window and interact with the data visualization behind it. Also, in the launched graphlet window, there is a Local Data Filter (LDF) that facilitates filtering from the base graph; this LDF can be easily updated. And remember, the presets are recursive, so after applying the histogram graphlet preset, you can drill down multiple layers of data simply by hovering over an element in the launched graphlet window.

Paste Graphlet

To construct a graphlet from scratch instead of using a predefined graphlet preset, Paste Graphlet makes it easy to do in three simple steps:

Build both the base graph (left) and a graph that to use as a graphlet (right). NOTE: This is not exclusive to Graph Builder; any platform that supports an LDF can be used.
Using the red triangle menu, save the script of the graph to the clipboard.

Return to the base graph and right-click Hover Label to access the Paste Graphlet option to complete the operation for a graph with a custom-built graphlet.

Unlike graphlet presets, Paste Graphlet will have static role assignments and will not be recursive, since its graphlets need to be drilled down one level at a time. However, by nesting Paste Graphlet operations from the bottom layer up to the top/base graph layer, you can create multiple, custom drill-downs. Continuing with this example, repeat the three steps, save the above bar graph with a pasted scatterplot graphlet to the clipboard and paste it on a new graph. By making a pie chart and pasting the script from the clipboard, there are three layers to the graph:

Built-in graphlets

There are several JMP platforms that have preintegrated graphlets: Function Data Explorer, Process Screening, Multivariate Control Chart, Outlier Analysis of Principal Components, and Process Capabilities. Using Outlier Analysis of Principal Components, below is an example of a graphlet that is automatically generated when hovering over elements:

These examples are just a brief overview of the porcelain features that enable you to do some quick customizations with hover label graphlets in a matter of clicks.

Be sure to check out the next blog post in this series that reviews the plumbing – the low-level interface that allows you to explore rich customization options, plus the full range of hover label extensions.

How to make a wafer map in JMP in under 30 seconds

MikeD_Anderson — Thu, 21 Jan 2021 19:00:44 GMT

The Engineering Mailbag

Episode 6: Wafer Maps in Under 30 seconds

TL;DR

Here’s me walking you through how to do this in under 30 seconds*!! If you want details, read on!

*The walk through takes me about 3 minutes and 40 seconds, but the mouse clicks take about 30 seconds, tops.

Introduction

How do you make a wafer map in JMP?

I get this question…a lot. So, here’s a quick note from the Engineering Mailbag (an Engineering Postcard, maybe?) on how to create a quick and dirty patterned wafer map in Graph Builder.

The Basic Map

For this discussion, I’m going to focus on the patterned wafer map. These are dead simple to create in a pinch and just require, at a minimum, three pieces of information: the X Die ID, the Y Die ID and something to color the map by (sort bin, Vt, film thickness, particle count, you get the idea). We could go crazy with this topic, and I’ll touch on some of that toward the end, but for this discussion, I want to focus on a super basic, “just-get-me-pointed-in-the-right-direction” wafer map. And, for that we will just need those three pieces of data. Do you have that data?

You do!

Sweet. Then, let’s do this.

If you don’t have a data set handy, there is a sample data set included with JMP (which is also the data set I’m going to use) that you can use to follow along until you do. Just go to Help > Sample Data Library in JMP and select Wafer Stacked.jmp.

The patterned wafer map is just a fancy retooling of something called a heat map. And it turns out that a heat map is just a really, really low resolution image where each pixel in the image represents one thing (a device, die, reticle field, etc.). And, as it happens, heat maps (and therefore wafer maps) are really easy to make in Graph Builder! Funny how that works, eh?

First, we need to make sure the data is set up correctly. It works best with a tall table with one die per row. Making the heat map is also a lot easier if you make your X Die and Y Die columns ordinal data. When you pull data into JMP from a factory database or manufacturing tool, the X Die and Y Die columns will usually be continuous. Normally this isn’t a big deal for analysis, but for visualization purposes, they need to be ordinal. So, right click on the data icon and change them from blue triangles (continuous) to green sorted histograms (ordinal).

Now, for the fun part. Open Graph Builder (Graph > Graph Builder in JMP). Drag your X Die column to the Bottom X Axis Drop Zone. Drag your Y Die column to the Left Y Axis Drop Zone. Drag your parameter of interest to the Color Drop Zone on the far right of the window. Last, select the Heat Map Graph Element from the list along the top of the interface. Click Done! You have now created a wafer map.

And here’s a finished wafer map in Graph Builder using the Wafer Stacked.jmp data set (and some tinkering with colors).

If you want to break the map up (by wafers in a lot, process step, etc.), just put the variable you want to map by in the Wrap Drop Zone (to the left of the Color Drop Zone). You can break it up further using the Page Drop Zone (bottom right of the interface) or the X/Y Group Drop Zones. Try them all out and see which ones you like. They all have their advantages and disadvantages. Here’s an example in which I look at lot defect totals using the Wrap Drop Zone (again using some creative coloring):

Refining the Aspect Ratio

Let’s do a little refining. We generally expect wafer maps to be circular, but the aspect ratio of the die can confuse the heat map. This is really an aesthetic consideration as the information is still conveyed effectively even if the wafer looks a little wonky. Think about how many slide decks you’ve seen with distorted screenshots of wafer maps. They may look awful, but they still do the job.

Anyway, there is a trick that we can use to pretty-up the map a bit. Just right-click on the graph and select Graph > Size and Scale > Size to Isometric.

This will usually correct the problem by adjusting things, so the aspect ratio more closely approximates what’s in the data set. However, if you’re not looking at full wafer maps, you might need to manually adjust things using the Frame Size option (which is above Size to Isometric in the menu).

Conclusions and a Shameless Plug

So, that’s really it. Wafer maps are just that simple. Now, if you came here looking to make un-patterned wafer maps (also called bare wafer maps), those are a little harder to create because the data set is sparse and the sampling plan sometimes imparts some bias to the results. But, I did do a paper on how to do bare wafer maps at the 2016 JMP Discovery Summit. You can find that here: All Wafer Maps Are Wrong: An Adventure in Semiconductor Data Visualization. Now, if you want to make the basic heat map look even prettier, or want to quickly make bare wafer maps, or just want to have an add-in do the heavy-lifting for you, there is an add-in called the Semiconductor Tool Kit (STK). It was launched as part of that same Discovery Summit presentation. I’m not going to go too deep into it now, but look for some additional articles on the latest version of the STK later in 2021.

So for now, just remember one thing: Wafer maps are just heat maps – that look like circles.

Hover labels: Beyond details-on-demand

nascif_jmp — Mon, 25 Jan 2021 16:28:18 GMT

What do we mean when we talk about details-on-demand?

Exploring data to find insights usually follows a familiar process. We clean it up, create visualizations, interact with them, do some analysis. Rinse and repeat. We don't usually have to think much about the process, which is a testament to how easy it is to do it in JMP.

But there is an underlying process to exploratory data analysis. It has been the subject of many studies over the years. One of most influential is "The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations" (Shneiderman 1996). This seminal paper introduced the Visual Information Seeking Mantra, or Infovis Mantra for short. It goes like this:

Overview first, zoom and filter, then details-on-demand.

After we create a visualization, more often than not, we have to interact with it to find interesting patterns. Zoom and filter change the visualization focus. Details-on-demand add extra information (details) only when it becomes necessary (on demand). To reduce the impact on the exploratory flow, we use the mouse to point to the area of interest. A brief pause triggers the interaction. And voilà: a hover label pops up showing details, on demand.

How do we decide which extra information is relevant in this context? First, JMP gathers data from Labeled Columns, or row numbers if none is defined. Then it adds information extracted from the underlying visualization. For example, the numeric or categorical values associated with the underlying visual element. These can be extracted from the X and Y axis. The final result is displayed as text in the hover label window. The window can then be pinned to the graph, repositioned, or dismissed by moving the mouse away.

Beyond Text

JMP 15 introduced hover label extensions. These features expand the type and content that hover labels can display.

Textlets allow the display of entire paragraphs of rich text. The content can be static or dynamic, based on the evaluation of JSL variables.
Gridlets provide fine control over the hover label default content, or grid. JMP builds this content from labelled columns and graph roles. The gridlet extension can rename, remove, reformat and restyle grid entries. It can also add new ones, with both static and dynamic content - including clickable links.
Graphlets display graphical or visual content inside the hover label window. The most common usage is to show new visualizations based on the hovered visual element data. For example, a distribution of the values aggregated by the bar in a bar chart. Other possibilities include previewing associated image files and even web content.

Here is a quick example showing the basic hover label details-on-demand interaction, and how it can be extended using graphlet presets.

Hover label interaction and customization with presets

Beyond Details

The Infovis Mantra is comprised of four tasks, details-on-demand being the last one. But Shneiderman's paper describes three more, which the new hover label extensions support.

Relate: The Paste Graphlet creates thumbnails from preconfigured platform graphs. Use it with the Fit X by Y platform to visualize correlations in the current data context. Other platforms can be used to show different types of relationships.
History: The graphlet thumbnails are clickable images. When clicked, they launch the platforms used to generate them in independent windows, or they can launch in place, replacing the current visualization. In either case, an Undo action recovers the original visualization. You can drill-down many levels knowing that you can always go back.
Extract: Visual elements correspond to subsets of the data. The hover label menu gives direct access to these subsets. Also, launched graphlets include a pre-configured Local Data Filter, which allows the associated query to be copied or even modified.

More documentation on hover labels is available at the JMP website.

Beyond This Post

In the next posts, we will cover different aspects of the hover label extensions architecture. The first one talk about the high-level interface that exposes the vast majority of the new functionality. No coding skills required!

In the meantime, check our Discovery 2020 presentation about this same topic: From Details-on-Demand to Wandering Workflows: Getting to Know JMP Hover Label Extensions

References

[Shneiderman 1996] "The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations," by Ben Shneiderman. Department of Computer Science, Human-Computer Interaction Laboratory, and Institute for Systems Research. University of Maryland College Park.

Customizing JMP graphs

Ross_Metusalem — Tue, 02 Feb 2021 19:48:17 GMT

JMP is a powerful data visualization tool, helping you visually explore and understand your data, as well as enabling you to effectively communicate your findings to others. For the most important reports and presentations, JMP graphs need to look great: customized and presented in high resolution.

This is the first of two blog posts on creating publication-quality graphs in JMP. The focus in this post is on customizing a graph’s appearance to meet specific aesthetic or technical needs; the second post covers getting graphics out of JMP in high-quality image formats.

Note that I’m not going to cover the specifics of making graphs in Graph Builder here. That’s a topic worthy of separate treatment and is covered extensively elsewhere (for example, in this Mastering JMP webinar). Instead, I’m going to focus on customizing a graph’s appearance beyond the default settings applied by JMP, whether in Graph Builder or an analysis platform. I won’t cover every possible customization option, of course, but I will highlight some of the most useful ones. And while the example is taken from Graph Builder, the discussion also applies to graphs produced in analysis reports.

JMP’s graphs may be more customizable than you think

Graphs throughout JMP are highly customizable. Each graph is produced according to default settings (a number of which are themselves customizable in the Preferences), but many customization options are available to help you get the graph looking just how you want it.

Consider the line graph below, which was made in Graph Builder from the Corn Wheat Soybean Production sample data in the Sample Data Library. It shows acres planted across time for three U.S. crops, with a reference line and annotation to show the authorization of the Renewable Fuel Standard (RFS) program in 2005. Two years later, there is a 25.7 million acre difference between corn and soy that many have attributed to increased ethanol demand triggered by the RFS program.

A customized line graph made in Graph Builder

This graph has had a number of customizations to get it looking like this. It began as the graph below, produced by Graph Builder’s default settings. Both graphs show the same trends, but the customized version above better communicates the message. For example, the line colors align conceptually to the crops (e.g., yellow for corn, amber for wheat), the reference line indicates the timing of the RFS authorization, and an annotation makes explicit the size of the corn-soy difference in 2007. The customized graph arguably looks a little better aesthetically, thanks to increased line weights, the addition of horizontal grid lines and the placement of the legend inside the graph area. All these customizations – and many more – are possible in JMP.

A line graph made with Graph Builder's default settings

Customizing the graph elements and area

Each of JMP’s graphs contains one or more elements (lines, points, bars, etc.) that populate the graph. In Graph Builder, many element properties are set with the properties areas on the left side of the Graph Builder window. However, these element properties specify what the elements represent (e.g., Should the line represent the mean or the sum?) rather than how the elements should look (e.g., What color should the lines be?). The latter options are available by right-clicking inside the graph and choosing Customize. By doing so, the Customize Graph window is displayed, listing each element and, upon selection, that element’s customization options.

Below I’ve selected the corn crop’s line element and set its color to yellow and width to eight. Note that the ordering of elements in the left list dictates the order in which elements are drawn, determining which element will appear on top of or beneath another when multiple elements overlap. This order can be changed using the up and down arrows above the list.

The Customize Graph window

Element customization options are also available via the graph legend. For example, right-clicking on a line element in the legend brings up options to set color, style, width and transparency. This is a nice way to make a quick change to one element without navigating the full Customize Graph window. The legend is also the place to go for customizing color schemes or gradients, which is particularly useful when setting a color gradient for a heatmap or contour element.

The graph area is the region in which all the individual elements are drawn, and it has its own customization options. You may wish, for example, to change the graph background color from white to gray or to remove the border. In my line graph, I’ve removed the graph area’s top and right borders. Graph area options are found under the Graph submenu available upon right-clicking in the graph area. You will find background and border settings there, as well as some other options, such as marker size settings and scaling options.

TL;DR: A graph’s lines, points, colors schemes, borders, background, and more are all customizable. Right-click inside the graph or on the legend, and you’re bound to find the option you’re looking for.

Customizing legends

In Graph Builder or analysis reports, graph legends can be customized via right-clicking on the legend title and selecting Legend Settings or by double-clicking on the legend itself. In my customized line graph, the legend has been oriented horizontally inside the graph area and the title has been removed. The horizontal orientation was set using the Item Direction setting, and the title was removed by deleting the text in the Title field. Note that the labels of the individual legend entries are editable via double-clicking on the label text, but I haven’t edited them here. This feature is handy if you want to enter something more readable or informative to your audience than the default labels that have populated the legend. Below I’ve double-clicked on “WHEAT” to enable editing of that label.

The Legend Settings window

In Graph Builder, the positioning of the legend relative to the graph area is an option under the red triangle. Options include placing the legend to the right or bottom of the graph area or overlaid inside the right or left side of the graph area; I chose inside right for my graph.

Customizing axes

JMP is quite good at choosing default axis settings when making graphs, but you might not always want to stick with the defaults. In my custom graph, I changed the Y-axis range and the X-axis increments and added horizontal gridlines. Changing axis ranges can be done interactively simply by hovering the pointer over the axis labels until the arrow changes to the grabber, which is shaped like the gloved hand of a certain beloved cartoon rodent (albeit with an extra finger). Once you see the grabber, click and drag to scroll or scale the axis, as in the video below. Note that hovering over the middle of the axis reveals a horizontal grabber that scrolls the axis, while hovering over the ends of the axis reveals a vertical grabber that scales the axis.

My customized graph also contains horizontal grid lines on the Y-axis and a reference line on the X-axis, which are two of many options available by double-clicking on an axis. The Axis Settings window contains options for axis scale and increments, tick marks and grid lines, axis label font and orientation, and reference lines. The Axis Settings window also includes a preview of the axis’ appearance, so you can make sure it looks just how you want before applying the settings. The image below is of my X-axis settings. Note that the Label Position and Label Axis Side options for my reference line are new in JMP 16, coming March 2021.

The Axis Settings window

Finally, what about axis titles? You may have noticed that the default Y-axis title read “Commodity Acres Planted” and was oriented vertically, but my customized Y-axis title reads “Acres Planted” and is oriented horizontally. Axis titles can be edited simply by clicking on the title and entering new text, and the rotation option (as well as font options) are found by right-clicking on the title.

Adding annotations

On my graph, I note that 2007 – two years after RFS authorization – saw a 25.7 million acre difference between corn and soy production. I used the annotation and shape tools to add the text and lines to my graph, respectively. After adding the annotation and line shapes and dragging them into position, right-clicking on each revealed further options, allowing me to fill the text box with white with the Filled option and to change the color of the two connector lines to gray.

Now that my graph is ready…

Everything above is only a taste of all that’s possible with JMP’s graph customization options. Hopefully this has boosted your intuition for what other options may be available for getting graphs looking just right. JMP’s online documentation explains many more customization options; here’s a good place to start if you’re interested in learning more. When in doubt, right-click on the thing you want to customize, and there’s a good chance you’ll find just what you’re looking for. And failing that, the JMP User Community is always ready to help.

Now that my graph is ready, it’s time to get it out of JMP and into a document or slide as a high-resolution image. That’s what the next post is all about. It also returns to the topic of customization at the end, where you can see that if you export the graph in the right format, it’s editable outside of JMP, opening up virtually infinite customization options.

Follow me, dear reader, to the next post.

Create live reports in JMP Clinical 8

john_cromer — Mon, 23 Nov 2020 19:52:27 GMT

JMP Clinical offers multiple ways to share report results, from static reports to an in-product review viewer. New to JMP Clinical 8 is the ability to publish these results to JMP Live.

Findings Bubble Plot in JMP Live.

Patient Profiler in JMP Live.

Since clinical data analysis and review are collaborative efforts, it is important that generating the analysis and sharing the results are both secure and convenient. JMP Live is the ideal platform for users to interactively explore the results of a JMP Clinical analysis from a web browser.

Once your site administrator has prepared a JMP Live instance, publishing results to JMP Live involves little more than a couple of mouse clicks. Let’s see how easy it is.

First, generate or open some JMP Clinical results. Do any of the following:

Run a single report in Review Builder.
Run an entire review in Review Builder.
Open a saved review through Review Viewer.
Run Patient Profiler.

Second, click the Create Live Report button. Note that for Review Builder or Review Viewer, this button can be found at both the report level (to publish a single report) and at the review level (to publish an entire review).

The Create Live Report button can be found on the report and review toolbars in Review Builder.The Create Live Report button can also be found on the Patient Profiler toolbar.

The Create Live Report dialog is surfaced.

The Create Live Report dialog.

Click Publish to JMP Live to allow for maximum interactivity, server-side recomputing power, and sharing. Click Publish to File to save a less interactive copy to your machine alone.

Clicking Publish Data sends a copy of underlying JMP data tables to JMP Live, which is recommended for maximum interactivity.

With Allow data to be downloaded, JMP Live users can download the data tables and scripts.

By checking Warn if exceeds: and specifying a value, JMP Clinical will warn you if the estimated data set size sent for publishing exceeds a size of your choice. This feature helps you avoid unnecessary publish attempts if your JMP Live administrator has specified a maximum package size that is smaller than the sum of data set sizes that you are attempting to publish.

Specify a URL to the JMP Live instance.

You can choose to Share with: Only me (yourself), Everyone at your organization who has access to JMP Live, or only specific Groups of JMP Live users at your organization.

Note that the settings in this dialog are retained for the next time you click Create Live Report.

Click OK to publish to JMP Live.

A notification window appears, indicating if the operation was successful.

Notification window for successful publish to JMP Live.

Your browser automatically takes you to JMP Live, where the published report appears.

Depending on what you published, you will either be taken directly to the analysis or to an index page where you can drill down to individual analysis pages.

A JMP Live index page allowing navigation to individual report analysis pages.

On the analysis pages, data filters, column switchers, hover labels and platform-specific controls allow for interactive exploration of the results.

Adverse Events Distribution in JMP Live.

Click the Details button to view or edit the title, description and thumbnail, to view or download data tables, and write comments.

The Details pane in JMP Live.

As you can see, JMP Live reports are a secure, powerful and interactive complement to the existing options for sharing JMP Clinical results. Be sure to check out interactive examples of JMP Live reports (1, 2) on JMP Public.

JMP Clinical 8: Controlling how graphs and tables look with Study Metadata (Color and Value order management)

john_cromer — Mon, 23 Nov 2020 19:55:32 GMT

The Set Value Order and Color in Studies feature in JMP Clinical enables users to customize the display of CDISC variables and levels collected in study metadata. This customization includes specifying a natural order, alphabetical order, or custom order, according to the nature of the variable and the user’s preferences.

An example of a natural order.

New to JMP Clinical 8 is the ability to assign customized color preferences as well. Users can choose from a variety of predefined color themes to assign color based on position of the level in the list. They also can override one or more level colors with any color of their choice.

An example of one override (gray for Placebo) applied over a color theme.

These study-level preferences can be set at the time the study is added, updated or any time after, from the study management toolbar.

Two ways to access the feature.

These preferences are also carried over to the extent possible, through study data updates. In this example, the initial preference was a gradient theme, and three levels existed in study data. Upon registration of a new snapshot, a fourth level was introduced into the third slot based on the order preference. Its color was updated accordingly to the appropriate third color.

Value order and color preferences automatically applied to a newly introduced level.

Users are notified when the data changes affect the value order and color preferences. For instance, a new value was introduced in the data, so the user may want to revisit the order and colors selected for that variable.

Notification to review value order and color preferences upon data updates.

All these preferences are apparent in the results for the majority of JMP Clinical reports. They allow for a consistent and unified user experience, which enables a more efficient visual interpretation. Here you can see the consistency within a report. Notice that the colors for mild and severe can be reserved even when moderate is missing.

Consistency in level colors within a report.

There is also consistency across reports. Here you can see that the colors chosen for treatment levels are held constant between two reports.

Consistency in level colors across reports.

Dialog

To access this feature, click on its icon in the Study Management toolbar.

Accessing the dialog through the Set Value Order and Color in Studies button on the Study Management toolbar.

A dialog will open containing the preferences for all currently selected studies.

The Set Value Order and Color in Studies dialog.

The dialog is organized by study, followed by clinical domain, variable and level. Outlines allow the user to control vertical space and can be collapsed if necessary. Horizontal space is managed automatically by grabbing the sides of the window.

Within each variable, the levels are exposed. The left side contains the value order controls. Users can sort Natural Ascending, Natural Descending, Alphabetical Ascending, Alphabetical Descending, or by any custom order of their choice.

As order is changed, the legend preview in the middle changes. It reflects both order and color preferences.

On the right side are the color controls. By default, colors are assigned to each level based on its position in the list, according to the color theme.

Users can also override any of these choices with a specific color by clicking on a color box in the legend preview and picking any color from the rainbow.

Overriding a theme color for the NOT APPLICABLE level.

Once an override is made, a plus sign (+) appears to the left of the level.

Example of an overridden color, NOT APPLICABLE, denoted by a plus sign (+) to the left of the level name.

Colors from overrides are protected from any subsequent value order changes.

Example of how overriding a level color prevents that color from being modified after making subsequent value order changes. It remains yellow despite being moved into the last position.

Other color controls override all level colors (which reserves and protects all the current colors from subsequent value order or value color theme changes), swap two or more colors, clear overrides, and reset themes and overrides to factory defaults. Treatment variables have another button to apply the color preferences to all other treatment variables.

Additional color theme and override controls.

At the bottom of the dialog, controls are available to make global changes within all currently selected studies. The first three options affect the outline view in the dialog. The next group is for global order changes. The third group is for global color changes.

Through the “All…” button, additional controls are available for making more global changes to view, order and color on the dialog.

Finally, click OK to save the preference changes, or Cancel to discard them.

Results

Let’s look at one report to see these preferences in action.

The Adverse Events Distribution report reveals the chosen colors as we step through the Adverse Events Stacking columns. Notice that Severity is shown with stoplight colors for Mild, Moderate and Severe.

Adverse Events Distribution report results reflecting the chosen stoplight colors for Severity/Intensity.

The Counts Table reveals the chosen level order as we step through these columns again. Notice “Y” is shown before “N” for Serious Event.

Adverse Events Distribution report results reflecting the chosen value order for Serious Event.

You can also see interactive examples of these preferences in action in JMP Live reports (1, 2) on JMP Public.

That is the Set Value Order and Color in Studies feature in JMP Clinical 8. I hope you enjoy using it.

How to do a test for curvature in a DOE with JMP

MikeD_Anderson — Fri, 19 Mar 2021 18:32:51 GMT

The Engineering Mailbag

Episode 6: Watch out for the curves!

TL;DR: How do you do a simple test for curvature when you put center points in a DOE? Watch the video to find out!

As a new feature - I'm going to be providing a quick video walk through with these posts on the key points from the article. Just in case you need to get to the answer quickly.

Introduction

Every now and again, we systems engineers run into interesting questions that fall somewhat outside the typical range of JMP usage. Other times, the query is just interesting and gives me an opportunity to delve deeper into why we do things the way we do in JMP.

Today's particular question is something that needs to be addressed. It's one I get quite often when talking with people who were trained in classical DOE methods. So, this is an opportunity for me to help out a colleague from Europe and show everyone else something interesting that you might not know how to do in JMP.

The question

Hi, Mike,

I still have a question… If I do centre point runs, I am doing it to check if there is a quadratic term, but I may not be sure which term is quadratic. I should be able to check if there is significant curvature without associating it with any one factor, or am I wrong?

Kevin

My response

Hi, Kevin!

… I did a bunch of research on this curvature test. It turns out that it’s just a t-test! I found the equations in Montgomery’s books. More importantly, it’s really easy to do. It’s not a one-click solution, but I actually like it better, because it forces you to understand the hypothesis. I’ve attached a data table.

Create a column in your data table that indicates where a row is a corner or center point. This could be done using recode if you have a column with the design code in it.
Go to Fit Y by X. Put your Response into Y, Response and the column you created indicating corner or center point into X, Factors.
Click OK.
Under the red triangle, run a t-test (not the pooled one, unless you have replicates). You want the p-value for Prob > |t|. The null hypothesis in this case is that the difference in the means (between center points and corner points) is zero. Montgomery’s explanation is that if the t-test indicates the means are different, then the center points lie on a different plane than the linear elements of the model (there’s curvature).

If you want everything in one report (à la [a software package which shall remain nameless but rhymes with “really drab”]), you just have to include the column that flags the center points in the model as a main effect (see screenshot). When you click Run, the Effects Summary at the top would lead you to a similar conclusion (that differentiating between center-points and corner points isn't significant). Have a look and see if you can confirm on your data set.

Best,

What the heck did I just suggest?

Before I go down the rabbit hole on this topic, let me give a bit of background on this question – because my colleagues and I get it all the time. Basically, the issue stems from classical screening designs (Fractional Factorial, Plackett-Burman, etc.). These designs generally employ a two-level with multiple center point strategy. The upshot of this is that, within the confines of classical design (more on that later), it’s really efficient resourcewise. The downside is that it only lets you look for main effects and some interactions (if you’re lucky). The center points are added as a hedge to tell the experimenter if they need to consider quadratic terms in later rounds of experimentation. And that’s why you need this test for curvature business.

OK, back to my suggestion. I made what might seem like an odd suggestion. I’m proposing that it’s possible to locate curvilinear ($10 word!) behavior in a designed experiment simply by using the lowly t-test. I mean, it seems like we should need something sexier when looking for a sexily named phenomenon like curvilinear behavior. Well, we don’t. And it turns out, this is not my idea. As I mentioned, the first example that I found of the technique is in Douglas Montgomery’s book, “Design and Analysis of Experiments.” This book was required reading in my first stats class in grad school. Good class, great book. Anyway, back to the topic at hand.

I’m not going to go into all the math of why this works, but here’s the basic idea: If the data collected from a system under study has strictly linear relationships, (e.g., you can connect all points in the data set using just straight lines), then the confidence interval for the mean of data set (excluding the center points) should contain the confidence interval of the mean of the center points. Therefore, if we do a t-test on the means of the edge and corner points vs. the global center points, low p-values would be evidence of curvature somewhere in the system and high p-values would indicate insufficient evidence for curvature in the system. That doesn’t mean it’s not there, just that things are too noisy to tell. Now, since that’s a lot to take in, let’s have a look at some examples so you can see what this looks like.

A trivial example

This is probably easiest to visualize in a trivial case with two continuous factors. Note here that I’m simulating the data and am running with a lot more data than you normally would with a two-factor design. This is for illustrative purposes, so just go with me. Also, note that all the examples in this post are on JMP Public, so you can get the data sets there if you want to play with them.

OK, here’s a designed experiment with two factors and three center points.

The center points are colored red so that they can be located later. Now, let’s simulate some data into the design without any quadratic effects. In JMP, we can do this using the simulate response capabilities in the DOE platforms. Now, before we look at that t-tests, let’s have a look at what’s going on in the data itself. We’ll turn to a 3D scatterplot. I’ve added the fitted model with 95% confidence interval to show the plane that Montgomery mentions.

You can see that when there is no curvature present, the data points sit within the region defined by the upper and lower confidence intervals (blue and purple meshes).

Now, let’s do one where I’ve put some curvature into the simulated model. I’m using the same linear and two-factor interaction coefficients and just added in some quadratic effects to the data. First, let’s have a look at the 3D plots:

Notice how the red center points are no longer in the plane defined by the blue edge/corner points? Do you also see how the center points are outside the confidence interval for the surface?

Now if we look at the t-tests for these data sets (I used a column switcher in JMP Public to show both), we can see that the confidence intervals for the center points and edge locations overlap nicely in the case where there is no curvature. The p-values for that t-test are quite high, just like we would expect. Then you can switch to the case where there is curvature and see the differences in the confidence intervals and the p-values.

The t-test indicates that there is strong evidence that the center points are not in the same population as the edge/corner points. However, we know that the data is collected from the same system, so there has to be another explanation – which is that the system is more complex that we thought and has curvilinear behavior.

Something a little more complex

Next, let’s turn up the heat on the problem a bit: five factors. The question here is, does the same logic hold for multiple factors as for two? We can’t really visualize this, so we’re going to have to rely on the p-values from the t-test. But we’re asking the same question: does the confidence interval from mean for the center points overlap with the mean from the edge/corner data points? That’s it. And, the same logic still holds.

In this example, I’ve done basically the same thing as in the first one. Below is the scatterplot matrix for the design space. Red marks are center points, blue ones are edge or corners. Also note that some of the data points are missing from a classic full factorial design. I did that to show that this works even with the JMP custom design methodology.

And if we look at the t-tests for the two simulations, we can see the p-values indicate really clearly when there is curvature present in the data set.

And that would appear to be that. Right?

Well, not really. You see there’s another school of thought around DOEs, and I’m going to argue that it solves this problem more efficiently than just slapping some global center points on a screening design and calling it a day.

What modern DOE says you should do about curvature

This whole issue of curvature in a DOE stems from two problems in classical design. The first is the fact that, at one point, people had to do the calculations for DOEs on an actual pad of paper with an actual pencil and probably an RPN calculator (for true geek cred). The designs were therefore chosen to make these calculations as easy as possible. This imposed the idea of orthogonality into the world of experimental design. Orthogonality makes the math a lot easier for people to wrap our heads around. The second problem was the actual creation of the design. Orthogonal designs are fairly easy to conceptualize and fit nicely with the traditional scientific method. They could be easily codified into experimenters’ reference books.

Both of those problems are now solved. Odds are pretty good that you’re not doing data analysis with a calculator (RPN or otherwise). And, you’re probably not consulting some dusty tome to determine your experimental set up, although you might be using the digital equivalent and not know it. The fact is that, unless you’re in a stats class with a really hardcore prof, you’re going to use software to analyze your design and probably generate it, too. This means that you don’t have to do DOE analysis with a pencil and paper – you can let the computer figure it out. It also means that you can let computer algorithms figure out where experiments should go in the design space, allowing you to create a custom design to your exact specifications. Throwing the old requirements out the window gives you access to an incredibly powerful class of tools called modern experimental designs.

I’m not going to go too far into modern DOEs (there are great books out there on the topic: this one and this one are good places to start) and new articles are coming out all the time in the literature. I am going to focus in a little on one part of modern DOE: the optimal design. The nickel tour version of optimal design is that it allows you to define where you think the curvature is from the beginning. The algorithm then sets up the appropriate experimental conditions to test if it’s there without wasting anything. And that’s important. It’s pretty rare to run into a situation where you don’t have some idea if there’s curvature in the system of study. If you’re working with subject matter experts during the design generation process, they should know that kind of thing. Here’s a snapshot of what that looks like in JMP:

In the model section, I’ve indicated that I think main effects are important and that the interaction is important. The green box around the two squared terms indicates that I think there is curvature in the system in those two factors. The optimal design will then put the center points in the correct places to test for that. Now, if I had five factors, I could choose which ones I want to explore for curvature by including only them in the model terms, like this:

Note all the combinations I’m not including here (interactions and powers); they are all material that I don’t have to use. And when we analyze this design, we’ll still be able to check for curvature as part of the model generation process:

Now, if you want to check for curvature as part of a screening design, you also have a modern method: definitive screening designs. They have center points baked in and are still much more efficient that fractional factorial and other design types. There are some great resources by Brad Jones on the JMP Community. Here are some of my favorites:

Final thoughts

So, that’s it. It’s a great idea to watch out for curves in your designed experiments. And you can easily test for curvature with center points in JMP using a t-test. There’s nothing magical about the analysis. I’d also encourage you to have a look at some of the modern experimental design methods coming out. Until next time, TTFN!

Working with graphics segments and how to create spider charts in JMP

MikeD_Anderson — Wed, 21 Oct 2020 17:44:58 GMT

The Engineering Mailbag

Episode 5: I. DO. NOT. LIKE. SPIDERS!!! (And spider plots can take a hike, too.)

I can't ignore spider plots forever, so here we go...

Every now and again, we systems engineers run into interesting questions that would fall somewhat outside the typical range of JMP usage. These applications are generally clever, often bringing home how using data isn’t just for business or technical problems. Other times, the questions are just unexpected, challenging problems. These “curve balls” (as I like to call them) come in many different forms: coding problems, interesting analyses, ways of visualizing data…you get the idea.

Occasionally, I’ll get questions about problems that I really don’t want to answer. Today's query is in that little pile, and I have avoided this one for years! The subject of this entry in The Mail Bag is generally regarded as a scary, unpleasant plot by the data viz community, so I guess it’s appropriate for a Halloween post.

Since this type of plot shows up regularly for cosmetics and consumer analytics customers in my day-to-day work, I can't ignore it forever. So, I’m taking this as an opportunity to help out a number of users groups in New York and New Jersey. But I have to say…I really do not like spider plots.

The question

Like I said, I’ve received this question more often than I care to mention. So, I’m going to skip the actual question email, you'll just have to trust me that there have been several.

There were also some in-person requests.

And a few posts on the JMP Community.

Yeah.

Below is an example from a peer-reviewed paper that someone gave me as a reference for what they would like to see:

(J. Inst. Brew. 2012; 118: 325–333) Author's note to any data visualization specialist that may be reading this: Please forgive me. I know this is an awful graph.

The response

My general response for these requests historically falls along the lines of what most data visualization experts have said, which is to discourage people from using this plot if at all possible. However, recently, I started thinking about this persistent question a little differently. I’ve since taken a different tack on the problem.

My current philosophy about spider plots is that I’m open to helping people see how to make one if I can also show them better ways to present the same information along the way. As was the case with the other visualization question I answered, for me, the process of making this chart was much more instructive than the actual result.

I. HATE. SPIDERS.

OK, I have (or rather, had) a strong antipathy for spiders for many years. I wouldn’t quite call it a full-blown case of arachnophobia, but for many years I would dispatch anything with eight legs (and occasionally six; you can never be too sure!) with extreme prejudice. I gave no mercy. I gave no quarter. I. Took. No. Prisoners. Anyway, I eventually got over it. And, with the exception of ticks (vampire spiders!), I have made peace with my eight-legged friends. We generally just stay out of each other’s way now. I’d like to think that part of my aversion to spider plots came from my issue with spiders, but it could also be that they’re a royal pain and I instinctively knew that making one would be a bit scary. That’s not to say there aren’t some interesting coding problems involved in constructing them, particularly if you want them to have the interactivity users expect from JMP (just that I knew the code itself was going to be a bit intimidating). Anyway, let’s have a look at the problem.

Scoping the problem

Practicing what I preach, I started with a problem statement:

I want to make a JMP add-in that displays a spider plot and other possible ways of visualizing a data set in a framework that I can add on to later.

The next step is to create a punch list of the different things that I need to do:

Get the dimensions for graphing from the user.
Get a label column from the user for each data subset.
Construct an interactive spider chart for the provided dimensions by each subset.
Construct an interactive radial chart for the provided dimensions by each subset.
Construct an interactive parallel chart for the provided dimensions by each subset.
Construct an interactive table of pairwise correlations for the provided dimensions.

Scoping the problem revealed some good news, specifically, I can offload a lot of this stuff to JMP if I use Application Builder, which will also make it really easy to add new visualizations later if needed. The challenging bits were mostly around the first two charts and making them interactive. JMP’s graphs work in Cartesian coordinate systems (x and y coordinates). Those graphs are, for all intents and purposes, in a polar coordinate system (r and φ coordinates). As a result, I am going to need some code to convert back and forth between Cartesian and polar coordinate systems. Let’s start there.

Coordinate transform functions

Here’s the function that I used for working between the two coordinate systems:

polarToRect = Function( {r, t},
            Eval List( {r Cosine( t ), r Sine( t )} )
);

It’s fairly simple. Just what you’d find from a geometry textbook. It is a pretty nice little bit of code, in that it takes radius (r) and angle (t) and returns a coordinate pair (x,y). BTW, this function is also in an application that I reference later on that’s included in the JMP sample applications.

Because I’m working in Application Builder, just about everything else I’m going to do will be either a function or an expression. Everything about this project is also spectacularly repetitive, so it will make the iterative parts of my code easier to read if I’m just calling functions.

The easy bits

If you have a look at the final app source, you can see that I’m leveraging some existing parts of JMP: Graph Builder for a parallel plot, and Multivariate for the pairwise correlations. They were created, almost exclusively, using JMP generated code or capabilities in the Application Builder. So, we’ll get them out of the way first.

Because of the way that Graph Builder treats columns in this case, and the fact that I have to be able to handle an arbitrary number of variables, it’s easier for me to build the Graph Builder script as an expression and then append it to the main display tree in the correct place. Here’s the code for that:

drawParallel = Function( {pCol},
            {default Local},
           
            // build the list of variables with the format "X( :col1), X( :col2)" as a string
            // also building a list of plot elements
            colStr = "";
            gElemStr = "";
            For( i = 1, i <= N Items( pCol ), i++,
                        If( i == 1,
                                    colStr = colStr || "X(" || Char( pCol[i] ) || ")";
                                    gElemStr = gElemStr || "X(" || Char( i ) || ")";
                        ,
                                    colStr = colStr || ", X(" || Char( pCol[i] ) || ", Position( 1))";
                                    gElemStr = gElemStr || ", X(" || Char( i ) || ")";
                        )
            );
           
            // Get the grouping variable as a string
            xVarStr = Char( gCol );
           
            // Insert the strings into the main string
            Eval(
                        Parse(
                                    Eval Insert(
                                                "parallelVLB << Append(Graph Builder(
                        Size( 766, 256 ),
                        Show Control Panel( 0 ),
                        Show Legend( 0 ),
                        Variables( ^colStr^, Color( ^xVarStr^ ) ),
                        Elements( Parallel( ^gElemStr^, Legend( 3 ) ) ),
                        SendToReport(
                                    Dispatch(
                                                {},
                                                \!"Graph Builder\!",
                                                OutlineBox,
                                                {Set Title( \!"Parallel Plot\!" ), Image Export Display( Normal )}
                                    ),
                                    Dispatch( {}, \!"X title\!", TextEditBox, {Set Text( \!"Variables\!" )} ),
                                    Dispatch( {}, \!"graph title\!", TextEditBox, {Set Text( \!"\!" )} )
                        )
            ))"
                                    )
                        )
            );
);

Notice that all I’m doing is formatting the variables as strings and plugging them into the right spots in the code and then running it. The end result is a fully interactive parallel plot in my app with very little coding on my part. I tried to script up a parallel plot manually; it worked, but just barely. I don’t recommend it. Just use Graph Builder and Append(). I’m also using the Eval Insert() construct. It’s a little easier to read than constructing the string using concatenations.

The multivariate plot is just a simple Parameterization of the Multivariate Platform in JMP. I didn’t even have to write code to get this bit! There’s an Advanced Mastering JMP webinar that shows how to do that with just mouse clicks. Again, I’m just doing these bits this way for convenience and because I really didn’t want to spend a huge amount of time on this part of the app. It’s not being lazy – it’s being an efficient coder. JMP gives you the code, so you might as well save yourself some time.

The radial plot

The radial plot is actually a port (with some modifications) of an example app that’s included in JMP’s samples directory. The big modification is that I’m using Marker Seg() instead of Marker(). And, that’s actually an important point. It’s possible to make Marker() interactive using some mouse capture commands and some logic, but Marker Seg() handles all this automatically. You just have to tell it which data table you need it to monitor.

drawFlies = Function( {nP, pCol, obj},
            {default Local},
           
            // Create a list of angles for the spider plot
            th = ((1 :: nP) - 1);
            th = Shape( th, nP, 1 );
            th = 2 * Pi() * th / nP;
           
            //Get the data from the table as a matrix.
            sel_dat = J( N Row( DataTable1 ), N Items( pCol ), 0 );
            For( i = 1, i <= N Items( pCol ), i++,
                        sel_dat[0, i] = pCol[i] << get as matrix
            );
           
            // now get a matrix with two columns and nvars rows to keep track of the min and max of each variable
            min_max = J( nP, 2, 0 );                   // column1: minimum, column2: maximum. row for each variable chosen
            For( i = 1, i <= nP, i++,
                        min_max[i, 1] = Min( sel_dat[0, i] );
                        min_max[i, 2] = Max( sel_dat[0, i] );
            );
           
            // Create some empty matrices for the converted values
            xMat = [];
            yMat = [];
           
            // Convert the data matrix
            For( i = 1, i <= nR, i++,
                        current = sel_dat[i, 0];
                       
                        // for each variable, subtract the minimum and then divide by the range.
                        adj = J( N Row( current ), 1, 0.0 ) + 1.0 * ((current - min_max[0, 1]`) (min_max[0, 2] - min_max[0, 1])`);
                        adj = adj`;
                        total = Sum( adj );
                        {x, y} = polarToRect( adj, th );
                        x = Sum( x ) / total;
                        y = Sum( y ) / total;
                        xMat = xMat |/ x;
                        yMat = yMat |/ y;
            );
           
            // Draw markers
            annotateRad = Expr(
                        // Matrices and book-keeping variables
                        xCoord = xxx;
                        yCoord = yyy;
                        obj[FrameBox( 1 )] << Append Seg( Marker Seg( xxx, yyy, Row States( DataTable1 ) ) );
            );
           
            // Substitute values into the annotation expression
            Substitute Into( annotateRad, Expr( xxx ), xMat, Expr( yyy ), yMat );
            annotateRad;
);

As you can see, A LOT of the code is dealing with the radial transforms, etc. The interesting bit for this discussion doesn’t happen until you’re almost at the end of the function. (Hint: Look for the Draw Markers comment.) I’m appending a Marker Seg() to the Graph Elements stack. If you right-click on the radial plot in the finished report and select Customize, you can see it in there:

The reason that Marker Seg() is so powerful is that it has that Row States() reference in it. Remember, everything in JMP is linked through the data table via row states. So, by telling JMP which data table the report needs to communicate with, I can sync up all the charts in the report and have all the interactivity I’m used to with JMP. And it's just by going from Markers to Marker Segments (which blew my mind when I got it working).

The spider plot

OK, we’ve got nothing else that I can talk about except that dang spider plot. So, let’s get into it.

Since a spider plot is basically a parallel plot wrapped around central axis, we’re more or less going to have to make a parallel plot – with all the points projected onto the polar coordinate system. And, if that sounds painful, you’re right. It is. It’s also really, really hard for your brain to process radial information. So, that one change – from Cartesian to polar – makes spider charts significantly harder to read. Anyway, I’ll save the discourse on the merits of this graph for another time. On to the code!

The good news is that since I’m really only dealing with one dimension per variable, I have full control over the angular part of the coordinate system. So, there’s some transformation work there, but it’s not that bad. It gets messy when you need to consider that the user might want a wider scale displayed than the data actually covers, e.g., the values are between 3 and 5 but the possible scale is 1 to 7. I was able to handle that by looking for an axis column property and then getting the max and min values from there. Since that’s not a critical point in the narrative, you can see how that was done in the source code I’ve included with the add-in.

The radial spokes and reference lines are just lines drawn using the Line() command. The hardest part of this whole thing was making the webs for each row interactive and getting the legend to work. I made the lines interactive by employing a Line Seg() and a Marker Seg(). The legend was a repurposing of a really old piece of JMP I ran into recently called a Row Legend. Let’s look at each of them.

Here’s the code for creating the lines on the spider chart. As with the radial chart, it takes a lot of data manipulation just to get it into a format that makes sense to graph:

drawTrails = Function( {nP, pCol, obj},
            {default Local},
           
            // Create a list of angles for the spider plot
            th = ((1 :: nP) - 1);
            th = Shape( th, nP, 1 );
            th = 2 * Pi() * th / nP;
           
            //Get the data from the table as a matrix.
            sel_dat = J( N Row( DataTable1 ), N Items( pCol ), 0 );
            For( i = 1, i <= N Items( pCol ), i++,
                        sel_dat[0, i] = pCol[i] << get as matrix
            );
           
            // now get a matrix with two columns and nvars rows to keep track of the min and max of each variable
            min_max = J( nP, 2, 0 );                   // column1: minimum, column2: maximum. row for each variable chosen
            // Check if the user wants to use axis column property values for max and min.
            If( axisProp == 0,
            // Directly calculate the values from the data table. 
                        For( i = 1, i <= nP, i++,
                                    min_max[i, 1] = Min( sel_dat[0, i] );
                                    min_max[i, 2] = Max( sel_dat[0, i] );
                        )
            ,
            // If the user wants to use axis column property values
                        For( i = 1, i <= nP, i++,
                                    // Pull the column name and check if the axis property is defined
                                    col = pCol[i];
                                    axisFlag = Contains( col << Get Properties List, Expr( Axis ) );
                                    //If the axis property is present pull the values from the property
                                    If( axisFlag == 1,
                                                axisPresent = col << Get Property( "Axis" );
                                                min_max[i, 1] = Eval( Extract Expr( axisPresent, Min( Wild() ) ) );
                                                min_max[i, 2] = Eval( Extract Expr( axisPresent, Max( Wild() ) ) );
                                    ,
                                    // if the property is not present, calculate the values directly from the data
                                                min_max[i, 1] = Min( sel_dat[0, i] );
                                                min_max[i, 2] = Max( sel_dat[0, i] );
                                    );
                        )
            );
           
            // Create some empty matices for the converted values
            xMat = [];
            yMat = [];
           
            // Convert the data matrix
            For( i = 1, i <= nR, i++,
                        current = sel_dat[i, 0];
                       
                        // for each variable, subtract the minimum and then divide by the range.
                        adj = J( N Row( current ), 1, 0.0 ) + 1.0 * ((current - min_max[0, 1]`) (min_max[0, 2] - min_max[0, 1])`);
                        adj = adj`;
                        // an error check form missing values in the vector
                        For( k = 1, k <= N Rows( adj ), k++,
                                    If( Is Missing( adj[k] ),
                                                adj[k] = 0
                                    )
                        );
                        {x, y} = polarToRect( adj, th );
                        xMat = xMat |/ x;
                        yMat = yMat |/ y;
            );
           
            // Draw Lines and Markers for spider plot
            annotateSpider = Expr(
                        // coordinate matrices
                        xMat = xxx;
                        yMat = yyy;
                        nParam = nnn;
                        nPoints = N Rows( xMat );
                        nGroups = nPoints / nParam;
                                   
                                    // Reshape the matrices to make them easier to work with
                        xSMat = Shape( xMat, nGroups );
                        ySMat = Shape( yMat, nGroups );
                       
                        // Create an empty string to hold a list (as a string)
                        pathList = "";
                                   
                        // Draw the lines as polygons using the reshaped matrix
                        For( i = 1, i <= nGroups, i++,
                                               
                                    // Get the coordinates for a given path and add the first value to the end to close the path
                                    xCoord = xSMat[i, 0];
                                    xCoord = xCoord || xCoord[1];
                                    yCoord = ySMat[i, 0];
                                    yCoord = yCoord || yCoord[1];
                                   
                                    //
                                    obj[FrameBox( 1 )] << Append Seg( Line Seg( xCoord, yCoord, Row States( DataTable1, {i} ) ) );
                                               
                        );
                                               
                        // Draw the markers
                        For( i = 1, i <= nP, i++,
                                               
                                    // Get the coordinates for a given path
                                    xCoord = xSMat[0, i];
                                    yCoord = ySMat[0, i];
                                   
                                    // Run Markers Seg (need to move and reformat the matrix)
                                    obj[FrameBox( 1 )] << Append Seg( Marker Seg( xCoord, yCoord, Row States( DataTable1 ) ) );
                        );
            );
           
            // Substitute values into the annotation expression
            Substitute Into( annotateSpider, Expr( xxx ), xMat, Expr( yyy ), yMat, Expr( nnn ), nP );
           
            // Run the annotation expression
            annotateSpider;
);

The important bits are the Line and Marker segments toward the end. I had to draw each connecting line individually. There is a Path Seg() that would have made this really easy, but it creates a filled polygon if you try to color it. So, Lines and Markers it was! By using Graphic Segments, it's really easy to hook back to the data table through the Row States operator. Now, because all the graphs are looking at the row states from the main data table, it becomes possible to color them all simultaneously by assigning colors to each Row State in the data table.

When you right-click on some of the graphs in JMP, there will be the Row Legend option. It’s been in JMP for a while and does two things. First, it colors the rows in the data table by column (like the Red Triangle Menu option in the data table does). Second, it creates a little legend next to the visualization. That’s all I had to do to get the colors in sync with one another across four graphs. Here’s the code:

drawLegend = Function( {gVar, obj},
            {default Local},
            // extract the column name with the grouping variable
            colName = gVar[1];
           
            // pass the variables into the expression
            annotateLegend = Expr(
                        obj[FrameBox( 1 )] << Row Legend( ggg, Color( 1 ), Continuous Scale( 0 ) )
            );
           
            // substitute into the expression
            Substitute Into( annotateLegend, Expr( ggg ), colName );
           
            // run the expression
            annotateLegend;
);

Since all my graphs are looking at the same data table for row states, they all automatically inherit the coloring the Row Legend assigns! Super slick.

Wrapping things up

And that’s it. For scripters, the main thing I’d like you to get out of this is the power of the Graphics Segments. A lot of the bits that make JMP so special are wrapped up in what those functions can do. For everyone else, there’s now a spider plot add-in. I’m pretty proud of how it turned out. But, please don’t use it. There are much better ways of visualizing that type of data. ;)

Author's note

No spiders were harmed in the writing of this blog, although some may have taken umbrage at being lumped in with insects (six-legs) or ticks (also arachnids, but they’re vampire spiders, so I’d like to think that even spiders hate them).

Control chart warnings in JMP Live 16

Daniel_Valente — Wed, 09 Sep 2020 18:36:13 GMT

Control charts provide feedback on key variables and show when a process is in, or out of, statistical control. In JMP, scientists and engineers construct these charts using an interactive control chart platform called Control Chart Builder. In JMP 16, when control charts are published to JMP Live and have warnings, interested parties can automatically be notified so they can appropriately triage and act upon issues as they arise – all from the convenience of any device with a web browser.

Statistical process control engineers often are tasked with monitoring hundreds or thousands of processes daily. With JMP, there are tools and techniques that let these engineers separate the trivial many processes that are running fine from the vital few that need appropriate triage, follow-up and action.

JMP Live fits naturally into the workflow of a statistical process control engineer and can now receive notifications of warnings on JMP Live control chart posts.

When a JMP Live control chart post is published, republished or regenerated (for example, after a data update), JMP tells JMP Live about any control chart warnings. JMP Live then stores these warnings and sends out notifications to users who care about them.

JMP Live considers "users who care about them" to be the publisher of the post, as well as all members of any group the post is shared with, if that group has the Enable Warnings option turned on.

There are 15 possible warning types, and you can turn testing for these 15 types on or off by right-clicking a chart in JMP. The warnings that you might care about monitoring vary by industry and process.

Let’s see how this works. I am using the Washers sample data set available from the sample data library. I run the “P Chart: Control Chart Builder” script in the data table. In order to test for warnings and share them with interested parties when I publish to JMP Live, I need to enable the Tests. From the red triangle menu, I click the control panel and then turn on Test 1.

I now have a chart with a couple of warnings. To publish this to JMP Live, I’ll use File -> Publish. On the Configure New Post screen, there are a couple of new options in JMP 16. The first is a checkbox to enable warnings. If the post you are creating is able to send warnings, the option will show up. I’m going to share this with a Quality Engineers group I am part of to make sure that my interested parties will be notified.

Here is the group information for the Quality Engineers group I just published this chart to. As you can see, this group has warnings enabled so that if a chart has any warnings, all interested parties will be notified in their alerts when they log into JMP Live and also by email by default. Both the onscreen notifications and the email notifications can be turned off (independently) by the user. Also, Enable Warnings is not on by default when setting up a group. You must click on the Enable Warnings option from the group menu.

This is the card view of the post I just published. A red line on the top of the post as well as the warning indicates that this control chart indeed has warnings that need to be appropriately dealt with.

From within a group, you can see that the control chart has a warning from a red bar in the card, the warning indicator and also a system-wide notification. Warnings are displayed in the info as well with more information. The Warnings Log gives you more details.

All interested parties will have also received an email with a direct link to the post and information about what tests were performed, which warnings were triggered and a description of the failed subgroups, tests failed and a reminder of what the test means.

While most comments on a post are made by JMP Live users, the JMP Live system itself can also make comments on a post. When the post has warnings enabled, these comments will include a brief note about which columns have control chart warnings.

And just like other workflows in JMP Live, the publishing of these control charts can be automated through scripting and scheduled for regular update as new data comes in. The Control Chart Builder can also be part of a larger process screening workflow, which I describe in an earlier blog post.

The stack mole

bernd_heinen — Tue, 22 Sep 2020 19:45:47 GMT

Even though I’ve worked at JMP for more than 10 years, I’m still learning new things. Most recently, it was a new view of the Ishikawa diagram and the power of recursion. While both have been around for ages, it wasn’t until I had a specific challenge that I cared about them. This new motivation came when I was developing Geocoding 2020, an add-in that uses a REST API for communicating with a server. The data, coming from the server, was in form of a JSON array that could be quite large, depending on the data. One of the hidden gems in JMP (or more specifically, in JSL), is the command Parse JSON(), which translates a JSON array into a JSL list or associative array. It was a pretty good starting point, but still, if I wanted to grab specific parts of the data, I needed to know how to find and address them. Not an easy task if that array is made from several hundreds of items, deeply stacked in lists and arrays. So, I decided to write this helpful little program, stackmole.jsl, to give me an overview of the data structure. And, I wasn’t a respectable JMPer if I didn’t look for a graphical presentation.

Every server call could deliver data with a completely different depth, width and volume. The presence or absence of part of the data depends on the data itself. It’s impossible to know in advance what that data structure looks like. I needed a program that could decide for itself if it should go on looking for data, if it should look for the next list or array, or if it was at the end. Recursion was my answer to that problem. I know recursive definitions in mathematics, but how would that work in a data structure? Surprisingly well!

Data structures and recursion step by step

Recursion means that I apply the same rule or logic to the result that I produced the step before. The definition of a factorial of a whole number is the most widely known example. The factorial of a number is the product of that number with all the smaller numbers down to 1, which means the factorial of 3 is often written as 3! = 3 * 2 * 1 = 6. The Scripting Guide shows the statements for that routine:

myfactorial = Function( {a},
   If( a == 1,
       1,
       a * Recurse( a - 1 )
   )
);

Recurse can only be applied to a function; the command stands in the function itself and calls that function with the new argument. In the example of 3!, the function is called with 3 as the argument: myfactorial(3). Since 3 does not equal 1, the else clause of the IF statement is executed. This means multiplying 3 with the result of myfactorial (2). Again, that does not equal 1 and the else clause now multiplies 2 with the result of myfactorial(1). There is a result now; it is 1.

JMP remembers that the open task before was to multiply 2 with this result (2 * 1 = 1) and the task before that was to multiply 3 with the recursion result, which is 3 * 2 = 6. Since 3 was at the topmost level, the result is 6. The mechanism can be imagined as going down a stair. On every step, the open task is put down. The program goes down as many steps as necessary, until there is no more recursion. On its way back up the stairs, it collects all the tasks and ends up the starting floor with the result.

JSL knows some data collections, (e.g., variables that represent many values). The structures that can be analyzed by this program are lists and associative arrays. Lists are comma separated lists of elements such as:

list = {"a", "b", 1, 2}

Associative arrays are lists of key-value pairs.

assarr = ["a" => "This", "b" => "is", "c" => "a", "d" => "sentence"];

The first element of each pair is the key; the second is the value. Keys need to be unique and are internally sorted alphabetically.

Because of the different concepts, each structure needs different commands to change or read their content, they can be looked up in the Scripting Guide. What output would the stack mole produce from analyzing a simple list?

list = {"a1", "b1", "c1"};
stackmole (l);

The data table is the list of items that the mole found when drilling down into the depth of the structure. The rows are to be interpreted from a child perspective. The whole set is a list, with the first child (on level 2) as the first item in that list. This child (row 2) is the data and its value is “a1”. Data is the endpoint of every journey, which means if there is no other item, the item is zero.

More informative is the Cause and Effect Diagram, which is the Ishikawa diagram in its hierarchical style. At the bottom, there are the data values above them their indices in square brackets. In this little example, the name of the whole set is “list”, so, if you want to get “b1”, it is list [2].

The parallel plot shows that the whole set splits up into three elements, and the next level elements no longer split up, since they contain the data. While it helps to understand small structures, at the end of this article, I’ll show a colorful parallel diagram that doesn’t help anything.

The results for an associative array differ slightly from the list report in that they contain the names of the keys:

sentencearray = ["a" => "This", "b" => "is", "c" => "a", "d" => "sentence"];
stackmole(sentencearray);

The type of the whole set is aa, which is short for associative array. The whole set is called “sentencearray”; if you want to access the word “sentence” in that array, do so with sentencearray [“d”].

This method is straightforward since it uses just one structure with a set of elements. But JMP allows to stack these structures in an arbitrary way and depth. So, you may have lists of associative arrays of associative arrays of lists, … In this example:

array = ["2list1" => {{"4list1_1", "4list1_2", "4list1_3"}, {"4list2_1", "4list2_2"} },
        "2list2" => {{"4list3_1", "4list3_2"}, "4element1" }  ];

“array” is an associative array with two keys named “2list1” and “2list2”. The values of each key are lists, the first list again consisting of two lists, the second list consisting of a list and a data element. The numerals at the beginning of each term indicate the depth of the stack where those elements reside. In the analogy of the staircase, it is the number of steps that you need to go down to reach this element.

In this case, the stack mole delivers the following information:

The picture shows only part of the data table that now has 21 rows. The hierarchy diagram is complete. If you want to get to the data element 4list3_2 (in bold), the path is:

array["2list2"][1][2]

Below is an example of a parallel plot that doesn’t help at all, but still gives an interesting pattern:

The program has many comments that explain the single steps of the analysis and how the recursion is applied.

The crude MapQuest call

bernd_heinen — Tue, 22 Sep 2020 19:36:57 GMT

I used the new technique of REST interfaces to create a new geocoding add-in. At its core is the call to the respective REST interface of MapQuest, a company providing lots of routing services and applications. Each one of these services has its own URL, its own format to specify the request and its own format to return the result. This call is encapsulated in the JMP function that I have named MapQuest_rest.jsl. It is available in the JSL Cookbook. If you want to use it, save it somewhere on your computer and “Include” it in your JSL program (Include(“<path>/filename”)). To apply it, use a statement like:

GeoList = queryrest();

You, or any user of your program needs a vlid API-key from MapQuest.

The file itself contains the definition of the function and at the end, a call of the function with an empty parameter list. An empty parameter list always opens this information window:

This function is very general and transparent about the content that you want to use it for. It is only helpful if you are interested in the offerings of MapQuest and the details of the respective interfaces. You need to build or interpret the JSON structures yourself.

I used this function for two different interface calls, one for routing and one for geocoding. Both are made as functions and both are available in the JSL cookbook. Using them does not require any knowledge of HTTP, REST or JSON, since the standard JSL structures like variables and lists are sufficient. The description of the geocoding function is more straightforward than the routing function. Both can be found in the blogs.

Geocoding 2020 add-in

bernd_heinen — Wed, 30 Sep 2020 14:57:28 GMT

Geocoding has been around for some time. But now, this service is offered via REST interfaces which provide more flexibility and accuracy. I created an add-in that uses the service from Mapquest to find geo-codes for addresses and calculate distances, traveling times or complete routes between pairs of locations. This is the description of its functionality and usage.

I’ve always been fascinated by maps. Before digital maps became available, I bought maps for every vacation, usually in different scales and in much greater detail than I really needed. So, it should come as no surprise that I am also fascinated by JMP’s graph features that allow data to be represented on maps.

What is geocoding?

One of the interesting applications is geocoding, or the ability to locate points on maps. Usually, you have a point of interest and its address. To find its position and plot it on a map, you also need the geocoordinates. As you might expect, Wikipedia offers more details on geocoding.

Why a new geocoding add-in?

In recent years, computer communication technique known as REST has become more widely used. In a 2019 keynote, John Sall talked about the “Secret Features of JMP.” One of these hidden gems in JMP is really buried quite deep: the ability to access the REST APIs (advance programming interfaces) through JSL commands. Since computer communication data are often exchanged using the “JSON” data structure, JMP developers created JSL commands that transfer data between JSL data structures and JSON data with ease. This technique has become increasingly important. In fact, Johns Hopkins University and the Robert Koch Institut offer REST APIs for their data pools for information related to COVID-19; JMP customers use them to access SAP data for quality management; and I use one for geocoding. This interface extends the possibilities of information exchange, since it provides more than the geocoordinates of points of interest. It also allows users to get information about distances, traveling times and even routing. This article is about the usage of the add-in that I loaded into the file exchange. Technical background will be given in separate post, as summarized in the last paragraph.

What are the functions?

You need a data table with address information per observation. The addresses can be used to add three different levels of geo information, where every level contains the information of the previous one.

Geocoordinates

For every observation, the latitude and longitude of its location is added. In addition, you will get the hierarchy of administrative units, country, state, district, postal code, city, street and an assessment of how close that address has been hit, as far as the information available in the database.

Distance and traveling time

For every pair of consecutive rows in your data table (e.g., row 1 –> row 2 / row 2 –> row 3 /….), the distance between the addresses and the required travelling time will be added to the second row. The connections become enumerated in an extra column. The traveling time is calculated based upon the user’s choice of transportation: car, bike or walking.

Routing

An additional table is created that contains all the above information, as well as the geocodes of all points that describe the route between two consecutive rows.

Presuppositions

There are several internet sites that offer mapping and routing services. A good overview is given by the journal ProgrammableWeb. I decided to use MapQuest. The company offers a variety of paid services but also a free API. If you want to use it, you need to register first and get a free API key. Of course, you also need the Geocoding 2020 add-in. It will install itself in the Add-Ins menu in the “Map Shapes” submenu.

Upon the first call, you will be informed that you can get the necessary key from MapQuest and provided a field to paste in the key.

The reminder of the dialog is disabled. After the key is pasted, it is tested to determine if it is a proper MapQuest key; the key must be verified before you can proceed. If you provided a valid key, it is saved in the user’s directory in folder AppData/Roaming/SAS/JMP. If this directory does not exist, it will be created.

For all of the commands in the add-in to work, you must set the JMP language to English and select the “Use JMP language…” option. Unfortunately, that can’t be done via JSL. You must make these selections using your language equivalence to File -> Preferences -> Windows specific. The add-in will verify that the language settings are correct and will alert you if that is not the case.

Let’s go

The menu consists of two parts. The first part is the upper red area, which is built like nearly all JMP platform dialogs; it contains the column list, the role part and the action panel. In the role area, there are two parts, emphasized by blue frames. In your data table, the address information may be stored in one of two formats: either as one column that holds the complete address as a text string, or as separate columns, with each column holding a part of the address like country, city, street. You can provide either of these formats, but they are mutually exclusive. In both cases, your address must contain the country. It can be given as name or as an ISO code, whichever you prefer.

The second part is the lower red frame that shows which options are available. They are organized in groups of radio buttons (the screen shot below shows the defaults). All options add a group of additional columns to your data table; the combination of “Routing” and “Detailed Routes” creates an additional data set with a copy of your data. These points are identified as “Points of Interest.” In addition, there are “Waypoints,” the points that need turns along your route.

An example

To test how it all works, I gathered some address information from the internet. I grew up in Solingen, a small town in the western part of Germany. It was well-known during medieval times for producing high-quality swords and cutlery, which is why you’ll find the German Museum of Cutlery here. Next to the museum is a small but very impressive baroque church. There are also some other famous places nearby:

	Deutsches Klingenmuseum	German Museum of Cutlery
	Gräfrather Klosterkirche	Baroque monastery church
	Das Neandertal	The region where the remains of a branch of our ancestors were first discovered
	Die Wuppertaler Schwebebahn	A 13-km-long suspension railway, the oldest electric elevated railway with hanging cars in the world, inaugurated in 1901
	Die Müngstner Brücke	The highest railway bridge in Germany, built in 1897
	Schloss Burg	A reconstructed castle, originating from the 12^th century

After searching the internet for the addresses, I copied the strings into a JMP data table. None of the addresses included a country name, so I prepended “Deutschland, ” to all the strings. The strings now look like this

Deutschland, Talstraße 300, 40822 Mettmann

Deutschland, Alte Freiheit 24a, 42103 Wuppertal

Deutschland, Klosterhof 4, 42653 Solingen

Deutschland, Klosterhof 3, 42653 Solingen

Deutschland, Müngstener Brückenweg, 42659 Solingen

Deutschland, Schloßplatz 2, 42659 Solingen

I choose the German country name “Deutschland” on purpose. In the dialog, assign this column to the “Single Address” role and go.

As result of the run, these columns are appended to your data set:

Geo-Information

GC#Latitude

GC#Longitude

GC#Country Code

GC#State

GC#District

GC#Postal Code

GC#City

GC#Street

GC#Quality Assessment

GC#Return Message

I think that the column names speak for themselves. MapQuest provides an estimated quality rating for each hit. It is quite elaborate, so I summarized that information into four categories: Exact, Close, Approximate, Far. The rows in your data table get symbols and colors according to this rating. All positions are displayed in a map, and a script that reproduces this map is added to the data table. Depending upon the area over which your points are distributed, the background map is either Street Map or Simple Earth.

In that data table, I split the address string into its single components (it’s easy if you use Cols à Utilities à Text to Column…) and requested the coordinates using the “Separate address fields” roles.

There is no need to fill all the role fields in this section. Country is a minimum requirement. The more information you provide, the better the hit rate. Results from both methods for requesting coordinates are not completely identical. The coordinates themselves may deviate a little bit and for the detailed request, more information (such as district) may be given. The quality assessment may vary as well.

For performance reasons, single address fields are concatenated to one string and single string requests are sent to MapQuest if there are more than 200 addresses in your data table.

If you choose the combination of “Routing” and “Distances/Time only,” you will get the same information back as for geocoding, plus three additional columns. From time to time, you may receive error messages:

Point of Interest	GC#Return Message	GC#Distance (km)	GC#Time (Bicycle)	GC#Legs
Neandertal		.	.	0
Wuppertaler Schwebebahn	Unable to calculate route.	19.1287	:0:00:52:39	1
Deutsches Klingenmuseum	Unable to calculate route.	0	:0:00:00:00	2
Gräfrather Klosterkirche		0	:0:00:00:00	3
Müngstner Brücke		9.92	:0:00:26:26	4
Schloss Burg		5.0163	:0:00:13:55	5

Consecutive rows build one leg of the total route. The target row holds the information of the leg. In the example, the first leg is from Neandertal to Wuppertaler Schwebebahn. The distance for a bicycle route is roughly 19 kilometers, and the estimated cycling time is 52 minutes.

For whatever reasons, cycling routes could not be found to Deutsches Klingenmuseum or Gräfrather Klosterkirche. Therefore, legs two and three have no travel information, while legs four and five provide the respective information again.

This combination of options requests details for a route between two consecutive rows in the data table with distances given in miles:

Your data table will get extended by the columns described above for geocoding. For the routing, an extra table with a copy of each row of your data table is created; between these rows, new lines are inserted, one per significant point along the route. The last segment of the Solingen tour looks like this:

Row 49 holds the next-to-last address in the data table, which is where leg 4 ends. The route leading from the place before to this point was 9.3 miles long and has an estimated travelling time of 15 minutes by car. It is a “Point of Interest.” The next row is the end of the first segment of leg 5 after +/- 300 feet and five seconds of driving time. It is the first waypoint in a series of seven, before the next (and last) point of interest is reached. Maps show the results, and their scripts are in the tables as well.

The data table is part of the add-in and can be opened using the “Help” button. You may use it to start playing around and looking at the pictures of the attractions that I included.

And if you should ever come to this area and visit some of the places, don’t forget to look for a restaurant that serves Bergische Kaffeetafel, a very rich combination of local specialties including bread, sausages, sweet rice, waffles and more. It’s well-known all around the area that I picked as a demonstration, so you should be able to find it at many restaurants.

At the first run

MapQuest requires you to have a personal API key. You get it from their developer’s website for free. The add-in asks for this key and provides a field to paste it into.

The add-in stores the key in the user’s appdata directory for SAS\JMP. Before every run, it tries to read the key and will use it as long as it’s there.

Excluded rows/routing not possible

The add-in assigns symbols, colors and labels to the rows, but it also respects the “excluded” row state, which has specific consequences for the routing application. If a row is excluded or routing to its address is not possible, then this row is ignored and a route to the next address is searched for.

In this situation, routes between Neandertal and Wuppertaler Schwebebahn, between Wuppertaler Schwebebahn and Müngstner Brücke and between Müngstner Brücke and Schloss Burg will be searched.

Similarly, locations will be skipped if routing to them is not possible.

Technical details

The attentive observer may have noticed that the main dialog is contained in a tab. Tabs are useless if they stand alone. This tab reflects the idea that locations for routings may appear in different formats. If there should be requests for alternative ways of providing locations for routing, that tab may get siblings.

I was motivated to write this add-in as a result of my combined interest in maps in general and the REST API technique. To make the latter one more transparent and perhaps even more widely used, I put three files into the JSL cookbook. They are defining functions that can be called by other programs. One is the REST call to MapQuest. I describe content and a description of the interface in a blog post about the crude MapQuest call. The other scripts are envelopes for the REST call. They take JMP lists as inputs and build the JSON structures that are needed for the geocoding or routing call. If there are many addresses to be coded, it packages them into blocks of 50 for better reliability and performance. In my blog posts about the geocoding and the routing subroutines, I describe these pieces of code in greater detail.

As mentioned before, JMP programmers did a great job of providing JSL functions that translate in both directions between JMP data structures and JSON. In JSL terms, there are lists and associative arrays involved. These elements can be stacked. You may have an associative array that contains a list of associative arrays that contains… . A single MapQuest call delivers a rich set of data from which I only needed some of the elements. Sometimes, these were two levels deep in the stack; other times, many levels deep. So, I found it useful to write a little script that iterates through this structure. And I would not be a respectable JMPer if I didn’t look for a visualization of the result. I found the perfect way in the Ishikawa diagram. Programming this reminded me of the power of recursion, which is the best technique to step down a deep stack and return safely. I thought it might help or inspire other programmers as well. I put the script into the cookbook and – naturally – I wrote a post about recurse and draw hierarchical structures.

I used JMP 15 for the development, using a JMP Project. I wouldn’t say that my little project couldn’t have taken place without Projects, but Projects made it a lot easier.

A routing function

bernd_heinen — Tue, 22 Sep 2020 19:56:15 GMT

One of MapQuest’s services is routing. For every pair of addresses provided, MapQuest sends back the latitude and longitude of each location, as well as administrative information such as country and state. It also supplies the locations of all waypoints between the two points and where to change direction to travel from the first address to the second. Summary information such as total distance and estimated traveling time is also provided. If the function is called with just two addresses, it returns the route and/or distance between them. If you provide a list of addresses, then routes will be calculated for each pair of consecutive list elements. Connections will then be given between element 1 and element 2, between element 2 and element 3 and so on. A list of n locations will produce n-1 routes.

Since data tables are the usual way to collect information in JMP, analysis starts with the user assigning roles to the columns of the data table via a user dialog. The Geocoding 2020 add-in works the same way. After the user has completed the dialog, the program collects the relevant data from the data table, translates it into the structure needed by the REST API and then sends it to MapQuest. So, the rough structure of the program is: 1) complete the user dialog, 2) collect data, 3) prepare data for interface, 4) send interface request, 5) interpret results, 6) write JMP data table(s). The third step, data preparation, is defined as a function and is available as MapQuest_route.jsl in the JSL Cookbook. You can embed it in your own geo application or use it to get ideas of how to structure data for REST calls.

Baseline functionality

You, or any user of your program needs a vlid API-key from MapQuest.

JMP data comes in form of data table columns, while REST interfaces most likely use JSON structures. As a result, there needs to be functionality that translates between both formats. The JSL also has structures for data collection, either lists or associative arrays. The purpose of this function is to encapsulate all non-native JSL commands. The necessary data is provided with the typical JSL variables in character, numeric or list formats. The data that is returned from the call comes as JSL lists as well. So, embedding this function in a JSL script doesn’t require any additional knowledge beyond JSL standards.

The interface

JSL functions are called with a statement like:

result = function();

If the function needs or accepts parameters, they are listed in the brackets. This function, GC_route, can be called with empty brackets or with a full list of parameters. Calling it with empty brackets invokes a parameter definition window with an outline for input and output, as shown in the following sections:

Parameters in a function call are always positional parameters. When the function is called with parameters, ALL mandatory parameters need to be supplied, but they may be empty.

In my blog post about my geocoding add-in, I used a few of the local tourist sites around my hometown as examples of how to use the add-in. Let’s take two of those to see how this interface works:

	Alte Freiheit 24a, 42103 Wuppertal, Deutschland	Wuppertaler Schwebebahn, a 13-km- long suspension railway, the oldest electric elevated railway with hanging cars in the world, inaugurated 1901.
	Schloßplatz 2, 42659 Solingen, Deutschland	Schloss Burg, a reconstructed castle, originating from the 12^th century.

First, you need an access key from MapQuest, let’s assume it is 1234ABCD. Also, assume that:

Every address is just one string of text.
You have provided a pair of address strings; no data table is involved.
You want to travel by car.
You’re only interested in distances and travel times.
You want the distances in miles.

One way of using this interface is to take the address strings as they are. In this case, the format indicator (Position 2) needs to be 1 or larger. Then, the function call looks like this:

routes = GC_route ("1234ABCD", "", 1, "fastest", "m", {"Alte Freiheit 24a, 42103 Wuppertal, Deutschland",
 "Schloßplatz 2, 42659 Solingen, Deutschland"}, {}, {}, {}, {}, {}, {});

If the addresses are not in one string, but in separate elements, the format indicator needs to be 0 (zero), meaning your call would look like:

routes = GC_route ("1234ABCD", "", 0, "fastest", "m", {}, {"Deutschland", "Deutschland"}, {},
  {"Wuppertal", "Solingen"}, {42103, 42659}, {"Alte Freiheit 24a", "Schloßplatz 2"}, {})

Of course, you don’t need to put the values in as parameters; you can also supply a variable, e.g., a list variable with the list of country names.

In any case, the result is the same list of data:

{200, "HTTP/1.1 200 OK", "", {}, {""}, {51.256414, 51.137675}, {7.148503, 7.152488}, {"close", "exact"},{"DE", "DE"},
{"North Rhine-Westphalia", "North Rhine-Westphalia"}, {"Wuppertal", "Solingen"}, {"42103", "42659"},
 {"Elberfeld", "Unterburg"}, {"Alte Freiheit", "Schloßplatz 2"}, {., 16.002}, {., 1600}, {}, {}, {0, 1}, [=>], }

Both locations are found, and the distance and traveling time are calculated. Therefore, parameter five is a list with an empty string. At the end, there is the symbol for an empty associative array ([=>]). It is empty because parameter 13, the detail flag, was set to zero. Otherwise, the routing information could be found in this array.

A minor change in the parameter list of the call causes a significant change in the output. That is the change of parameter 13, the “detail” flag. If it is set to 1, the output is extended by an array that holds all the routing information:

{200, "HTTP/1.1 200 OK", "", {}, {""}, {51.20971, 51.21327}, {7.07255, 7.08376},{"exact", "exact"}, {"DE", "DE"},
 {"Nordrhein-Westfalen", "Nordrhein-Westfalen"}, {"", ""}, {"42653", "42653"}, {"Solingen", "Solingen"},
 {"Klosterhof 4", "Lützowstraße 347"}, {., 1.0767}, {., 965}, {}, {}, {0, 1},
 ["gc#city" => {"Solingen", ., ., ., ., ., ., ., ., "Solingen"},
"gc#coordtype" => {"Point of Interest", "Waypoint", "Waypoint", "Waypoint", "Waypoint", "Waypoint", "Waypoint", "Waypoint", "Waypoint", "lPoint of Interest"},
"gc#ctrycd" => {"DE", ., ., ., ., ., ., ., ., "DE"},
"gc#dist" => {., 0.0064, 0.0499, 0.1481, 0.0853, 0.2366, 0.2993, 0.2478, 0.0032, 1.0767},
"gc#district" => {"", ., ., ., ., ., ., ., ., ""},
"gc#lat" => {51.20971, 51.2099, 51.209949, 51.210171, 51.209187, 51.209667, 51.210789, 51.212135,51.213226, 51.21327},
"gc#leg" => {0, 1, 1, 1, 1, 1, 1, 1, 1},
"gc#long" => {7.07255, 7.072152, 7.072176, 7.072798, 7.074032, 7.074655, 7.077532, 7.081177, 7.084009, 7.08376},
"gc#msg" => {"", ., ., ., ., ., ., ., .},
"gc#postalc" => {"42653", ., ., ., ., ., ., ., ., "42653"},
"gc#ql" => {"exact", ., ., ., ., ., ., ., ., "exact"},
"gc#resp" => {0, ., ., ., ., ., ., ., .},
"gc#state" => {"Nordrhein-Westfalen", ., ., ., ., ., ., ., ., "Nordrhein-Westfalen"},
"gc#street" => {"Klosterhof 4", "Klosterhof", ., "Gerberstraße", ., "Gräfrather Heide", .,
    "Gräfrather Heide", "Lützowstraße", "Lützowstraße 347"},
"gc#time" => {., "00:00:05", "00:00:45", "00:02:13", "00:01:17", "00:03:32", "00:04:28", "00:03:42", "00:00:03", "00:16:05"},
"gc#wpidx" => {0, 1, 2, 3, 4, 5, 6, 7, 8, 0}], }

For simplicity’s sake, this example uses a route between two places that are close to one other. The route is composed of eight waypoints between the start and end point. They are enumerated in the last list of the associative array with the key “gc#wpidx”. Some information is only available for the start and end positions, which is why some lists have so many missing values. Every list has the same number of elements so that the whole array can easily be turned into a data table.

Technical details

The function takes a pair of addresses or a list of addresses as input. MapQuest sends back location and geoinformation about the endpoints of each journey, each location’s coordinates, the directions for each point between them, and where a change in direction must take place. For this simple task, MapQuest offers different input formats. The address information can be supplied as one text string with comma separated parts, like “Deutschland, Schloßplatz 2, 42659 Solingen”, which is the address of a famous castle near my hometown. Alternatively, the address can be given as single elements: country = “Deutschland”, Street = “Schloßplatz 2”, city = “Solingen”, postal code = “42659”.

I assume that, in most cases, more than one address will be looked up. So, the function expects to get address information as lists, either a list of strings with complete addresses or several lists with address elements. Each address, regardless of its format, needs to provide its country. And there is no check if the required minimum information is given. All other elements are optional.

If detailed routing is requested, an array with lots of lists is returned. The first element of each list belongs to the first location, subsequent elements to the route, then again one element for the second location and so on. In order to distinguish between the locations that have been input to the request and the routing points that have come back, the gc#coordtype names the original locations “Point of Interest” and the points in between “Waypoints.” The gc#leg list glues together all points that lead to a destination. The first location does not belong to a leg (my definition). For the route to and including the first destination, it is one; for the route to the second destination, it is two, and so on.

If a JSL variable with a reference to a data table is supplied for parameter 2 and detailed routing is requested, then it is assumed that the list of locations is identical to the list of locations in the data table. All content of the referenced data table will be copied and inserted into the array with the detailed routing. This information can only be linked to the points of interest, since all lists have missing values for the waypoints.

If a route can’t be found, an error message is given at the position that corresponds to the point of origin for that leg. Subsequent routes will be calculated, if possible.

If you provide a link to a data table and that data table has excluded rows, then you must supply a list of excluded rows (

excludedrows = As List( dt << Get Excluded Rows )

); as parameter 12. Otherwise, lists will get out of sync. In the message list, these observations are marked as “excluded”. If there are more “excluded” observations than originally in the data table, these observations failed to provide a minimum of location information.

I hope this piece of code can help you with your project or just provide some insight into the application of JSL. I’m open to any questions or suggestions, but since it is part of the Geocoding 2020 add-in, I will only make changes that are consistent with its intended use in that add-in.

A geocoding function

bernd_heinen — Wed, 30 Sep 2020 14:54:20 GMT

I used the new technique of REST interfaces to provide a new geocoding add-in. At its core is the call to the respective REST interface of MapQuest, a company providing lots of routing services and applications. One of MapQuest’s services is geocoding. For every address provided, MapQuest sends back the latitude and longitude of the location, as well as administrative information such as the country and state.

Since data tables are the usual way to collect information in JMP, analysis starts with the user assigning roles to the columns of the data table via a user dialog. The Geocoding 2020 add-in works the same way. After the user has completed the dialog, the program collects the relevant data from the data table, translates it into the structure needed by the REST API and then sends it to MapQuest. So, the rough structure of the program is: 1) complete the user dialog, 2) collect data, 3) prepare data for interface, 4) send interface request, 5) interpret results, 6) write JMP data table(s). The third step, data preparation, is defined as a function and is available as MapQuest_geocode.jsl in the JSL cookbook. You can embed it in your own geo application or use it to get ideas of how to structure data for REST calls.

Baseline functionality

You, or any user of your program needs a valid API-key from MapQuest.

JMP data comes in form of data table columns, while REST interfaces most likely use JSON structures. As a result, there needs to be functionality that translates between both formats. The JSL also has structures for data collections, either lists or associative arrays. The purpose of this function is to encapsulate all non-native JSL commands. The necessary data is provided with the typical JSL variables in character, numeric or list formats. The data that is returned from the call comes as JSL lists as well. So, embedding this function in a JSL script doesn’t require any additional knowledge beyond JSL standards.

The interface

JSL functions are called with a statement like:

result = function();

If the function needs or accepts parameters, they are listed in the brackets. This function, GC_geocode, can be called with empty brackets or with a full list of parameters. Calling it with empty brackets invokes a parameter definition window with an outline for input and output, as shown in these two graphs:

Parameters in a function call are always positional parameters. When the function is called with parameters, ALL mandatory parameters need to be supplied, but they may be empty.

In my blog post about the Geocoding 2020 add-in, I’ve used a few of the local tourist sites around my hometown as examples of how to use the add-in. Let’s take two of those to see how this interface works:

	Alte Freiheit 24a, 42103 Wuppertal, Deutschland	Wuppertaler Schwebebahn, a 13-km-long suspension railway, the oldest electric elevated railway with hanging cars in the world, inaugurated 1901.
	Schloßplatz 2, 42659 Solingen, Deutschland	Schloss Burg, a reconstructed castle, originating from the 12^th century.

First, you need an access key from MapQuest, let’s assume it is 1234ABCD. One way of using this interface is to take the address strings as they are. In this case, the format indicator (Position 2) needs to be 1 or larger. Then, the function call looks like this:

coordinates = GC_geocode ("1234ABCD", 1, {"Alte Freiheit 24a, 42103 Wuppertal, Deutschland",
 "Schloßplatz 2, 42659 Solingen, Deutschland"}, {}, {}, {}, {}, {}, {});

If the addresses are in separate elements rather than in one string, the format indicator needs to be 0 (zero), meaning your call would look like:

coordinates = GC_geocode ("1234ABCD", 0, {}, {"Deutschland", "Deutschland"}, {}, {"Wuppertal", "Solingen"}, {42103, 42659},
 {"Alte Freiheit 24a", "Schloßplatz 2"}, {})

Of course, you don’t need to put the values in as parameters; you can also supply a variable, e.g., a list variable with the list of country names.

In any case the result is the same list of data:

{200, "HTTP/1.1 200 OK", "", {0, 0}, {"", ""}, {51.25603, 51.13768}, {7.14865, 7.15245}, {"exact", "exact"}, {"DE", "DE"},
 {"Nordrhein-Westfalen", "Nordrhein-Westfalen"}, {"", ""}, {"42103", "42659"}, {"Wuppertal", "Solingen"},
 {"Alte Freiheit 24A", "Schloßplatz 2"}, 0}

Since the addresses could be found and tagged, the response code for each address is “0” (Element 4), which is why there are empty texts in element 5, where error messages would appear otherwise. The rest can easily be understood, given the interface declaration above.

Technical details

The basic functionality is simple. We send an address to MapQuest and get latitude and longitude back. For this simple task, MapQuest offers different input formats and process modes. The address information can be supplied as one text string with comma separated parts, like “Deutschland, Schloßplatz 2, 42659 Solingen”, which is the address of a famous castle near my hometown. Alternatively, the address can be given as single elements: country = “Deutschland”, Street = “Schloßplatz 2”, city = “Solingen”, postal code = “42659”.

I assume that, in most cases, more than one address might be looked up. So, the function expects to get address information as lists, either a list of strings with complete addresses or several lists with address elements. MapQuest has two different URLs for the two address formats. The single string interface operates in a batch mode, i.e., one call can resolve multiple addresses. The interface for separate address elements only accepts one address at a time. For performance reasons, the function combines separate elements into one string and runs the batch mode when many addresses need to be located (currently: > 100).

Each address, regardless of the format supplied, needs to provide its country. And there is no check if the required minimum information is given. All other elements are optional. The more precise you describe the address, the better the quality of the result. If the whole address is provided as a single string, the sequence of address elements within that string doesn’t matter.

If you are reading the locations from a data table and that data table has excluded rows, you must provide the list of excluded rows (excludedrows = As List( dt << Get Excluded Rows )) as parameter 12. Otherwise, the lists returned may not be in sync to your data table. In the message list these observations are marked as “excluded.” If there are more “excluded” observations than originally in the data table, the observations failed to provide a minimum of location information.

MapQuest is an American company, and I found consistent results for all the U.S. addresses I tested. With addresses in other regions of the world, that was not always the case. Location information such as state, district or city is often returned in the local language but sometimes the results are in English.

The new thing I'm going to show people when they start scripting: The Enhanced Log

MikeD_Anderson — Thu, 28 Jan 2021 14:30:25 GMT

This new capability has me breaking out my thesaurus, scrambling for new superlatives.So yeah, so much for being done with the series…

Normally, when I write a series like this, I write the whole thing – start to finish – before I start publishing anything. Normally, I don’t want to leave my readers hanging while I write the next entry. Normally, I try to avoid the stress of writing under a deadline. Alas, there is nothing normal about this entry. This entry is being written as the other entries are being published. I normally wouldn’t attempt something like this, but the information that I’m going to share here is not just salient to the discussion, it completely changes how I’m going to approach this topic going forward. As a refresher, here are the links to the other entries:

And now, let’s turn nearly everything I just finished telling you over the past weeks on its ear.

There is a cycle to how JMP releases work. Usually, the Product Management team doesn’t start discussing the new release with the general public until about two or three months before the release date. So, you can imagine my shock when I was given the green light to write about this new capability – a full six months before JMP 16 is slated for release.

As I sat in a meeting where the developers demonstrated this new capability to JMP staff, I shared my excitement about this feature and my eagerness to show it to customers. Imagine my surprise when, later that day, I was asked to write an article about it. So, what is this magical feature that has my heart all aflutter? What has me breaking out my thesaurus, scrambling for new superlatives? What is the new feature that’s going to completely alter how I introduce people to the world of JMP scripting? Well…and, stay with me here…it’s actually the log.

A preview of things to come

JMP 16 will introduce a new log, currently called the Enhanced Log. But, that’s a bit like saying a smartphone is a thing to make phone calls (you do remember they do that, right?). The Enhanced Log tracks a fair bit of what you do when you interact with JMP and (here’s the awesome part) will generate JSL for it with a mouse click! Let that sink in for a second.

This goes way beyond Source scripts and the red triangle Save Script options. They’re not trying to boil the ocean, so some stuff isn’t going to be captured in the first release; data import and data table manipulation are the primary goals for JMP 16. But it covers a lot of ground right out of the gate. Let me show you what this thing is already able to do (with six months of development time to go).

Turning on the Enhanced Log

First things first, turning on the Enhanced Log is as easy as going into the preferences in JMP (under JMP > Preference for MacOS; File > Preferences for Windows) and changing the Log Style drop-down to “Enhanced.” Click OK and you’re off to the races! (note the screenshot below is from an early release and the log now has its own menu item toward the bottom of the list.)

Now open the log (Window > Log for MacOS; View > Log for Windows). You’ll see something that looks a little different than you’re used to. I’ve cleared mine out in the screenshot, just so you can see the different regions. The top part is where actions will be recorded, which I’ll cover in a moment. The bottom part is where the JMP generated JSL will appear.

How that fits into our story

I’ve put everything into a JMP Project, just to keep everything organized. When the Enhanced Log goes into a project, it’s automatically put at the bottom and goes into a side-by-side configuration (Enhanced Log on the left; the JSL output on the right).

Since this isn’t meant to be a blow-by-blow discussion of how I did things, I’m just going to go through a few steps and point out some things. But I’m going to use the same example we used back in my third post. If you’ve been following along, have a look back and compare how easy this is now.

First, let’s get some data into JMP. I opened Big Class Families.jmp. The Enhanced Log area has an entry that shows the action was a user click (under Type), what it did, where the action came from in JMP (under Origin), and what happened. It also has a nice time stamp. Now have a look at the JSL area. There’s an Open(…) command (properly commented!) ready for you to use! No diving through the Scripting Index required!

Let’s continue with our analysis. I made a Distribution with the Height, Weight, Sex, and Age columns. In the Enhanced Log, there is now an entry showing that a user command started with a data table and resulted in a Distribution Platform window. In the JSL area, I’ve got the properly formatted (and commented!!!) entry to reproduce the Distribution platform.

Now, let’s do a Fit Y by X of Weight vs. Height. The same story – there’s an entry in the Enhanced Log and JSL areas.

But, that’s not generally why we open Fit Y by X, is it? We want to do line fitting or hypothesis testing. If we do that, there isn’t an update to the Enhanced Log or any new JSL code.

We appear to have hit a roadblock. But wait! We haven’t, have we? We haven’t followed the JMP Workflow yet! So (following the workflow) the question is, how do we get the Enhanced Log to capture the changes we made to Fit Y by X by fitting a line? As always, the answer is found by PUSHING THE ANSWER BUTTON! For those who missed my first blog series, I call the little red triangle the Answer Button.) Since we’re in a JSL blog series, we’re going to the Save Script To… part of the little red triangle menu, where we find a new entry, To Log.

Clicking that puts a new pair of entries in the Enhanced Log for Fit Y by X with the fitted line options on.

So. Freaking. Cool.

Side note: Did you notice that it’s also handling the glue operators (;)?

Now, the last bit. How do we get that JSL out of the Enhanced Log? Well, I just asked a question, so let’s push the answer button for the Enhanced Log. And, there we find a Save Script > To Script Window option.

And we’re done! I could delete the extra Bivariate call (the first one, without the line) and put the two report windows into a New Window() function, but the point here is that I got 80%, maybe 90% of the way to a completed script without writing a lick of code. IT EVEN COMMENTS FOR YOU!!! (I’m totally geeking out at the moment…hold on a sec…I need to take a breath.)

Here's the finished code, if you want to play with it (Note: I did change the file path so that it would work on other versions of JMP):

// Open Data Table: Big Class Families.jmp
Open( "$SAMPLE_DATA/Big Class Families.jmp" );

// Launch platform: Distribution
Distribution(
            Continuous Distribution( Column( :height ) ),
            Continuous Distribution( Column( :weight ) ),
            Nominal Distribution( Column( :sex ) ),
            Nominal Distribution( Column( :age ) )
);

// Launch platform: Bivariate
Bivariate( Y( :height ), X( :weight ) );
Bivariate( Y( :height ), X( :weight ), Fit Line( {Line Color( {230, 159, 0} )} ) )

Where to go from here…again

Here we are at the end of the blog series…again. I promise this time is for real. So, what’s your homework for this entry? Well, you might be tempted to just sit on your hands and wait for Enhanced Log to arrive. I know it’s tempting, but I recommend that you practice what we’ve gone over in the earlier episodes. It’s important stuff, and you should use what you’ve learned. Good luck and happy coding!

The (other) 5th thing I wish I knew when I started scripting: How to package things up

MikeD_Anderson — Tue, 11 Aug 2020 19:11:15 GMT

You've created a script. Now how can you share what you’ve created?

Welcome back! If you’ve made it this far, you’ve probably got a working script of some sort. Congrats! If you’ve just discovered this series, you might want to put a pin in this article and start from the beginning.

Here are links to the ones you may have missed:

So, you’ve got a nice new bit of code. What are your plans for it? If you’ve gone to the trouble of creating a script, you should probably consider how to package it for others (or yourself) to easily use. In this last entry, I want to give you some ideas about how to share what you’ve created. JMP has several options depending on how you want to share it.

Sharing your work

There are four or five methods (depending on who you talk to) for packaging scripts for others to consume: JSL files, Query Builder files, JMP add-ins, JMP applications, and JMP Live and/or JMP Public reports. For the level of scripting we’re doing here, some of these methods are a bit out of scope (particularly applications; we’ll save those for another series). What I’d like to do here is give you some options and good resources for each method and work through some examples for the ones that will give you the most bang for the buck in the current phase of your scripting journey.

JSL files

This is what people think of when they talk about working in JSL. If you’ve been following along in the series, this is where your script currently lives. The thing is, JSL files aren’t the best way to store or share files. They can be accidentally edited. They require someone to know how to run a script. And, to be honest, for some people, seeing the code is a bit intimidating.

JSL files should probably only be used for the most trivial of tasks or for prototyping methods for a larger project. There are exceptions, of course. If you’re creating a library of functions or something, it’s going to exist as a JSL file. There are also bits of JMP that let you provide JSL directly (for example, the code folding keywords file), but these tend to be the exception rather than the rule. My personal bias is to stop at a JSL file only as a last resort, if I’m collaborating with another coder on something or if the code is entirely for my personal use (e.g., a snippet for testing something).

Query Builder files

JMP has Query Builder support for most file types through ODBC drivers or JMP data tables. If your script involves a data manipulation step, chances are that you could probably offload a bunch of the heavy lifting to the Query Builder file and use the post-query script (more on that in a second) to do the bits that Query Builder can’t handle. Using these files as a packaging method simplifies the workflow and makes the script more robust. They also make the script portable. Plus, you gain the ability to use the built-in filter prompts to make your code even more flexible.

Let’s look at a simple example. I’m going to use the JMP data table Query Builder under Tables > Query Builder rather than the one for ODBC or SAS connections just to keep things simple, but the workflow would be similar for those cases, and there are resources to show how those work.

First, let’s set up a Table Query. I’m not going to write that out. The JMP documentation team did a great job with an example for the SATByYear sample data. Just follow the instructions in this example and stop after you run the query, but keep the Query Builder open. Once you have the data table, create a Graph Builder map:

Open Graph Builder (Graph > Graph Builder).
Drag State to the Map Shape drop zone below the Y-axis.
Drag % Taking (2004) into the Color drop zone.
Go to the red triangle next to Graph Builder and select Redo > Column Switcher.
Select % Taking (2004) as what you want to switch out and click OK.
Select all the continuous variables as what you want to switch between and click OK.

Select Done to close the control panel.
Now, that’s a lot of clicking around for a graph and a very common view for this kind of data. Using our new scripting skills, we can have Query Builder reproduce this graph (and any additional ones we may choose to add later). Here’s how you do it:

Select Save Script > To Clipboard from the Graph Builder.
In the Query Builder select the Post-Query Script tab in the bottom half of the window.
Paste (CTRL+P for PC or CMD+P for MacOS) the script into the Post-Query Script tab.
Save the Query Builder file by clicking Save.
Now, close the data table Query Builder created and Graph Builder.
Click Run Query.

See how that works? By using Query Builder, you got some interactivity for free in the form of the filtering dialog and were able to significantly increase the efficiency of this workflow. If someone had the two data files, you could send the Query Builder file to them and it would produce the same result.

JMP add-ins

If you want to make your code show up in JMP, an add-in is the way to go. Add-ins show up in the Add-in menu in JMP, which will only appear after you’ve installed your first add-in. They are stored in the JMP file system, so the user doesn’t have to remember where a script file is. And they provide a certain element of polish to a completed project. Admittedly, they can be a skosh intimidating at first, but they’re really easy to work with using the Add-in Wizard.

I’m going to create a really simple add-in that uses this code:

// Set up the namespace
Names Default to here(1);

// open the log window and clear it
Open Log();
Clear Log();

// write my message
Write(“Hello World!”);

The code will open the log and print a message. We’re going to call this add-in “My First Add-in.” The first step is to open an Add-in Builder (File > New > New Add-in). After that, it’s just a matter of filling in the form. There are five tabs along the top of the interface. We’re just going to need the first two for our add-in. The first tab (General Info) is more about bookkeeping and getting JMP to know where things are. The second tab actually sets up the bits that are in the JMP GUI.

Under General Info, the Add-in Name and Version fields are pretty self-explanatory. You can limit legacy versions of JMP from running your code with the Minimum JMP Version. You can also limit your add-in to run on MacOS or Windows using the Host. I recommend setting the Minimum Version and Host to match the configuration of the system that the code was written on (unless you have the ability to test on other configurations). It’s easier than trying to make sure your code is bulletproof for all situations, particularly when you’re just starting out. The Add-in ID is a unique folder name that is created in the JMP file system when the add-in is installed. JMP recommends using the “Reverse DNS” naming convention (no idea why). It basically means you create something that looks like a website address and reverse it. For the add-in, we’ll use com.myAddin.MyFirstAddin. Here’s what the first tab should look like:

Under the Menu Items tab, you just need to click on the Add Command button. Then under Details, give the menu item a name. If you want to be fancy, you can give it a tool tip to explain what your add-in does when you hover over it. Then you just copy and paste your JSL into the Action area. If you’re using the code snippet above, your Menu Items tab should look like this:

And that’s it. Just save the file, and you’re done! Because the default action (on the General Info tab) is to install the add-in when you save it, clicking Save will both create a file to send around to others and install it on your own instance of JMP. Once you’ve saved the add-in, you should see an entry like this in your add-in’s menu:

JMP applications

So, you want to make something that looks like it’s a native part of JMP? Application Builder is, in my opinion, the most flexible and efficient method for generating high-quality, custom JSL solutions. It provides you with the ability to control every aspect of the software, leverage existing parts of JMP, and use custom-designed routines. That flexibility and power does come at a cost in the form of a learning curve.

That said, you shouldn’t let it intimidate you. There are lots of resources out there for getting started with Application Builder, and the things that you can do with it are incredible. It’s just going to take some work to get up and running. There’s also a “lite” version of Application Builder built into JMP. You’ll know it as the drag-and-drop tool called Dashboard Builder. Once you’ve got a bit of experience with JSL, taking the Application Builder course from SAS Education would not be a bad way to learn about this very powerful tool. I’m also toying with doing another “Five Things” series on this topic at some point, so keep an eye out for that!

JMP Live and JMP Public

While JMP Live and JMP Public are heavier on the "consume" end of things, you can create and publish dashboards and reports for others to consume using this method. For the most part, you’re just using the bits of JSL that I’ve shown you in this series. Dan Valente did a great blog post on scripting the JMP Live (and JMP Public) publication process, so I’m not going to belabor the point.

Homework

Well, you made it! The last homework assignment is really easy. Package up your code, give it to a colleague, and have them try it out and see if it works. If it doesn’t, look at the error messages and figure out why. If it does, then you’ve made your first JMP add-in! Nice work.

Where to go from here

Like I said earlier, this series was not meant to blow your hair back and dazzle you with my scripting prowess. It was meant to point your nose in the right direction, give you a good map, a pat on the back, and some help to reach out to in case you get stuck. It’s a starting point; there’s still a lot more road ahead of you than there is behind you.

So, from here, the trick is to keep trying and experimenting. Keep taking people’s add-ins and applications apart to see how they work. Keep trying new things and looking for opportunities to make your workflow more efficient with little scripts. The more you do that, the better you’ll get and the more approachable the bigger projects you’re thinking about will seem. Just keep at it.

The 5th thing I wish I knew when I started scripting: How to string things together

MikeD_Anderson — Tue, 04 Aug 2020 15:06:01 GMT

Ready to learn about stringing everything together into an actual script?Welcome back to my series on the basics of scripting in JMP. This is the fifth entry in a series that is structured for you to work along. If you’re just joining us, you’re really going to want to have a look at the earlier entries in the series, because this one’s a doozy. Here are the links to them for you to review:

All right? Still here? Did you look at the earlier episodes? Excellent! Onward to glory!

From the homework a couple of weeks ago, we should all have a bunch of JSL snippets from using Save Script to Script Window or the Scripting Index to get pieces of code that we need. You did do your homework, right? Good! Let’s get into stringing everything together into an actual script. But first, we gotta have a little chat.

One of Mikey's coding pet peeves

Okey-dokey, folks. Let me, just for a moment, get up on my soapbox about something. When we write code, we must always comment it! We can’t make any assumptions about the person who will read our code after us, including ourselves.

Back in college, I took a Visual Basic coding class. The first thing we learned was how to comment code and write pseudocode. Pseudocode, by the way, are comments that you write before writing code to help you map out what you need to do (they are similar to the punch list I had you write in that first post). The instructor called commenting and pseudocode “being a good citizen in the coding community.” We were even graded on how well we commented the code we submitted. It’s that important.

The truth of the matter is that you rarely have complete control over who sees your code. Case in point: In this blog series, I have given you step-by-step instructions on how to reverse-engineer someone else’s add-in. I can guarantee that some (if not most) of the people who wrote those add-ins were not expecting others to peek under the hood at their code. I can also guarantee that you’re not going to find as much commenting in the code to help you figure out what’s going on as you’d hope.

You need to comment your code to make sure that someone who views it later can understand what you’re doing easily. Yes, commenting is a pain. And, yes, it takes extra time. But, no, you can’t just skip over doing it. It will pay dividends in the long run. And I do practice what I preach here. I’ve got add-ins that I wrote five years ago that I can still understand because I did detailed commenting (and I still remember how tedious it was to do!).

So, those of you who already code, make sure you’re commenting your code so that someone else can understand what you’re doing. Those of you who are learning, get into the habit and stay in the habit of heavily commenting your code. Now, let me get down from this soapbox and show you how.

Commenting in JSL

Let’s open up JMP and open that Big Class Families.jmp data table we’ve been using. If you didn’t save the script example from the last session, go ahead and recreate it. You should have something that looks like this:

There are two ways to put comments into your JSL code. The first is to comment out individual lines using a double forward slash ("//"). Anything after the slashes until a carriage return (hitting return on the keyboard) will be ignored. I generally use this method to do my short inline commenting. It’s also useful to turn off single lines of code that you don’t want to run. Let’s have a look at what that means:

The second way you can comment is used in cases where you want to turn off large blocks of code or to generate large blocks of comments and text. This type of comment is called a comment block. It’s done using an opening comment flag made of a forward slash and an asterisk ("/*") and a closing comment flag made of an asterisk and a forward slash ("*/"). Here’s an example of how this might be used:

Note that I’m not discussing commenting style here. That’s actually up to you. You’ll develop your own style as you go along and may even have different commenting styles for different purposes. My inline comments (like the ones in the examples) are completely different from the commenting I do in something like an application. The point is to write the comments and to be internally consistent in a script with how you do it.

The hardest bits in this entire series

Really, this entire series has been leading up to these next items. There are three core things you need to know to get started in JSL. The first is commenting (see the sermon earlier in this article). The second and third are how to name things and how to send messages to them.

I suppose it’s possible to code up your own ANOVA algorithm (a colleague of mine tells a very entertaining story about that). But, it’s a lot easier to tell a data table to run an ANOVA using Fit Y by X through a message, like we learned in the previous episode. That basic workflow (name something, send a message to it) is a lot of what JSL is about. Yes, you can get into logic and loops, GUI development, etc., but brute force scripting really only requires names, messages, and a little searching the Scripting Index.

Naming things

Let’s start with naming things. You name things by giving JMP a name following it with an equal sign and then writing the text, number, code, etc., you want that name to represent. The name is called a variable. You can think of variables as containers that you can put stuff into or as nicknames for open reports in JMP.

Let’s look at an example using an ANOVA. To create the code for an ANOVA, first run the analysis in Fit Y by X (under Analyze), then turn on the ANOVA test, and then save the code out to a script window. Here’s the report for that using Big Class Families.jmp and the resultant JSL code saved from the red triangle menu (height as Y, sex as X, and selecting Means/Anova/Pooled t under the red triangle) :

This little JSL snippet will produce an ANOVA for any data table that has the same column names as the original data table. (We’re not going to worry about parameterization in this series, though. It’s a bit out of scope.)

Now, what if you’ve got a bunch of data tables open and you want to control which data table you want to run that ANOVA on? Well, you can edit the script by giving a name to the data table you want to run the ANOVA on and then sending the JMP-generated code as a message. If you were to look in the Scripting Index (see the article on getting help), you would find a couple of ways to do this. You could either call a data table directly or grab a reference to the data table JMP is currently focused on by using Current Data Table(). There is a third option that’s a little more advanced, where you give a data table a name as you open it. I’ve put examples of all three below (and again note the use of comments to convey information!):

Calling the data table by name will work as long as there is a data table open that has the name you are trying to use. This can be handy if you have a list of data table names to work from. It can also be handy if you’ve hidden or minimized your data table. In either case, you can still send messages to the data table even if you can’t necessarily see it. This method uses the form:

<variable name> = Data Table(“<data table name>”);

Probably the most useful way to assign a variable to a data table for our purposes is to use the Current Data Table () function. This function grabs whatever is the active or current data table and assigns it to the variable that you’ve indicated. Once the table is assigned to the variable, it doesn’t matter if another data table becomes “current” (e.g., by creating a summary table, you’ll always have a reference to the first data table until you redefine the variable). It has a simpler form:

<variable name> = Current Data Table();

The last method is useful if you want to assume that the data table may not be open when someone runs the script. By using this method, JMP will generate a file dialog that allows the user to select the file they want to open. When the user confirms their choice, the file will be opened and assigned the variable name you’ve given it. Again, it’s a simple form:

<variable name> = Open();

Using the Open(…); method can also be handy if you’re comfortable working file paths and directories. You can put the file path inside the parentheses to tell JMP where to look for the file, like this:

Also, note that I’m using something called a Path Variable in the Open(…); command to shorten the string. There’s a section on them in the Scripting Guide.

Sending messages

Now, once you’ve named something, it’s possible to send it a message. It works similarly to sending a text message to someone on your phone. When you send a text, the first thing you need to know is their name, email address or phone number of the recipient, something to identify them. You enter that into the messaging app and then you send the message. Here’s how to do it using the Big Class ANOVA. I’m going to have some fun and have JMP ask me to open a file in this case:

Let’s unpack what we’re seeing here starting at line 5. The first thing that we see is the variable name of the data table we want to give an instruction. Just like when we send a text or give an instruction to a person, we have to identify who we are talking to. Next there are two “less than” symbols (<<). We’ve already seen something like this with the Glue Operator (;). The “<<” symbol is referred to as the Message Operator. It lets the message recipient know that what follows is a message. After that, we see a function called Oneway(…); with the arguments that the function needs to create the ANOVA report. Note: you can get all the information on the options for that function in the documentation or Scripting Index, but you don’t actually have to write any of it out longhand. Simply run the analysis through once and grab the finished script (with all the options) using one of the Save Script to... commands I showed you earlier.

Some odds and ends

Okay, there are three last details that need to be addressed. The first is the issue of stringing things together. For instance, if you wanted to do an ANOVA and create a Graph Builder in the same script, you’d need to warn JMP that further instructions are to follow. To do this, use that Glue Operator (;) that I keep mentioning.

Like a semicolon in written English, the semicolon at the end of a command lets JMP know that there are more instructions to come. Glue is found at the end of an instruction like Bivariate(...);, or Distribution(...); , or giving something a name: stuff = 5 + 6;. As you may recall from earlier posts, JMP includes the semicolon automatically when we use Save Script…To Script Window, but when scripting freehand, this happens to be the most common error I commit during scripting – forgetting to glue things together.

The second detail is ensuring your scripts are good citizens in a JMP session. This is done by assigning all the variables you create to a namespace. A namespace is a little sandbox that keeps the variable names that you create from interfering with other parts of the software. The default namespace is called the “Here” namespace. In scripts it looks like this (see line 2):

This instruction should always go at the beginning of a script (until you learn when not to use it!) to keep your variable names from colliding with other scripts or parts of JMP. Before you start pulling out your hair on this point, here's a TL;DR version of what I just said: Just remember to always put "Names Default To Here(1);" at the beginning of your script and you'll be fine for the moment. If you want to learn more about Namespaces, there's a paragraph on the topic in this other post I wrote a while back.

The last detail I need to discuss is how to change a column name in a saved script. It’s really simple. Look in your saved script. You should see your column names with a colon (“:”) in front of them. That colon indicates that they are column names (it can mean other things, but for now, it means column names). To change a column name, just change it, but remember to put the colon in front. If your column names contain special characters, you might run into something like :Name(“…”) when JMP outputs the script for you. That’s how JMP handles the fact that your column name has characters that would cause problems if it wasn’t clear that they were part of a string. Even though it’s bad coding form to use :Name(“…”) in our scripts, the software gets to use it because it doesn’t have a choice. You should try to choose better column names.

But you forgot about...

OK, I realize some of you are probably getting a little twitchy at this point. You may be thinking, “YOU FORGOT ABOUT FUNCTIONS!!!” or “BUT WHAT ABOUT EXPRESSIONS!!!” or even, “YOU TWIT!!! YOU COMPLETELY GLOSSED OVER LOGIC AND LOOPING!” First off, it’s not nice to call names. Second, I didn’t forget about any of that. I consciously omitted it.

The thing is, this series is not meant to blow your hair back. Like I said toward the beginning of the series, it’s meant to point your nose down the right path, give you a good map, some snacks, a pat on the back, and show you where to look for help in case you get stuck. It’s a starting point. (You have my permission to get a snack at this point, btw.)

For those of us who script regularly, a lot of what I’m talking about here is so foundational – so basic – that it’s muscle memory. The thing we’ve forgotten is that, just by using the foundational knowledge I’m covering here, a neophyte scripter can cover a lot of ground. Anyway, if you’re hyperventilating because I didn’t cover anything super fancy, relax. We’re going to go way down the rabbit hole in the next series. For now, let’s get back to the discussion and talk about this week’s homework.

Homework

All right – here we are. The big homework exercise. This week, you’re going to take the bits of code that you made last week and glue them together. Do this one bit at a time. If your script is going to live outside a data table (as a JSL file), then start with opening your data table and putting it in a variable. After that, regardless of where your script is going to live, add a bit and check that it does what you expect. Then add more and check again that you get the expected result. Remember the Community, help files, etc., are your friends. If you get error messages, look at the log (View > Log for PC and Window > Log for MacOS) and the previous episode.

Good luck and see you next week!

The 4th thing I wish I knew when I started scripting: What to do when things go wrong

MikeD_Anderson — Wed, 30 Sep 2020 15:51:11 GMT

What should you do when things go awry with your scripting?Welcome back!

If you’re just joining us, you’ve jumped in right in the middle of a series that aims to get people started with using JMP scripting language. Unlike the previous entries in the series, you probably don’t need to read the others for this one to make sense, but if you want to play along and do the homework from the other entries, here are the links

Also, a big thanks to Christian Stopp () for suggesting this entry in the series.

This part of the series is very much a standalone entry. But, in the flow of things, this is probably the point where you’ve started to play with some code and may have seen an error message or two. So, before we start with any serious code wrangling, let’s cover what do to when things go sideways.

The most common mistakes in JSL

Justin Chilton () and Wendy Murphrey () are among the best scripters at JMP. At the 2020 Americas JMP Discovery Summit, they (virtually) presented a paper titled, “The Most Common JSL Mistakes and How to Avoid Them.” The PowerPoint file for that presentation can be found here on the Community.

For the new scripter, this is 14 pages of solid gold. In the context of what we’re doing in this series, some of it is a bit out of scope but will be invaluable for the next steps in your scripting journey. Just do me a solid, download the pdf, put it somewhere that you won’t forget about it, and give the paper a thumbs up on the Community page.

Mike’s most common JSL mistakes

As a bit of a riff on Justin and Wendy’s work, I want to add a couple of things to their list. These are pretty basic and will generally cause a script to crash before it’s even run (which is why they don’t show up in the Discovery Summit paper).

Glue

Missing Glue Operators are probably the most common JSL mistake out there. Like I mentioned before, the Glue Operator is the semicolon (“;”) that you see at the end of every instruction in JSL. If you’re saving scripts to a script window using the red triangle menu (see the previous post), JMP will add the glue where it needs to go. If you’re writing code manually, there’s a trick that you can use to make sure you haven’t forgotten one somewhere. Right-click in the Script Editor and select Reformat Script.

This trick will actually catch a lot of basic syntax related errors in the script. There is also a keyboard shortcut when you are in the Script Editor (it doesn’t work in “script editor boxes” though). I prefer the keyboard shortcut myself. For Windows, it’s CTRL+M. For MacOS, it’s CMD+M.

Now, before I show you how to unpack what Reformat Script tells you when there is an error, let me show the other error that drives me nuts. Since they both use a similar debugging workflow, this will make sense in a second, I promise.

Parentheses

Getting your parentheses (brackets, braces, etc.) out of place (or missing) is probably the most insidious bug to track down. And I do mean it is a special kind of frustrating. There is one tool that you have available in JMP to help you with tracking down mismatched pairs of parentheses. When you put your cursor on the outside of a parenthesis, the Script Editor will highlight it (and what it thinks its match is) in blue.

If the Script Editor is confused, or can’t find a match, it will highlight the parenthesis in red.

So, if the blue match for a parenthesis is not where you expect it to be, or if the parenthesis is highlighted in red, you probably have a problem.

When you get errors for missing parentheses, you might be tempted to just keep adding them to the end until the error goes away. Let me just say that’s a bad idea and move on to some better strategies for this issue and for the issue of missing glue. And they both revolve around setting up JMP’s Script Editor.

Setting up (and using) the Script Editor for success

The JMP Script Editor (we saw it in the previous post) has a number of features that make working with code and debugging easier. I have these on by default in my preferences, and I’d suggest you do the same. To access the JMP Preferences Dialog, select Preferences from either the JMP menu (MacOS) or File menu (PC).

I’ve highlighted the options that I find really useful in the screenshot. The code folding comes in handy for larger scripts. The line numbers and autocomplete options are more important to our current discussion.

How to use the log

Now that we have our Script Editor set up, let’s learn how to use the log. First, to open the log, go to View > Log (Window > Log on a Mac). This will open a window that looks similar to a Script Editor. It is different from the Script Editor in that it shows the output and results of the commands run in the Script Editor. If you would prefer to keep your input and output in the same window, you can turn on the Embedded Log in the Script Editor by right-clicking in the Script Editor and selecting Show Embedded Log.

Let’s have a look at how it works using this script:

The script is pretty simple. Aside from line 2 and the comments, it was created using Save Script…To Script Window. If I run Reformat Script (also found by right-clicking in the Script Window), I get the following error message:

In the log, you get something similar:

In the gray area, you can see what was input. Below it, you see a message about something unexpected and a suggestion about what could be the problem. In this case, JMP thinks there is missing glue (;). It also tells you where the unexpected input came in the script with a line number. Since JSL is run sequentially (from top to bottom), this means that the error is somewhere above line 5. Because we’ve got the line numbers turned on, we can see that line 2 is missing the Glue Operator, which means JMP isn’t expecting a new command to start at line 5 and is caught by surprise. After adding the semicolon at the end of line 2, running Reformat Script shouldn’t return any errors.

Similarly, if we’re missing part of a bracket pair (e.g., parentheses), like this (notice line 7):

we would get this error message:

with a log message:

These messages from JMP are a little less clear-cut, but here’s how to interpret them. First, from the error message, we know that the problem is in the Distribution. Between the log and the error message, it’s most likely that there is a missing parenthesis somewhere. The log says that the something unexpected happened at line 11 (the “Column …” can sometimes be helpful, but you have to count characters to use it). The color highlighting the parentheses shows that the first one is missing its pair (note the red color on the parenthesis next to Distribution in line 5 after placing the cursor to its left).

Work forward using the highlighting to figure out where the missing one should go (line 7 in this case). For the most part, the log is a really useful window into what JMP thinks is going on. However, if you want to check your JSL for errors or work through the code stepwise, there’s another tool for that.

How to use the JSL Debugger

The JSL Debugger is a bit of an odd beast. If you’ve accidentally hit the debugger button:

you probably already know that it locks up the main interface in JMP while running. When I started scripting, I really didn’t use it a lot. Now that I’ve had a few chats with the development team about it, I can say that I’m coming around to it. Let’s talk a bit about it and what it can do.

First, that bit about it locking up JMP. It’s a little confusing, but what it’s actually doing is creating a second instance of JMP to run the debugging. The main instance of JMP isn’t going to be responsive as long as the JSL Debugger is active. If you close the JSL Debugger, you’ll get control back.

When you launch the JSL Debugger, in addition to another instance of JMP, you get the JSL Debugger window. This window contains your code, lists of the different variables, a log, and other information. It looks something like this:

When you’re debugging, this window is your eyes and ears into what your script is doing in JMP. If you’ve got a second screen, move it over there, so you can always see it (sometimes it can get buried behind other windows).

Now, I could go crazy with describing everything that you could do with the JSL Debugger, but the documentation has a great chapter (with examples!) that you can find here.

There is also a good blog post by Melanie Drake () that covers some of the cool things that you can do with the JSL Debugger. So, if you want to go into the weeds a bit, have a look at those. For our purposes, I want you to pay attention to these buttons:

The first two run the script. The first one runs the script to its end or to where it encounters a breakpoint (a point that you want to stop evaluation if you don’t want to run the entire script). You can set a breakpoint in your code by right-clicking next to the line number in the JSL Debugger. A red dot will appear next to that line (the breakpoint needs to be on a line of code, not a comment!).

If you were to run this script using the first Run button, it would run until line 10 and then stop. A second press of the Run button would then finish the script. The second Run button runs the entire script without paying attention to breakpoints. The red Stop Button exits the debugger, which is useful if you opened it by accident or have finished your debugging session. The Repeat button resets everything to run the script again (e.g., if you want to change where a breakpoint is and then rerun the code).

Homework

The homework this week is pretty light. I want you to get the Script Editor set up, find the log, and start playing with the JSL Debugger using some of the example scripts from the documentation. Have a look at which variables pop up in the tabs on the lower left of the debugger window. Also, find the Log tab in the lower right (so you know where it is).

Next week, we’re going to get into the last bit of the “coding” we’re going to cover in this series.

The 3rd thing I wish I knew when I started scripting: The Answer Button is your friend

MikeD_Anderson — Tue, 04 Aug 2020 14:50:40 GMT

Make the Answer Button (the red triangle) your friend.Welcome back to my series on the basics of scripting in JMP. This is the third entry in a series that is structured for you to work along. If you’re just joining us, you’ll probably want to have a look at the earlier entries in the series. Here are some links to those:

Have a look; it will help you with what’s coming up.

All right, buckle up. Thus far we haven’t actually worked with any code (unless you count the JSL Scripting Index). That ends today. Today we are going to use the JSL script window. I know that can be anxiety-inducing to some of you. Let’s take some deep breaths and get to it.

Don't script for the sake of scripting!

Let’s be honest about something: Coding is not an efficient process in the short term. For one-off analytics and graphics, drag-and-drop tools are going to almost always beat out coding in terms of ROI (the cost of someone going through a development exercise is just too great). There are those who insist on scripting regardless of whether a GUI solution is faster. I don’t know why, and I am not in their number. I will consider coding when a workflow is difficult or circuitous enough that using a GUI introduces a significant chance for errors down the road. I will also consider coding when I’m going to do something over and over again and I can justify the time through the efficiency I gain by automation. That’s not to say I don’t code just to see if I can do something. Or that I don’t choose a more convoluted way of coding just to learn something new. But those cases are the exception rather than the rule.

Now, I’ve probably tweaked a few people’s noses with that generalization. The problem with this kind of blanket statement is that it’s pretty black and white. The truth, just like everything in life, is that there’s a gray area where the value proposition is a toss-up between a point-and-click solution and coding. The gray area also happens to be where most aspiring coders find themselves. They know they can probably gain some efficiency through scripting, but they just don’t know where to start or they are worried that the time lost to the initial learning effort will be too great. What’s nice about the design of JMP is that it gives us tools that we can use to start introducing JSL into a workflow without losing too much efficiency when manipulating JSL. And, like all things in JMP, it starts with The JMP Workflow.

The JMP Workflow (scripting edition)

The JMP Workflow generally ends with a user going to a red triangle menu to answer a question. Another function of the red triangle menu is to provide methods for users to save their work (save scripts).

There are three main ways to save scripts:

To Data Table: This method saves the scripts to the data table as "Table Scripts." These scripts save the method used to generate a given report in JMP and are portable (i.e., can be copied and pasted between data tables with the same column names).
To Script Window: This method saves the information needed to reproduce a given report to a script window. It is useful for multiple-step operations or producing multiple reports. The contents of a script window can be turned into a table script as well.
To Clipboard: This method is useful for moving the code needed for a report into other JMP Platform (e.g., Query Builder, Application Builder, etc.) or for generating prototype code for larger projects.

It turns out that really efficient coders will leverage these red triangle options extensively during their coding sessions. They use them to speed up their coding (why pound keys when you don’t need to?) or generate platform prototype code that would otherwise be easy to mess up and difficult to debug. For the beginning coders, these options provide different opportunities to learn about and utilize JSL while easing into the details of working with it. This is where we want to start our coding adventure.

Save Script > To Data Table

If you happen to be a JMP user in the NY or NJ area: Hi! I’ve probably visited your company at some point. (If not, drop me a PM; we need to fix that.) And, when I’ve come on site, you’ve probably heard me harp on this point: You MUST save your work, and by “save your work,” I mean use the Save Script…To Data Table option under the red triangle menu. Going back to my first experience with automating a workflow, this is how I started: by saving a bunch of standardized Graph Builder reports to a JMP table that I would regularly update. I cannot stress strongly enough the of power this one little tip. It’s like having Dr. Who’s Sonic Screwdriver. The possibilities are endless.

Let’s look at a quick example:

Open Help > Sample Data Library > Big Class Families.jmp
Open a distribution dialog: Analyze > Distribution
Put age, sex, height, and weight into Y, Columns and click OK.
Under the Distributions red triangle menu, select Save Script > To Data Table
Give it a name (or just use the default). If you use the default in this case, it will show up in the data table as Distribution 2 because there’s already a Distribution script in the data table.
Now look back at the data table in the upper left corner. You should see your table script in the list with a green check mark next to it.

In the context of this series, those table scripts can be thought of as little applets (mini applications) or macros. They are directly portable between data tables (just copy and paste), provided the column names used in the script are the same. They can also be edited (right-click on one and select Edit). So, they are also portable to data tables that don’t have the same column names if you are willing to edit the column names in the script, which is not a big deal. We’ll cover how to do that in the next post. Table scripts can also be used to string smaller scripts to automate a larger workflow. We’ll also cover that in the next post.

Save Script > To Clipboard

We’re going to cover the last entry in the red triangle menu: to clipboard item. We’re going to cover this one next because the entry is identical to the to Script Window option, but a lot less instructive. That’s not to say it’s not useful, just that you can’t see what’s going on. So, for the sake of time, here’s what it does. It takes the code that is created by Save Script…To Data Table and puts it in the clipboard so that you can put it in another script somewhere. It’s handy. And, as your scripting skills grow, you’ll figure out where it’s useful for you. For the moment, I want to back up into the more interesting and useful option: saving to a script window.

Save Script > To Script Window

The next entry in the Save Script to… red triangle menu, we are going to cover what is probably the most instructive of the three I’m discussing. Saving a script to a script window does just what it says: It creates a Script Editor window (we’re going to go deeper on the Script Editor in the next episode) and populates it with JMP-generated JSL to recreate whatever was in the window you were working from. The important point with this entry is that you can study the code and learn from what JMP generated.

Let’s look at a trivial example using Big Class again. Go ahead and click the green button next to your saved script to rerun the analysis. Then, in the report window, select Save Script > To Script Window from the red triangle menu. Now, look at what JMP has written in the Script Window.

JMP has created a Distribution Report: Distribution(…);. JMP has put two Nominal Distribution subgroups into the report: Nominal Distribution(Column(…)) and two Continuous Distribution subgroups: Continuous Distribution(Column(…)). At the end of the script, JMP has put an incredibly important piece of code: the “;”. It’s called the “Glue Operator” and we’ll cover it in more detail in the next episode. For now, you just need to know that it’s a way to signal to JMP when it’s running the code that another instruction is coming. The fact that JMP puts that on the end automatically makes the novice scripter’s life a lot less complicated.

The last point that makes this entry so useful is what happens if you leave that script window open. Go ahead and close the Distribution Platform, but leave your Script Editor open. Now, using the open data table, create a Fit Y by X report (under Analyze) using Height as the X variable and Weight as the Y variable.

Next, select Save Script to Script Window again.

Did you notice how the window was updated with another entry? That script will now recreate the two windows you were just working with, once you know how to run one. Speaking of which…

Running scripts

So, you’ve got some code in a script window, now what? You probably want to be able to run the code that you’ve created. I mean, you could just leave it up to impress people that walk by your desk, but it’s probably better to run it. There are a few ways to run code in JMP: the main menu, keyboard shortcuts, the tool bars at the top of the scripting window, the Enter key, or just right-clicking. Let’s quickly go through each of these.

The main menu and keyboard shortcuts

The Run Script command is in the same place in Windows and MacOS. If you’re in a scripting window, go under Edit > Run Script. The hotkey for this is CTRL+R for windows and CMD+R for MacOS.

The tool bar button at the top of the scripting window

If you look at the top of the script window, you should find a Run Script button (unless you’ve got the tool bars hidden). In Windows, it looks like a little green play button, similar to the ones for table scripts. In MacOS, it looks like a little JSL document with a JMP Man and the green play button.

Using the Enter key

Note here that I mean the button in the lower right part of the number pad on an extended keyboard. If you push that button, JMP will run the line of JSL where the cursor is located. I used this one a lot before I got comfortable with the JSL Debugger.

Right-clicking

Selecting some code and right-clicking will bring up a context menu that has a Run Script option toward the bottom.

I recommend that you try the different forms of running scripts and find the one that you like best. I’m partial to the hotkey, except when I’m in Application Builder, where I prefer using the tool bar options. Over time, you’ll probably gravitate to one or two that become your favorites. Also, as a side note: All of these options do slightly different things depending on whether you have a section of code selected, your cursor is at the end of a line, or your cursor is on a blank line. So you might want to experiment a bit with that, too.

Wrapping up for this week

All right, believe it or not, you’ve now got enough information to be insanely dangerous with JSL...in a good way, of course. Let me show you what I mean.

Let’s use that script we created today. I’m going to make one little tweak by wrapping a New Window(); function around the distribution and bivariate functions JMP created. You can learn more about New Window(…); in the Scripting Index. Note that the first part of New Window(…); starts on line 1 and the last part is on line 10! I’ve got my Script Editor set up to draw a nice little line connecting them so I can see where they are (more on that later).

Then, let’s copy the text for that script and go back to the data table. Select New Script from the red triangle menu in the uppermost left corner of the data table.

Paste the script text into the dialog, give it a useful name and click OK.

If you’re following along, you should see your report in the script area of the data table. If you run it, you should get something like this:

And that’s the little trick I used to turn a two-day analytical marathon into a 15-minute exercise in cutting and pasting. I spent a little time getting the data into a JMP data table that I could easily update. I then exported all the graphs I needed for that report into a single JSL window using Save Script to Script Window. I wrapped them in a new window (so that they were all in one place) and put that script in a Table Script in the data table. Was I working with JSL? Yes. Was I writing lots of code? No. Did I increase my efficiency? Definitely.

Now that we are getting into the actual coding, I need to stress this last point. Coding is about increasing your efficiency:

How efficiently you can solve problems.
How efficiently you can create reports (so you can get back to solving problems).
How efficiently you can work with your data.

The code that we are going to create in this series is computer-generated or it’s simple. And that’s OK. It’s about efficient code not sexy code. When I was in college, one of my professors (a fellow musician) had a quote on the wall outside his office: Music that is simple is no better or worse than music that is complex. The same goes with code. I was reminded of this a while back by the person that taught me coding. So, remember: Code that is simple is no better or worse than code that is complex. All that matters is that it meets its intended purpose and increases the efficiency or productivity of those who use it.

Homework

For this week’s exercise, you’re going to look back at your problem statement. Think through the punch list you created and generate code using the workflows we’ve covered today. The code can be the actual code that you are going to use or prototype code that you think you’ll need to modify a little to answer your problem statement. Try to generate as many of the bits and pieces you think you’ll need using the red triangle menu and save them out as .jsl files. If you get stuck or can’t find a way to get JMP to generate the code you need, reach out for help, either in the Community or in the comments here. Don’t forget to search the Help, and don’t forget that you can search the Community, too!

Until next time, good luck and happy coding!

The 2nd thing I wish I knew when I started scripting: Getting help

MikeD_Anderson — Tue, 04 Aug 2020 14:49:29 GMT

These are my favorite helps for JSL.Welcome back!

Apparently, the esoteric content from my first entry in this series didn’t frighten you off. Great! If you’re just tuning in to the series, I’m discussing the five things I wished someone had told me when I started learning to script in JMP. (In all fairness, I was told a lot of this stuff in the JSL course I took from SAS Education and interactions with the instructors after the fact.)

The series is set up for you to follow along with some homework at the end of each episode. So, go back and have a look at the previous entry in the series so you’re not wondering what’s going on or check out the whole series.

Getting Help

In my day job, I talk to people about JMP. All day long. I talk to people that have never seen the software before and people who have been using it since before I knew how to use a mouse. It’s a lot of fun, and I learn a lot from all the questions that I answer. One of the things I cover with everyone (because some people refuse to use Google and some just don’t know any better) is where to get help with the software. A lot of programming books I read actually fail to address this incredibly important topic or shove it in the last chapter, almost as an afterthought. My position on this topic, particularly in the context of JSL, is more pragmatic. There is no way we can cover everything in the context of this blog series (or even the next blog series I’m planning, for that matter). There’s just too much to cover. So, before we follow the white rabbit any further down the hole, here are my favorite helps for JSL.

The Scripting Index

Open up JMP and go to Help > Scripting Index. Congratulations! You just accomplished one of the most important tasks for learning JSL. This one tool has got me out of more jams than I care to mention. Every scripter I know has this open during a serious coding session. Every. Single. One. Just get used to having it open.

It is the collected knowledge (with examples!) of the JMP developers. It’s updated every release with any changes to JSL or new commands for new platforms in JMP. Why is this so important? Unless you’re scripting all day, every day, there no chance that you will know everything about JSL. This tool fills that gap.

Here's a little example. Open up the Scripting Index, and type "powerpoint" in the filter field. The bottom entry in the second column says "Save Presentation." This gives you the form of the command export a report as a PowerPoint File and an example of how to use it.

JMP Tutorials and Sample Data

Now, in JMP go to Help > Sample Data. Note the buttons along the top of the interface. One of them says, “Open the Sample Scripts Directory.”

Guess what’s in there? Examples of completed scripts! If you want to know how to get a script to add a formula column to a data table, there’s an example in there. If you want to know how to get JMP to connect to an external dll file, there are a few examples in there. There is nothing wrong with starting from these scripts or using them as templates when you are getting started! That is why they are there.

JMP Documentation

JMP includes a massive amount of documentation, more than 6,500 pages in JMP 15. At ~1,500 pages, the JSL resources make up roughly 20% of that documentation. There are two books in the JSL resources: “The Scripting Guide” and “The JSL Syntax Reference.” This content is also fully searchable on the web, either here The Online Scripting Guide or The Online JSL Syntax Reference. These resources have a lot of useful information and examples about specific use cases. They are good counterpoints to the Scripting Index; the Index is a tactical tool, while the JSL documentation is more of a strategic tool or general reference. The two books provide a lot of material on the different things that JSL can do. In practice, you would look to the Scripting Guide for things like how to work with scripting graphs or how to use Application Builder. You would look to the Syntax reference if you wanted to know all the options for the Bivariate() function or how to use the F Sample Size() function. The Syntax Reference has a lot of overlap with the Scripting Index.

Reverse Engineering Scripts and Add-Ins

Now, let’s take a break for some trivia:

Did you know that there are multiple copies of da Vinci’s Mona Lisa?

It’s true. Da Vinci ran an art studio with many apprentices who studied his work and attempted to replicate it.

The Mona Lisa (1503–1516) by Leonardo da Vinci Isleworth Mona Lisa (16th Century) by Workshop of Leonardo da Vinci Prado Mona Lisa (16th Century) by Workshop of Leonardo da Vinci Portrait of a Woman (16th Century) by Raffaello Sanzio (a.k.a. Raphael)

Here’s another one:

Did you know that Michelangelo made his money early in his career by copying other artists’ work?

He studied the work of prior masters to improve his skills. He also took commissions to recreate ancient works for people (the Renaissance equivalent of ordering a high-quality touch-up print).

OK, one more:

Do you know what the difference is between an art student and an art forger?

One copies art to learn the techniques while the other tries to pass their work off as being done by the original painter.

Do you see a trend here? I am of the opinion that there is nothing wrong with borrowing from other people’s work with full transparency and proper attribution. Why should you reinvent the wheel? Find someone else’s wheel and see if you can make it better or better fit your needs, as long you give them a mention somewhere. That’s how you learn this stuff!

So, what does that have to do with JMP scripting? Well, there’s a wealth of information put out by people that have studied scripting for years (or actually wrote the code for JSL). It’s all there for you to learn from. But how do you do that? The easy way is to look in the JMP Community in the JSL Scripts section of the File Exchange or the JSL Cookbook. Now, what if you want to have a peek at what people are doing in their add-ins? You can do that, too; it just takes an extra step.

The thing most people don’t know about the JMP add-in transport file (the ones with the .jmpaddin file extension) is that it’s just a zip file with a custom file extension. Unless the author has encrypted their code, there is nothing stopping you from taking a peek under the hood. Just change the “.jmpaddin” to “.zip” and let your operating system (OS) unpack it for you. After the OS has unpacked the zip file, you’ll find a number of files in the folder (the ones you want have the .jsl extension). Tear into them (and pray that the person you’re cribbing from does a good job of commenting their code; more on that later).

You can also have a look at add-ins that you've installed already. You just have to go diving into the file system. The location can be a little hard to find (especially if your IT Admin's have made it hard to go into hidden directories). But, if you can get there, you can find the possible location of your add-in directory here (there are two possible locations each for MacOS and Windows). In that directory, you'll find a bunch of folders (one per add-in) that are named like a backward URL. Inside each folder are the JSL files for a given add-in. Go ahead and have a look!

Authors note (15Jul2020): After publishing this article, , the author of the Uncharted Blog and JSL wizard, reminded me in the comments section that you can get to installed add-ins directly from the JMP GUI. Have a look at the comments below for details!

Paid Resources (Courses and Books)

Strictly speaking, it is possible to learn JSL with just the resources provided in the JMP installation and help on the JMP Community. If all you want to do is string together some bits to make your life a little easier, then that will probably be enough. The thing is, there is a learning curve with JSL. If you want to take your coding beyond what we’re covering in this series, one way or another, you’re going to have to fight up; there are no shortcuts here. You can get some help using the resources I’ve mentioned already, but you’ll still have to put in the time. That said, this is one of the places where I really recommend using some paid resources if you can. If nothing else, you’ll be more committed by putting some skin in the game.

The first one is the Introduction to JMP Scripting Language course offered through SAS Education. I took this course when I started scripting seriously, and I highly recommend it. There is a distinct value in sitting in a classroom and being guided through increasingly more complex exercises. It also gives you someone you can ping if you get stuck on something. (I still talk with my instructor all the time. Hi, Di!) And the course notes are a really helpful resource. There are also two books out there that are really helpful: “Jump into JMP Scripting” and “The JSL Companion.”

Conclusions

That’s the end of this week’s lesson. And it’s the last of the preliminaries. Next week we’re getting into the code. Your homework for this week is to look through the resources I’ve outlined and get to know them. Specifically, start looking at the Scripting Index. Using your problem statement from the last lesson, search through the index for some possible tools or examples that you think might be useful.

The 1st thing I wish I knew when I started scripting: There are different kinds of scripts

MikeD_Anderson — Thu, 09 Jul 2020 02:10:22 GMT

Once more unto the breach!

I was talking with my dad a few weeks back. Not to date him, but he’s been using JMP since version 1.0. I remember seeing the 3.5” discs sitting next to our Mac II-se when I was first learning to use a mouse. The same machine was on display at the 2019 Discovery Summit conference in Tucson, Arizona. Here’s a picture of it:

Anyway, my dad and I were discussing my first Five Things blog series. He mentioned that he wished someone would write a series about how to get started with scripting in JMP. At the time, I was already in the process of writing some other articles, so I tucked it away for a future project. Life, however, had other plans. A week later, I was asked to help with a customer presentation and, to my surprise, one of the topics they wanted me to cover was an introduction to scripting! The universe loves jokes, apparently.

Anyway, here we go again! Following the format of my first Five Things series, here are the five(ish) things that I think everyone should know when they are starting to script. Up first, a conceptual thought.

See a need, fill a need

Before you start scripting, you need to consider the need. Odds are, if you want to script, there is some reason. Maybe you’re doing it to make yourself more efficient at work. Maybe you’re looking to standardize analytical methods for a group. Or maybe you are, in fact, one of those rare people that decided to pick it up for grins and giggles. None of these are bad reasons, although the last one may make things a little more difficult (I’ll explain why in a bit). The point is that you should have a reason in your head and a specific purpose for your first script.

There are scripts, and then there are scripts

There are, in my mind anyway, two different types of scripts: those that act on existing data, and those that create data. Scripts that work on existing data interact with a data table and existing platforms in JMP to create a new user defined output. These can be things as simple as wrapping a window around multiple reports or as complex as a custom-built report in Application Builder. The key commonality is that scripts like this are modifying existing content.

The second group are scripts that create data. Scripts that create data can be simulation tools, custom graphics, tools for managing metadata in a JMP file, etc. This group is a little more nebulous, but the common theme is that the scripts extend JMP in ways beyond enriching existing data or analyses. Most of the time, these scripts are going to create new content from scratch.

Scripts that modify existing content are generally where people start when they think about learning JMP scripting language (JSL). This is also where we are going to start our discussion. So, let’s take a step back and think about why you are interested in learning JSL.

Give me a sentence, Vasily. One sentence only, please.

Take a moment and think about your need. Which pain point are you hoping to address with JSL? Do you now see why scripting for grins and giggles (at least at first) can be tricky? You’re much more likely to fight up the learning curve if you’ve got a reason to reach the top. I’m not saying you can’t pick up JSL for the heck of it, just that the fight is harder without the extra motivation.

The next step, and the final one that we’re going to cover in this episode, is what you do with that need. The simple answer is that you are going to do two things. First, you are going to write down that (possibly nebulous) idea that’s driving you to learn JSL into a single sentence – a problem statement. And, I do mean just one sentence. Make sure you are able to crystallize the idea into something tangible that you can build off of. It will also help with reining in scope creep. Here are some examples:

I want to automate all the graph generation for my weekly report into one script.
I want to automate the formatting work for a customer provided data table.
I want to generate a set of control charts from a data file provided by my factory computer system.

Note that I’m not talking about specifics; I am simply stating the pain point that has me looking at JSL. And then I stopped writing.

Make your punch list

Got your problem statement? Good! Now we are going to write a series of specific deliverables for your script. It will be our punch list to help us locate the resources SAS provides for learning JSL. Do not list any graphical user interface (GUI) or user interface (UI) specifically. List the actions that the script needs to perform. Here are examples for the problem statements above:

Problem Statement 1: I want to automate all the graph generation for my weekly report into one script.

Punch List:

Pull a data set from our production database containing the last three weeks of data.
Make a graph of time vs temperature for my product.
Run an analysis of time vs temperature using a polynomial model.
Export the graphs and reports to a PowerPoint file.

Problem Statement 2: I want to automate the formatting work for a customer provided data table.

Punch List:

Rename columns based on internal standards (using table from specs).
Assign units to the needed columns (using table from specs).
Convert Lat and Long to Geographic coordinates.
Change defective units column from continuous to ordinal.

Problem Statement 3: I want to generate a set of control charts from a data file provided by my factory computer system.

Punch List:

Import data from network location.
Assign spec limits.
Build control charts.

Note here that I’m not boxing myself in too much. I’m just building a punch list of things I need the script to do. Since I’m assuming that you’ve never worked in JSL before, I’m also assuming that you don’t know how you’re going to attack these problems. By writing them down, you know what you need to look for in the help and resources. Which is where I will pick up in the next installment of this series.

Closing thought for this week

Since this series is going to span a few weeks, I decided to make it more of a work-along series than something you just read. So, take these last points I made as homework. If you want some accountability or feedback, put them in the comments below. I’ll chime in if I can help or let you know if you might be setting yourself up for some trouble down the road.

For bonus points, open up a script window in JMP (File > New > Script) and put your problem statement and bullet points into the script window, one per line with a double forward slash (“//”) at the start of the line. You may see the text color change to green for those lines in the JSL Script Editor. We’ll cover what you just did in an upcoming episode.

Oh, and if you’re wondering who Vasily is and why he should be giving you a singular sentence have a look here.

How to make JMP Live reports more interactive

john_cromer — Wed, 24 Jun 2020 14:19:20 GMT

Although many types of JMP graphics are currently interactive in the latest version of JMP Live, sometimes specific options within a JMP platform prevent full interactivity when enabled. By using an incremental approach, it may be possible to slightly adjust the appearance of a graph in order to obtain better interactivity on JMP Live.

JMP Live output is built using an HTML5 foundation. What does that mean? It means that while there are additional features beyond HTML5 found on JMP Live (for example, the inclusion of filtered rows in calculations used to regenerate graphs on the server), many of the interactivity tests can be performed without JMP Live by examining only the HTML5 output.

Let’s examine two approaches for examining and remedying interactivity: interactively (no pun intended) and through the script editor. Both approaches will use the graphics generated by running the following example JSL script.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp", Invisible );

jmpreport = NewWindow("Test",
    VListBox(
        DataFilterContextBox(
            HListBox(
                dt << DataFilter(Conditional, Count Excluded Rows(0), Local, Add Filter(columns(:age))),
                dt << GraphBuilder(
					Show Control Panel( 0 ),
					Fit to Window( "Off" ),
					X Group Edge( "Bottom" ),
					Variables(
						Y( :weight ),
						Group X( :age )
					),
					Elements( Box Plot( Y, Legend( 1 ), Jitter( "None" ), Box Style( "Solid" ) ) )
				)
            )
        )
    )
);

The result looks like this:

A data filter and box plot as they appear on the JMP client.

The Interactive Approach

One way to examine interactivity is through the Publish menu option. On any JMP window’s menu bar, click File > Publish. On the Select Reports window, check the Test box for the test results window. Notice the message immediately below the window list: “Interactive HTML is partially implemented for this report. Please see log messages.” This is the first clue that some aspect of your results may not be fully interactive.

The interactive Publish menu option.

Inspect the log window (View > Log). Among other messages, you’ll see this one: “Interactive HTML: Box plot with solid style is not interactive.” Now you can focus on this specific graphical element.

Log containing interactivity messages.

If you proceed to publish this report (either to File or to JMP Live) and hover your mouse pointer over a box in the box plot, you’ll see this tooltip: “This feature is not interactive.” (Although the boxes won’t have descriptive hover text for the corresponding data table rows, JMP Live still allows the graph to be redrawn based on filter selection changes in the Local Data Filter.)

A data filter and box plot in JMP Live. The box plot is marked non-interactive.

In this instance, if you’re willing to choose a box plot style other than Solid, you can achieve better interactivity. Go back to your original Test window in the JMP client, right-click on a box in the box plot and choose Box Plot > Box Style > Normal.

Changing the box plot options interactively in the JMP client.

If you repeat the Publish process, you’ll see descriptive text for data points when hovering over the box plot.

Interactive data filter and box plot in JMP Live.

The Script Editor

Another way to examine interactivity is through the script editor. Append the following code to the example script and run it to save the web report locally:

webreport = New Web Report();
webreport << Add Report(
	jmpreport,
	Title( "Test" )
);

url = webreport << Save("$DOCUMENTS/Test");

If( !Is Empty( url ),
	Web( url )
);

Alternatively, append and run the following code (substituting your actual URL, username, and API key appropriately) to publish to JMP Live:

webreport = New Web Report();
webreport << Add Report(
	jmpreport,
	Title( "Test" )
);

url = webreport << Publish(URL("https://your_url.com/"), username("your_username"), ApiKey("your_api_key"));

If( !Is Empty( url ),
	Web( url )
);

Inspect the log window. You will see the same messages that the interactive approach generated, including “Interactive HTML: Box plot with solid style is not interactive.”

Log containing interactivity messages.

When you inspect the report (either locally or on JMP Live) by hovering your mouse pointer over a box in the box plot, the tooltip “This feature is not interactive.” is revealed.

A data filter and box plot in JMP Live. The box plot is marked non-interactive.

You can make the same fix as you did in the interactive method by simply updating the script directly in the script editor window. Specifically, change the GraphBuilder() > Elements() > Box Style() from Solid to Normal.

Elements( Box Plot( Y, Legend( 1 ), Jitter( "None" ), Box Style( "Normal" ) ) )

If you re-run the script (which contains the Save or Publish message) and re-examine the output, you’ll see the descriptive box plot data point text when you hover over it.

Interactive data filter and box plot in JMP Live.

Bonus Tip: Moving Between Interactive and Scripting Approaches

You can easily move between interactive and scripting methods. On most JMP graphics’ red triangle menu on the client, there is a command to reveal the script used to generate the graphics. Click Save Script > To Script Window.

Saving a graphics script to a script window.

By doing so, you can open a new script window or append to the current script window with the script used to generate the graphics.

Graphics script saved to a script window.

Similarly, when viewing the graphics on JMP Live, you can navigate to the Details section (click the button that looks like a lowercase ‘i’) and click the download buttons (the downward-pointing arrows) next to each data table or the JSL Script to save a local copy of them. (Note that these download buttons are available only if the Publish Data and Allow Data to be Downloaded publish options were specified.)

Data table and JSL script available for download from JMP Live.

With both data table and script in hand, you can incrementally examine the effect of platform script option changes on the resulting interactivity, even if you were not the author of the script published to JMP Live.

JMP Live admin tricks: Administrating groups

andyzhou — Wed, 06 May 2020 16:40:37 GMT

Administrating Groups

Administrating groups in JMP Live presents a different challenge, as groups have their own subsets of users and administrators. A system admin is a group admin, but a group admin does not necessarily have to be a system admin. As a system admin, you are not immediately added to every group contained within the server. However, you can still view and administrate a group you are not a part of.

A view of the available options in the Posts tab, showing all published reports when a group is opened in JMP Live.

The following steps show how to view a group that you are not a part of, as well as how to access all the group’s settings:

Go the Groups section under the Admin tab to see all the created groups within the server.
Select the desired group by clicking on Group Name.
View the Manage Group tab. As an admin, you can:
1. Edit Permissions to change a user’s ability to publish and remove posts, as well as change a user’s status as an admin.
2. Add Members to add new members to the group.
Select the Posts tab. This shows all of the reports published to this group. As an admin, you can view and administrate the reports (see Administrating User Reports section).
Select the Join button in the top right to join the group.
Select the … button in the top right to perform the following actions:
1. Add Members.
2. Edit Groups to make changes to three possible settings:
  1. Change the Group Name.
  2. Change the Description of the Group.
  3. Change the Group Type, choosing among four different types of groups; each group is defined in a box to the right in the Edit Group window.
3. Delete Group.

A display of the Manage Group tab for a group, with various permissions available that can be edited for each user. The group admin is indicated by the Crown symbol.

A group admin can access the same settings using the same steps as above, but replacing Steps 1 and 2 with the following:

Go to the Groups tab to see all the groups that you are a part of, including groups for which you are not an admin.
To see if you are an admin of a group, click on Group Name. If you are not its admin then you will not be able to change any settings for that group.

There are three ways to assign a group admin:

A system admin changes a user’s permissions within a group to make that user a group admin.
A group admin changes a user’s permissions within a group to make that user a group admin.
The group owner (user who created the group) can set his/her permissions to be a group admin.

Trick #4: Editing user permissions using JSL code

A system/group admin can also script administrative actions using JMP Live’s REST API in conjunction with JMP Live’s HTTP Request() function.

The attached script (JMP Live Login.jsl) is used to log in the admin. This process requires four pieces of information:

JMP Live URL - line 8
API Key - line 9
Username - line 10
Password - line 11

Once logged in, the code below provides examples of administrative actions that can be scripted:

// --------------------------------------------------
// Get a list of all the groups and generate a table with all the groups
// --------------------------------------------------
request << reset( Url( webJMPUrl || "/api/groups/?all_groups=true" ), Method( "GET" ), Insecure );

data = request << Send;
dt2 = JSON To Data Table( data );

// --------------------------------------------------
// Add a member to a group with the following permissions:
// “allow_publish” – can this user publish to this group?
// “allow_delete” – can user delete content published by other members?
// “allow_admin” – can user add, delete and change permissions of group members?
// --------------------------------------------------
request << reset(
	Url( webJMPUrl || "/api/groups/" || Char( Group ID number ) || "/members" ),
	Method( "POST" ),
	Form( Fields( ["uid" => user’s id number, "allow_publish" => 1, "allow_delete" => 1, "allow_admin" => 1] ) ),
	Insecure
);
data = request << Send;

// --------------------------------------------------
// Change permissions
// --------------------------------------------------
request << reset(
	Url( webJMPUrl || "/api/groups/" || Char( Group ID Number ) || "/members/" || Char( User ID Number ) ),
	Method( "PATCH" ),
	Form( Fields( ["allow_publish" => 1, "allow_delete" => 0, "allow_admin" => 0] ) ),
	Insecure
);
data = request << Send;

// --------------------------------------------------
// Delete a member from a group
// --------------------------------------------------
request << reset(
	Url( webJMPUrl || "/api/groups/" || Char( Group ID Number ) || "/members/" || Char( User ID Number ) ),
	Method( "Delete" ), Insecure
);
data = request << Send;

Editor's note: Have you read the other installments in our JMP Live admin tricks series?

JMP Live admin tricks: Administrating users and reports

andyzhou — Wed, 06 May 2020 16:39:47 GMT

Administrating Users

Users and their respective permissions can be managed in the Users section under the Admin tab. Once you open the Users section, all of the users present within the server are listed by their username/email. There are five different edit options available to a JMP Live admin:

Changing their display name.
Checking and unchecking “User is enabled”:
- This checkbox determines whether the user has access to the reports in JMP Live
Checking and unchecking “User can publish content.”
Checking and unchecking “User has administrative privileges.”
Checking and unchecking "User can use the API."

Users section of the Admin tab, where admins have a handful of options available to them to determine user permissions.

Trick #3: Generate a table of JMP Live user data

A full table of data on users can be generated using a REST API through JSL’s HTTP Request() function.

The attached script (JMP Live Login.jsl) is used to log in the admin. This process requires four pieces of information:

JMP Live URL - line 8
API Key - line 9
Username - line 10
Password - line 11

Once logged in, the code below can then be used to generate a table containing data on the users.

// --------------------------------------------------
// List all users and generate a data table with users
// --------------------------------------------------
request << reset( Url( webJMPUrl || "/api/v2/users" ), Method( "GET" ), Insecure );
data = request << Send;
dt = JSON To Data Table( data );

Administrating User Reports

Private report where the edit options are indicated by the "Pencil" symbol.An important aspect of being a JMP Live admin is knowing how to manage your users’ reports. As an admin, you have access to all reports that are published publicly to the JMP Live server (e.g., any published report where the “shared to:” option selected is “everyone”). You also have access to all privately published reports (e.g., any published report where the “shared to:” option selected is “only me”) and all reports that are published to groups of which you are a member. You do not have immediate access to reports published within groups to which you are not a member, but these reports can be accessed by going to the Groups section under the Admin tab.

Each report can be managed by selecting the “three dots” button, which allows you four options:

Edit, so that you can edit the report.
Delete, so that you can delete the report.
Flag as Inappropriate, so that you can flag the report for future review in the Flagged Content section found in the Admin tab.
Stop Sharing with Groups, so that you can select the report to no longer be shared to a specific group.

Selecting “Edit” allows you to edit the user report with the following options:

Change the name of the report.
Change the thumbnail of the report.
Change the description of the report.
Allow downloading of the data behind the report.
Change how the report is shared:
- “Only Me” makes the report private to the user who published it.
- “Everyone” makes the report public to the JMP Live server.
- “Groups” allows you to select the groups where the report will be included. A report can be included in multiple groups, with a maximum of 8 groups allowed.

Editor's note: Have you read the other installments in our JMP Live admin tricks series?

Analyzing survey responses from virtual hands-on workshop

HadleyMyers — Wed, 22 Apr 2020 13:09:53 GMT

We hosted the first JMP Virtual Hands-on Workshop in Europe recently. After the workshop, we surveyed the participants to find what they thought of it and how we could improve their experience. We thought we’d share what we learned with everyone, to show you how we do our analysis in JMP.

One response was nearly universal: Post a recording of the workshop so people can watch it whenever they have the time. Consider your wish granted. The link to the recording can be found here, along with all data sets needed to follow along.

Virtual hands-on workshop participants answer survey questions to help us better understand their needs and preferences.

Length and Time of Day

We wanted to know two things about time: what time we should hold future workshops and how long they should be. As can be seen from the figures below, there is a significant interest in having shorter workshops, while the time of day doesn’t seem to make a difference. With that in mind, we plan to offer two sessions for each upcoming workshop, one in the morning and one in the afternoon.

Most people prefer shorter workshops (a, left), but no significant preference for time of day (b, right).

Topic Preferences

Next we asked for the topics that users would like to see covered in future workshops. Rather than give respondents a finite set of topics, we allowed them to answer with their own words, which we put into a word cloud using Text Explorer, which you can see below. Three of the top four words are “data,” “analysis” and “studies.” While these are inherently good words, they are rather nebulous. The fourth word – DOE – gave us more to work with. About 40 percent of respondents made some reference to DOE or designed experiments, such as Definitive Screening Designs.

Most used words in “What topics would you like us to cover in future workshops” free-form response.

The remaining 60% requested a range of topics, which we divided into three clusters:

Statistical modeling and analytical techniques (e.g., neural nets, principal components analysis).
Automating analysis through scripting and dashboards.
Unique challenges specific to their companies’ situations.

With those requests in mind, we will schedule workshops on DOE, modeling and analytical tools, and scripting and dashboards. For those with specific challenges, we will be reaching out to them to see how we can support them on a one-to-one level.

Comments

The last question was also free form: Any other comments? We used Latent Semantic Analysis in Text Explorer to group the responses for us, which yielded meaningful results into three clusters (see below):

Problems playing the posted video (red)
Complaints about the sound during the event; we had some muting problems (blue)
Compliments (green)

Clustering terms from “Additional Comments or Suggested” free-form response.

In response to the technical issues, we will involve more JMP team members to help with the workshops: one person to deliver the workshop, another to answer questions via the chat, and a third solely responsible for the managing the presentation medium and ensuring attendees are muting upon entry.

Thanks so much to everyone who attended the workshop and then took the survey. We’re happy to report that in the two weeks following our workshop, JMP has rolled out JMP Discovered, an online series on a range of topics. New workshops will be scheduled as part of JMP Discovered so keep an eye out for additional information. In the meantime, stay safe!

JMP Live admin tricks: Automating publishing

andyzhou — Mon, 30 Aug 2021 18:34:52 GMT

JMP Live workflow: from data input to automated report publication.As a JMP Live admin, you are going to be asked by the producers in your organization how to publish reports automatically on a routine schedule without having to manually click “Run.” That's easy!

Trick #2: Automate Those Reports

Tell them all they need to do is use JSL (JMP Scripting Language) in conjunction with Windows Task Scheduler.

Doing so allows users to automatically publish reports that are updated with fresh sets of data without any manual intervention. Armed with JSL’s HTTP Request function, users can easily create the scripting needed to pull in fresh data from online databases via API requests (more information found in this blog post). HTTP Request can even be used with online databases (like Salesforce) that require OAuth 2.0 token authentication to access the data contained within.

Once the data is pulled into JMP, the user can easily manipulate the data table with the various object calls available in JSL. After using this data table to generate a Graph Builder report, save the script with the Save Script option found in the red triangle menu of the Graph Builder interface. The combination of the HTTP Requests (described above), data manipulation scripts, and Graph Builder scripts is all you need to successfully generate the up-to-date report you want – with just the click of a button.

Once you have the scripts necessary to generate the up-to-date report, the next step is scripting the first publish to JMP Live. However, before the first publish can occur, you must create a JMP Live Connection object that will serve as the reference to your JMP Live site. This object is initialized and maintained interactively in the (File -> Publish -> Manage Connections...) menu. If you are creating the JMP Live Connection for the first time, select "Add New..." and input the JMP Live URL in the "URL: " field and the API Key in the "API Key..." field. The API Key is obtained by going to your Profile in JMP Live and selecting "Edit", which will bring up a window with a "Generate API Key" button. Clicking this button will bring up another window where your API Key is displayed. Finally, the input for the "Connection Name: " field can be set as anything that you desire, and will serve as the key used to reference a JMP Live Connection in JSL and in the (File -> Publish -> Publish to JMP Live...) interactive menu.

Now that the JMP Live Connection has been successfully created, you can begin scripting the first publish to JMP Live. First, start a connection to JMP Live using stored connection information by using the New JMP Live() function and referencing the "Connection Name" key specified earlier. After adding the reports you wish to publish to a New Web Report() object, use the Publish() function to successfully script the first publish of the web report. The Publish() function publishes a webreport to the JMP Live site specified in the JMP Live Connection object. Below is an sample script on how to start a connection to JMP Live and then publish for the first time:

liveconnection = New JMP Live( Connection( "MyJMPLive" ) );
webreport = New Web Report();
webreport << Add Report( r1, Title(“Dashboard of Air Quality”) );

jmpliveresult = liveconnection << Publish( webreport, Public( 1 ), Title( "My Report" ) );

After the report has been successfully published to JMP Live for the first time, there are two options for successfully updating a single report in JMP Live. The first option is to replace an existing JMP Live report with another single report using the Replace() function. Below is a sample script on how to replace a report:

liveconnection = New JMP Live( Connection( "MyJMPLive" ) );
webreport = New Web Report();
webreport << Add Report( r1, Title(“Dashboard of Air Quality”) );

jmpliveresult = liveconnection << Replace( webreport, ID( "Report Identifier" ) );

The second option is to use the Update Data() function to update the data table for a report without replacing the report itself. The Update Data() function allows for changing of the file name if desired. Below is a sample script on how to update the data table for a report:

liveconnection = New JMP Live( Connection( "MyJMPLive" ) );
dt = Data Table ( "Example Table" );

jmpliveresult = liveconnection << Update Data( "Report Identifier", Data( dt, "Example Table" ) );

And now your script is complete! Use Windows Task Scheduler to set the schedule of when the report is run by following these steps to set up the task correctly:

Once Task Scheduler is open, select “Task Scheduler Library” in the menu on the left.
Select “Create Task…” in the menu on the right, which will open a new window where the task is created.
Make sure to name the task and then select “Run whether user is logged in or not.”
Go to the Actions menu and select “New,” which will open a new window called “New Action.”
Keep Action as “Start a program” and select “Browse.”
Find the location of the jmp.exe file for JMP 15 and select “Open” for that file.
In the New Action window, copy and paste the file location of the script you wish to automate to the “Add arguments (optional)” box. Make sure the file location is contained within quotation marks.
Select “OK” to save your changes.
Select “Triggers” in the Create Task window.
Select “New,” which will open a new window called New Trigger.
Use this menu to set the schedule for the script. Make sure that Enabled is checked before selecting “OK” to save your changes.
Select “OK” to save the task; enter your system username and password.

There are several important items to keep in mind while using Task Scheduler:

JMP 15 sessions will remain hanging even if the script is executed without any errors. It is extremely important to include a Quit("No Save") line at the end of your script to prevent this from occurring.
JMP 15 sessions will remain hanging if the script is executed and an error occurs. The best way to prevent this is to modify your script to be Try ( your script, Quit("No Save") ). Doing so will quit the JMP 15 session and prevent it from hanging if an error occurs.
When your system password changes, you must go to each task and re-enter your new system password, otherwise it will not run. After double-clicking the task and selecting “OK,” you will then be prompted to enter your system username and password.

A successfully scheduled task with some of the required settings visible.

Editor's note: Have you read the other installments in our JMP Live admin tricks series?

Replicating Pandas GroupBy behavior in JMP

MikeD_Anderson — Thu, 16 Apr 2020 12:31:01 GMT

The Engineering Mailbag
Episode 4: Pythons and Pandas, Oh My!

Every now and again we systems engineers run into interesting questions that would fall somewhat outside the typical range of JMP usage. These applications are generally clever, and often bring home how using data isn’t limited to solving business or technical problems. Other times, the questions are just unexpected, challenging problems. These “curve balls” (as I like to call them) come in many different forms: coding problems, interesting analyses, ways of visualizing data… you get the idea.

I haven’t written from the mailbag in a while, but this particular question really seemed to be something that needed to be addressed. The tool I used to answer the question really isn’t discussed as much as I think it should be. So, this is an opportunity for me to help out a colleague from New York and show everyone something interesting that you might not be aware that JMP can do.

The Question

Mike[JMP],

Often, I need to do groupbys where I need to grab say the max value everything grouped by some label where the Python Pandas equivalent would be something like:

df.groupby(['Something', 'Something Else']).max()

Another might be something like pandas.nlargest with a groupby, where I would want say the top 5 things out of a groupby.

And after a couple minutes of fiddling with a column formula I give up and export the data to reimport it into JMP later. Tabulate sort of does this, but not always in the way I want. So a "How to JMP like Pandas" document would be super helpful for me, and might be helpful for JMP with how widespread Pandas has become…

Thanks,

Nick

My Response

Hi, Nick!
There is a trick that you can use to get more flexibility out of formulas that mimics a lot of what [Pandas] can do. It involves telling JMP what columns you want to group by. You do this by right clicking on the column name and selecting New Formula Column > Group By.

After doing that, any formula columns you create by right clicking on the column name and using New Formula Column> … will be grouped by the grouping columns you set.

After that, it’s just a matter of filtering or subsetting using data filters or local data filters. Let me know if that helps!

Best,

M

Pandas: Cute Mustelid or Data Analysis Tool

All right, for the uninitiated, Pandas is not a cuddly, chromatically challenged mustelid. Pandas (usually abbreviated as “pd” in python scripts) is a “…Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series [source]." It basically fills the role that the data organization and manipulation parts of JMP (data tables, Row and Column menus, etc.) do in the Python programming environment. So, it’s pretty important to Python data analytics workflows. Now, if you are looking to run Python from within JMP, this ain’t the blog article for you. Go have a look at this white paper. If you are looking to break yourself free from doing data transformation in a command line, read on.

DataFrames and Data Tables

Still here? Awesome. Okay, the rest of this article is going to go through some of the common bits of Pandas and their analog in JMP. It’s not a comprehensive list by any means; it’s intended to point your nose in the right direction when thinking about the two environments.

The first bit I want to cover is the “DataFrame.” In Pandas, this is where everything is stored. Like data tables in JMP, it is essentially a collection of data columns with a name (column name), etc. You act on columns and rows (transform things, summarize, etc.) but you’re always acting on either the DataFrame as a whole or some part of it.

In JMP, the primary data container is the data table. You can act on columns and rows in the same ways you would in a Pandas DataFrame. In both, you are able to deal with missing values in the data table. In JMP, they are designated as either a cell with a “.” in it (for numerical data) or a blank cell (for character data). You can transform data, summarize it, etc. (more on those bits later). In both cases, the more you tell the software about the data the less leg work you have to do, which brings me to my next point.

Column Properties and dtypes

In the matter of column properties and data types (dtypes in Pandas), JMP wins hands down. Pandas inherits most of its data type understanding from another Python package called NumPy, which is cool. NumPy is pretty sturdy code, but it’s really limited in this regard. It handles about eight types of data (see the table below[source]) with somewhat limited support for different formatting (it gets pretty confused by currencies without help, for instance).

Kind of Data	Data Type	Scalar	Array	String Aliases	Documentation
tz-aware datetime	DatetimeTZDtype	Timestamp	arrays.DatetimeArray	'datetime64[ns, <tz>]'	Time zone handling
Categorical	CategoricalDtype	(none)	Categorical	'category'	Categorical data
period (time spans)	PeriodDtype	Period	arrays.PeriodArray	'period[<freq>]', 'Period[<freq>]'	Time span representation
sparse	SparseDtype	(none)	arrays.SparseArray	'Sparse', 'Sparse[int]', 'Sparse[float]'	Sparse data structures
intervals	IntervalDtype	Interval	arrays.IntervalArray	'interval', 'Interval', 'Interval[<numpy_dtype>]', 'Interval[datetime64[ns, <tz>]]', 'Interval[timedelta64[<freq>]]'	IntervalIndex
nullable integer	Int64Dtype, …	(none)	arrays.IntegerArray	'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64'	Nullable integer data type

Strings	StringDtype	str	arrays.StringArray	'string'	Working with text data
Boolean (with NA)	BooleanDtype	bool	arrays.BooleanArray	'boolean'	Boolean data with missing values

The column properties in JMP comprise about 43 pages of the Using JMP book. They are significantly more extensive and are good at recognizing data types as they are imported. Of particular interest to a new JMP user is the fact that column properties extend well beyond the data types that Pandas uses. It can handle a host of metadata that trigger actions across the software. Now, to be fair, you can attach attributes to a Pandas DataFrame, but since the DataFrame is just a container, you have to either code in the support for the attributes or hope someone put in the support in the package you are using. All that is already done for you in JMP.

Tabulate, Column Formulas and GroupBys

The truth is that the previous two items don’t really matter that much in the context of the question I was trying to answer. For our purposes, the data table/ DataFrame is just a box to put stuff in. The column properties/types can help with analytical considerations, but, again, that’s not important for the question we’re working with here. The bit of Pandas that we are interested in is called GroupBy. And it’s for slicing and dicing data in preparation for doing other analyses. A quick example using the JMP Big Class data table and getting the average height and weight of boys and girls by age might look like this:

# Import Pandas import pandas as pd # Bring in the data data = pd.read_csv("/Users/mike.anderson@jmp.com/Desktop/Big Class.csv") # Put the data into a DataFrame df = pd.DataFrame(data) # Run the group by, get the means, and show the results summ_df= df.groupby(['sex','age']).mean() print(summ_df)

With the output looking something like this:

height weight sex age F 12 58.600000 100.200000 13 59.000000 95.333333 14 62.600000 96.600000 15 63.000000 102.000000 16 62.500000 113.500000 17 62.000000 116.000000 M 12 57.333333 97.000000 13 61.250000 94.250000 14 65.285714 103.857143 15 65.200000 110.800000 16 68.000000 128.000000 17 69.000000 153.000000

Normally, JMP users would look to Summary or Tabulate as their primary tools for these data transformation operations. They might even use the Formula Editor to create a column. But, there is a secret weapon in the data table (and in the platform column list) that you can use to do just about everything you’d want to do in Pandas and more. That secret is the ability to right click on a column name and select New Formula Column. This works anywhere that there is a column list! From there you have many different options for doing various transforms, such as combining columns, doing cumulative sums in rows, etc. That, in and of itself, while handy, is not particularly magical. The place where things get really interesting is when you turn on the “Group By” flag for a column in JMP. This will automatically make any formula columns you create aware of the levels in a grouping variable. Just think about that for a second -- you can create a formula column anywhere in the software that can be stratified based on a grouping column.

Let’s look at an example of what this means from another question I received recently:

An Example: TWO QUESTIONS FOR THE PRICE OF ONE!!

About the same time the initial question came through, I got a question from another customer about how to do a numerical integral (area under a curve or AUC) for a large collection of samples. Moreover, they were willing to provide me with a sample data set they had published previously for a journal article in Environmental and Molecular Mutagenesis.

It turns out that what they wanted to do was easily accomplished using Group By Flags and Function Columns. Here’s the procedure (you can follow along using the data from the JMP Public site:(

First, let’s graph the data. The details of the study aren’t important for this discussion, though it is interesting reading. The short version is that it’s a large collection of chemicals with different concentrations (Conc. (uM)) and a series of genetic responses. For our exercise, they’re interested in calculating the AUC for each of the genetic responses. Here’s a look in Graph Builder with a column switcher:

The first step in the process of calculating the AUC is setting up the grouping. We’re going to right click on the Chemical column, select “New Formula Column,” and then “Group By.” We’re going to do this again for the Biomarker column. Note that order can matter here. We’re grouping first by Chemical and then Biomarker within Chemical. After that, it’s just a matter of using the “New Formula Column” in the data table to build up the calculation.

Going forward, I broke each step out into individual columns for the sake of clarity. And, if you’re really interested, I’m using the Trapezoidal Rule Riemann Sum for the math here.

Ok, back to the calculation. The first thing we need is the distance between each point in the x-axis (the point-wise change in x). We’re going to select the Conc (um) column, right click on the title and select New Formula Column>Row>Difference. Now open the formula in the new column that JMP created. It should show something like this:

Note that JMP is handling calculating between rows and has adjusted the formula to reset when it detects a new Chemical or a new Biomarker type.

Next we need the average value of Value between each data point and the previous one. Right click on the column header and select New Formula Column>Row>Lag. Open the formula and modify it a little so that it’s calculating the difference between rows and not just giving the value of the previous row. That’s shown in this screenshot:

Those two columns (I renamed them dx and dy in the data table) need to be multiplied together to get the area of a segment. Select the two columns, and right click on the column name of one of them. Select New Formula Column>Combine>Product. The new column gives us the approximate area under the portion of the curve between each data point and the next one. I renamed that column iAUC(for instantaneous AUC) in the data table.

The last step is to sum up the values for each section. This is done by right clicking on the iAUC column and selecting New Formula Column>Row>Cumulative Sum. Because we selected the Group By columns at the beginning of the exercise, the tally is reset for each sample in the data set.

From here we could use Tabulate or Summary to create a table with the max values for each sample (the approximate total area under the curve). Here’s an example for a common chemical:

Final Thoughts

And, that’s it. You can easily replicate behaviors in Pandas using structures that exist in JMP. Just remember to use the Group By variable and you’re off. No mustelids, chromatically challenged or otherwise, required.

Editor's note: Have you read the other installments in our Engineering Mailbag series? See how Mike helps a user who was using JMP to decide whether to sign up for a snow removal service and develop unique visualizations for STEAM education programs.

JMP Live admin tricks: Optimizing loading speeds for reports

andyzhou — Wed, 06 May 2020 16:37:48 GMT

Learn tips to make your life as a JMP Live admin easier as you meet the needs of users.Life as a JMP Live admin poses a different challenge than the life of an average JMP Live user. Average users’ vistas are limited to their own published reports while admins have access to the entire world of JMP Live, including each user’s more constricted view. Admins can manage every aspect of JMP Live, including users’ public and private reports and all the groups these reports are contained within.

Admins also have the additional, unique responsibility of serving as primary contacts for JMP Live producers’ scripting and automation questions. Want to know more about the essentials needed to answer these questions, as well as the best server management practices? Read on here and this whole series.

Trick #1: Make a Table Summary Instead

A slow-loading webpage is perhaps one of the biggest pet peeves of any online user. Similarly, a slow-loading report on JMP Live will just as quickly try the patience of a JMP Live consumer. This headache can be avoided altogether by following a few easy steps to reduce the loading time of a published report.

Often, the main culprit behind slow-loading reports is the data table. The performance of a report depends on the size of the data tables, as a large data table with over 100,000 rows will certainly cause a report to take its time when loading. With these large tables, even making a simple change in the data filter can significantly bump up the load time. However, there is an easy way to prevent this from occurring.

To tackle this issue, it is important to understand how a good data visual is made. Often, a good data visual involves aggregating data into different groups (for example, the different bars on a bar graph). The Tables Summary function in JMP is the perfect tool for performing this task, since it easily aggregates the data in the exact format you need. The result? The significantly decreased number of rows in the data table allows you to create the same exact data visual but with much better loading times in JMP Live!

A great example can be found in the "Auto Raw Data.jmp" file found in the Sample Data library. This data table has 20,000 rows, which would result in a slowdown if a report using this table is published to JMP Live. By using The Tables Summary function in JMP, the data can easily be aggregated to produce a data table with fewer rows. In this example, aggregating the data by AgeClass and Gender produces a table with only four rows, which ensures a much better loading time in JMP Live!

Here's a trick: Use the Tables Summary function to easily aggregate data to produce a data table with a lot fewer rows.

Editor's note: Have you read the other installments in our JMP Live admin tricks series?

Help us help you!

HadleyMyers — Thu, 02 Apr 2020 12:00:34 GMT

On 31 March we hosted our first JMP Virtual Hands-on Workshop in Europe. There was a huge amount of interest leading up to this event, as evidenced by the number of registrations. More than 250 people joined the workshop live, and we really hope those watching found it useful. We know this is a difficult time for everyone, so we’d like to reach out to all of you to offer our help and support as best we can.

Prediction Profiler with "Simulate" turned on, as show in "Case Study #4."

If you watched the workshop live, please take a moment to complete the survey below. We want to hear what you thought of it, what you liked and didn’t like, what you would like to see more or less of, etc. We also want to know what topics or JMP features you’d like us to cover at future workshops. Since this was the very first event of its kind in Europe, we will use your responses to improve future events so we can provide you with the best support possible.

Even if you didn’t see the workshop but would be interested in attending future events, we still want to hear from you. Please take a moment to answer the survey so that we can discover what’s important to you and how we can deliver content in a way that suits your needs.

You have until 8 April to complete the survey; respondents’ names will remain confidential. As soon as we’ve compiled the results, we will post them and key takeaways in a follow-up blog post shortly thereafter.

If you’d like to see a recording of the event, click on the link below. Just below the link is the JMP journal and Excel dataset needed to follow along.

Survey

Getting to the point of your data using JMP

statpharmer — Fri, 10 Apr 2020 12:36:44 GMT

Leaders of organizations wear many hats and therefore have little time to digest statistical results. That’s why it’s so important for those who live and breathe data to make the most of their communications with leadership: Present clear, compelling data visualizations. Make your case and explain how the data supports it. Anticipate which questions will be asked. And – when possible – answer those questions in-real time with data.

This level of preparedness will not only allow for faster and more effective decision making, it will also help avoid the need for time-consuming follow-up meetings that can delay decisions, waste leaders’ valuable time and reduce the organization’s effectiveness overall.

Furthermore, good communication skills coupled with well-executed data analysis are a shortcut to earning credibility and trust. This is the best foundation to build upon for motivating leadership to make optimal decisions, hence; “get to the point.”

JMP 15 is chock full of features designed to help users get to the point.

For one, there is the visual workflow – the ability to use your cursor to drill down into a graph. Graphlets are a new feature of JMP 15 that allow the user to effectively share information in meetings. First, you create a high-level graph for initial consumption. The hover label editor in Graph Builder allows the user to select the next level of detail with a graph of choice.

Take, for example, a plot illustrating the annual results of a large organization. The hover label provides a graphlet that splits the summary into results for a specific facility or department. The practice can be extended as the data includes variables with meaningful splits. Organizations realize significant benefits from these dynamic graphical tools as better decisions result from extracting useful information from data in record time.

Tune in to learn new tips for using the JMP visual workflow to get right to the point.

SSL certificates for JMP Live

dieter — Wed, 04 Mar 2020 19:31:44 GMT

Need help getting and using the certificates that JMP Live requires? Here you go!

Are you setting up SSL certificates for JMP Live? The process described here will help you get and use the certificates that JMP Live requires.

SSL certificates are text files containing cryptographic keys that enable a browser and the web server to have secure communication. The primary reason for using SSL certificates is to keep sensitive information sent across the web encrypted so that only the intended recipient can access it. The keys are specific to an organization and are typically provided by third-party vendors like DigiCert, Symantec, and Network Solutions. JMP Live cannot run without these certificates.

JMP Live requires three certificate files:

A file containing the certificate itself.
A file containing the certificate’s private key.
A file containing the root certificate chain, also known as the Certification Authority (CA) certificate chain.

The JMP Live installer provides this set of certificates that allows for minimal testing on a local host setup. However, it generates warnings when the site is viewed in the local browser, and the site is inaccessible from browsers outside the install machine. You can get certificates yourself using the following process.

Requesting certificates from a Certification Authority (CA)

The certificate and private key are configured for your organization. The root certificate comes from the third-party vendor.

If you need to purchase a certificate from a third-party vendor, please refer to their documentation on how to obtain the required files.

If certificates are available in your organization, the CA Web Enrollment role service provides a set of web pages that allow interaction with the CA role service. These web pages are typically located at https://<servername>/certsrv, where <servername> is the name of the server that hosts the CA Web Enrollment pages (for example, https://certificates.company.com/certsrv).

Once you find the location of the certificate enrollment service, follow these steps:

Get the certificate: Request a certificate from the CA Web Enrollment role service web page. Follow the instructions provided by your organization to submit the request for a certificate. Once the certificate has been issued, download and save it to the install machine. When prompted for an encoding, select Base64. The certificate contained in the downloaded file is an encoded piece of data that usually starts with -----BEGIN CERTIFICATE ----. The downloaded file will be file Nr. 1.
Get the certificate’s private key: The private key is generated simultaneously with the certificate request. You won’t see the private key while generating the certificate request. It is usually created in the background and saved in the server’s filesystem without any notification. There are several ways to find a certificate’s private key (see Helpful Links at the end of this blog post or use Google for options).
Use one of the following methods to change the format of the private key into one that JMP Live can accept:
1. After exporting all certificates in the certification path and the private key, the downloaded file (PKCS #12) is usually password-protected. JMP Live cannot use the password-protected file. To remove the password, you can use OpenSSL (see Helpful Links at the end of this blog post or use Google for options). Run the following command to remove the password, where cert.pfx is the exported certificate and private key file:
  openssl pkcs12 -in cert.pfx -nodes -out cert.pem
  If the above command does not work for your version of OpenSSL, use this command instead:
  openssl pkcs7 -print_certs -in cert.pfx -out cert.pem
  After running the above command, the cert.pem file contains the unprotected private key that is needed by JMP Live as well as all certificates. Open cert.pem using the Notepad tool and copy and paste the private key into a new file. The private key contained in cert.pem is an encoded piece of data that usually starts with: -----BEGIN RSA PRIVATE KEY ---- This new file is file Nr. 2.
2. Some certificate utilities might allow you to export the certificate and key in Apache format. Doing so usually does not require a password, and the certificate and key are exported into two separate files. These two files can be used as File Nr. 1 and 2 without having to transform them using OpenSSL.
Get the root certificate: The role service web site usually contains a menu item "Download a CA certificate, certificate chain, or CRL." JMP Live needs the CA certificate chain. This item can be used to download file Nr. 3. When prompted for an encoding select Base64. If the downloaded file is password-protected, use the same steps to remove a password as described for the private key (step 2 above). If the file is in PKCS#7 or P7B format, it needs to be converted to Privacy Enhanced Mail or PEM format. To do that, run the following command:
openssl pkcs7 -in certnew.p7b -nodes -out root.pem

Verify the certificates (optional)

You can verify your PEM files by running the following command:
openssl x509 -in root.pem -text

This will display the content of the certificate and the different pieces, like Issuer, Subject, and expiration date, can be verified.

Using the certificates in JMP Live

Once you have the required certificates, you can begin using them in JMP Live:
Keep the files in a central location or copy them to the SSL directories in the JMPLive and JMPPool folders, typically at:
C:\Program Files\SAS\JMP Live\JMPLive\ssl
C:\Program Files\SAS\JMP Live\JMPPool\ssl
Edit the .env files of JMPLive and JMPPool to reflect the appropriate paths to the three files (certificate, private key, and root certificate).
1. Edit the JMP Live .env file. Note that the root certificate for JMP Live is in the JMPPool/ssl directory.
  SSL_CERT_FILE='c:\Program Files\SAS\JMP Live\JMPLive\ssl\ cert.pem'
  SSL_KEY_FILE='c:\Program Files\SAS\JMP Live\JMPLive\ssl\key.pem'
  SSL_CA_FILE='c:\Program Files\SAS\JMP Live\JMPPool\ssl\root.pem'
2. Edit the JMP Pool .env file (root certificate only).
  SSL_CA_FILE='c:\Program Files\SAS\JMP Live\JMPPool\ssl\root.pem'
Use the Services tab of the Task Manager to start or restart the JMP Live web application (websjmpsvc.exe) and the JMP Pool Manager (jmpppolsvc.exe).

Helpful Links

Finally, here are a few links to reference as you set up SSL certificates:

A simple way to automate "File Import/Append/Cleaning/Generate Report"

JerryFish — Fri, 20 Mar 2020 13:20:09 GMT

Ever wanted to automate processes and analyses? You can do it in JMP! This video shows how to automate the following steps, with a minimal amount of JSL (JMP Scripting Language) coding:

Create a file dialog to allow you to select and open a file
Append that data to a "master data table"
Clean the data (using stored Recode command)
Generate a report from the updated Master Data Table

When is a column not a column?

markbailey — Tue, 03 Mar 2020 17:25:27 GMT

Scripting JMP often involves using data column objects. We might want to compute statistics for a column of numeric data. We might want to change a column attribute or property. In any of these cases, we need a reference to the column so that we can provide it as an argument. There are several ways to get or use a reference. Unfortunately, they might seem to be equivalent up to a point until we are surprised. A scientific theory is testable. A theory holds until a test invalidates it. Let's test some theories about column references.

I am using the height data column in the Big Class data table from the JMP sample data folder, which is C:\Program Files\SAS\JMPPRO\15\Samples\Data in my case. Let's get started.

Names Default To Here( 1 );

// open a data table
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

Our first test will use several different forms of a column reference for height.

// use a reference to a data column
sum = Col Sum( :height );
Show( sum );
sum = Col Sum( Column( "height" ) );
Show( sum );
sum = Col Sum( Column( 4 ) );
Show( sum );
sum = Col Sum( :Name( "height" ) );
Show( sum );
sum = Col Sum( As Column( "height" ) );
Show( sum );

OK, what did we get?

Same result every time. So my theory is that these five different references are equivalent. My first test confirms my theory. But it is only the first test. Let's devise another test. Let's first get the column reference, store it, then use the stored reference to compute the same statistic.

// get a reference to a data column and then use it
col = Column( "height" );
sum = Col Sum( col );
Show( sum );
col = Column( 4 );
sum = Col Sum( col );
Show( sum );
col = :Name( "height" );
sum = Col Sum( col );
Show( sum );
col = As Column( "height" );
sum = Col Sum( col );
Show( sum );

Note that I did not include the very first form of a reference, :height because the value returned and stored in the expression col = :height is missing (".")! That result is not helping my theory! What did we get with the rest of test?

Wait a minute! I didn't get a single error, but I didn't get the same result each time. The first two results are correct, but the last two are wrong. Let's perform one more test of my theory. Let's again obtain a column reference, store it, and then use it to send a message.

// get a reference to a data column and then use it
col = Column( "height" );
type  = col << Get Data Type;
Show( type );
col = Column( 4 );
type  = col << Get Data Type;
Show( type );
col = :Name( "height" );
type  = col << Get Data Type;
Show( type );
col = As Column( "height" );
type  = col << Get Data Type;
Show( type );

Here is what I get with the first two ways.

The first two ways seem to work as expected. Let's try the third way.

col = :Name( "height" );
type  = col << Get Data Type;
Show( type );

This way didn't work.

What about the last way?

col = As Column( "height" );
type  = col << Get Data Type;
Show( type );

I get the same error as the last time.

It is clearly time to revise my theory. (And maybe to confide in the JMP developers.) My theory is now that

:height, Column( "height" ), Column( 4 ), :Name( "height" ), and As Column( "height" ) all work or act as valid a column reference.
Column( "height" ) and Column( 4 ) are functions that return a column reference.
:height, :Name( "height" ), and As Column( "height" ) are not functions that return a column reference.
As Column() is an alias for the colon (":") scoping operator, as seen in :height, which both return the value of the current row if used in an assignment.

So what is my recommendation?

Use :height, Column( "height" ), Column( 4 ), :Name( "height" ), or As Column( "height" ) as a column reference.
Use Column( "height" ) or Column( 4 ) functions to get a column reference.
Do not use :height, :Name( "height" ), or As Column( "height" ) to get a column reference.

I hope that this adaptation (abomination?) of the scientific method helped to clarify how column references can be used in JMP scripting. I hope that you follow the recommendations and avoid further confusion and frustration.

Visualizing the point(s) as the plot thickens

rcarver — Fri, 13 Mar 2020 20:31:07 GMT

Visualization is crucial throughout the process of data analysis, from initial exploration through communication of important findings and insights. We discover patterns, uncover lurking relationships, and communicate visually.

With high-dimensional data — that is, data tables with many columns and rows — it can be challenging to create graphs that summarize the data enough to reveal overall patterns and also show the scale and extent of the variability in the data. Let’s consider some of the ways of balancing these demands.

To illustrate, we will look at some airline departure and arrival data for all US domestic flights in March, 2019. The data table is provided with my book Practical Data Analysis with JMP, 3rd Ed. The full data table contains 38 columns and over 632,000 rows. First, we consider arrival delays, reported in minutes. In Graph Builder, we drag the ArrDelay column to the Y drop zone and produce the default graphic, an outlier boxplot.

This plot shows a highly right-skewed distribution of delays, with a median near zero. Perhaps 50% of flights arrive ahead of schedule, and 75% have only modest delays. Checking the 5 Number summary box tells us more, and one might wonder about the maximum value of 1,928 minutes, or a 32-hour delay.

The upper whisker in an outlier boxplot is constructed at Q3 + 1.5*(Interquartile range), which in this case is 31.5 + 1.5(21) = 37.5. A large, but indeterminate number of observations lie above 37 minutes. If we use the Arrow tool to select the points above the whisker, we learn that there are nearly 54,000 outliers, but in this initial rendering of the graph, we see a dark line with some attenuated points at the higher values.

A bigger issue is that the outliers can visually dwarf the fact that the large majority of flights arrived within a few minutes of their scheduled times. If our priority is to communicate the center and shape of the distribution, we might prefer a histogram or violin plot. The latter is a Contour plot option. Here are the two graphs side by side.

These visualizations highlight the location and shape of the entire distribution, without overemphasizing the outlying points. On the other hand, they really don’t show any of the data points at all.

Of course, we can add every individual point to any of these plots by right-clicking on the graph and choosing Add > Points. Here’s the violin plot with points overlaid on the contour.

The problem is that there so much overplotting that most of the points are not visible. What to do?

Here are five simple tactics:

1. Divide and conquer. Making a set of small graphs may not be your first choice, but we start with this one because it helps to more clearly illustrate the other tips. If there is excessive overplotting in an unidimensional graph, include another dimension. For example, our data contains a categorical variable identifying the principal cause of the delay when a delay has occurred. In this dataset, only about one third of the flights were delayed, so we are seeing approximately 200,000 points rather than 600,000.

Using Main Cause as a grouping variable helps to identify the underlying dynamic reasons for variation in delays and it also more clearly displays the individual data values. For example, it’s immediately clear that most delays seem to be caused by the carriers and late equipment, that the longest delays are attributable to the carrier, and that Security delays are rare and brief.

Note that in this rendering, the points are no longer aligned vertically, but have been jittered. That is, they have been repositioned slightly left and right to put white space between them. This bring us to the second tactic.

2. Add or adjust jitter. In this univariate graph, some points have been uniformly shifted to the left or right. The default jitter pattern spreads the points apart so that we can get a sense of how many flights there were at each value, but the points cover the contours.

In Graph Builder, the Jitter controls provide options.

If, for example, we choose Random Normal jitter and reduce the Jitter Limit, we can reveal the violins again:

Reducing the jitter uncovers the violins, but increases overplotting. The next three tactics deal more directly to reduce the ink density. We can invoke these options by right-clicking on the black dot and Arr Delay in the legend in the upper right of the graph.

3. Use open markers. Choose Marker > and select the open circle rather than the solid dot. Now proximate points visibly overlap, rather than overlay one another.

4. Use smaller markers. Even with open markers, the density of the points leads to considerable overplotting. Sometimes simply using smaller dots leaves more white space, allowing individual points to be more visible. We do this by right clicking on the black circle in the legend and choosing Marker Size > 1, Small.

5. Use translucent markers. Open markers let us see through the center of a point, but the outline of the points is opaque. To reduce the visual confusion, we can make the points semi-transparent. Again, right-click the black circle in the legend, and choose Transparency… to open this dialog:

Enter a value between 0 and 1 as indicated. Here is the result of entering 0.2, for 20% marker density.

In this version, the very sparse uppermost points fade nearly into obscurity, but the abundance of points at lower values are accentuated.

These five basic tactics work equally well with bi- and multi-variate plots. Here, for example, is a scatterplot of Arrival Delay vs. Departure Delay, using small open markers. To add a third dimension, we color the points according to the main cause of the delays.

We now are plotting all 612,000 points, the large majority of which are quite near or below the origin. The mass of black points in the lower left are flights that arrived early or on time, and hence have no main cause of arrival delay. The comparative frequency and severity of the delays are visible by cause. More importantly, despite the thick density of the data, we can detect the overall patterns as well as the individual points.

Earn a JMP certification while at Discovery Summit Munich 2020

Becksterg — Mon, 17 Feb 2020 18:17:47 GMT

Are you going to Discovery Summit Munich 2020? Take advantage of reduced rates for certifications.Have you been wanting to add a JMP certification to your belt but simply haven’t had the time? You can maximize your time and save money at Discovery Summit Munich 2020 and register for one of the three JMP certifications being offered at the conference.

Validate your skills and enhance your credibility in specific areas with these certification exams. Take any of the three exams during the morning or afternoon sessions on Tuesday, 10 March. Sign up for the exams when you register for the conference. As a conference attendee you’ll be able to take any of the exams for 50% off the standard price!

JMP Certifications being offered include:

Design and Analysis of Experiments: During this performance-based examination, candidates will apply the skills and knowledge necessary to design and analyze industrial experiments.
JMP Scripting: During this performance-based examination, candidates will apply the skills and knowledge necessary to perform a variety of scripting and programming tasks in the JMP environment using the JMP Scripting Language (JSL).
Statistical Thinking for Industrial Problem Solving: During this performance-based examination, candidates will apply the skills and knowledge necessary to apply statistical thinking and fundamental statistical methods to solve industrial problems.

Learn specific details about the JMP credentials by visiting SAS Global Certification. If you have questions, please contact Becky Gray

Importing and merging multiple XML files

Phil_Kay — Tue, 04 Feb 2020 07:04:20 GMT

I wanted to use all the data from my exercise activity tracking app as part of an analysis for a blog post. So I downloaded all the data from the last six years.

When I unzipped this there were 660 files -- one for each logged activity:

And they were all XML file format. I had never worked with XML before. For my analysis, I would need to get each file into JMP data table format and then stack them all into a single JMP table.

So it is great that there is a new XML Import Wizard in JMP 15. Even better, you can copy the settings from the wizard and use them to import all the files in a folder into a single JMP table in one step. Let me show you how...

First of all, your files will all need to have the *.xml file extension. My files had a different extension. There are tools available outside of JMP for changing the file extension for multiple files, if you have this same problem.

Next, in JMP, File > Open and navigate to the folder where you have your XML files. Change the file type to XML Data Files (*.xml):

Now, click on a single file to select it. Make sure you have Open as Data (Using Preview) selected:

Click Open.

The XML Import Wizard will launch. Having never worked with XML before I didn't know what to do here at first. But it didn't take me long to figure it out by trying a few clicks. The wizard can make some guesses at the structure of the data. And you can see whether the settings are right with the Import Preview. You can also change the Column Names that you will have in the resulting JMP table.

(I have obscured some of the details in the data here because you don't need to know exactly where I was on the 4th April 2014!)

Now, click OK and you should have a nicely formatted JMP data table, which is great if you just want to import a single XML file. But I didn't want to repeat this manual process for all 660 files. This is the really nice bit...

Right-click on the Source script in the data table and Edit. Now, copy the XML settings from the script:

In your clipboard, you now have the instructions for importing any of the XML files in your folder (assuming they have the same structure).

Now, File > Import Multiple Files. Click on the |...| button to select your folder of XML files:

You could also use the "Select files by..." options to filter out certain files, if required.

Click Settings and paste the XML settings into the Script box in the XML tab:

At this point, I kept Stack Similar Files option checked to bring all the files together into a single JMP data table.

All you need to do now is click Import, and you will have your data in JMP ready to explore!

Removing white space from control charts

tonya_mauldin — Tue, 14 Jan 2020 18:50:04 GMT

Do you have excluded observations in your data? Are you using the Local Data Filter? Are there large white spaces in your control chart that you would like to remove for a cleaner look? If you have JMP 14 or later, you can use the “Show Excluded Region” command. This command shows or hides the regions of the chart where samples have been excluded.

Example

Open Socket Thickness.jmp found in the Quality Control sample data folder and run the script attached to the data table.

dt=Open("$SAMPLE_DATA/Quality Control/Socket Thickness.jmp");
dt<<run script("Control Chart Builder");

The result is a graph that shows how your process is doing overall. For each cavity and hour, Thickness measurements are plotted. Each cavity has its own set of control limits. This enables you to do easy comparisons across groups.

Suppose you are concerned only with Cavity 4. You think that Cavity 4 is not producing the results it should be. Rather than starting from scratch, you can add the Local Data Filter to the report. Click on the red triangle next to Control Chart Builder and select Local Data Filter. Select Cavity and click on +. Select 4.

While this does give us what we want, the graph of interest is very small, and there is a lot of white space. To view only the data from Cavity 4 with no white space, from the red triangle next to Control Chart Builder, deselect “Show Excluded Region”.

This removes the white space from the graph. You get a cleaner look at the data from only Cavity 4 so that you can do further investigation.

Conclusion

“Show Excluded Region” shows or hides the region of the chart where samples have been excluded. This can be useful when you are using the Local Data Filter, as well as when there are excluded observations in your data. Use this option to remove white space from your graph.

Control Chart automatic sorting blues

tonya_mauldin — Thu, 12 Dec 2019 13:56:39 GMT

Control charts are a way of studying process changes over time. Hence the data is plotted in time order. When Control Chart Builder was introduced in JMP 10, this became easier because JMP sorted the data for you prior to creating the chart. In the legacy control chart platforms, you had to sort the data, prior to creating the control chart. However, there are some cases in which this sorting does not fit the data. JMP 15 introduces the option “Sort by row order”. This option sorts the subgroup and phase variables by the occurrence in the data table.

Example

As an example, open Braces.jmp found in the Quality Control sample data folder.

dt=Open("$SAMPLE_DATA/Quality Control/Braces.jmp");

Notice that although the first column is Date, the format is not one in which JMP supports as a date/time format. The modeling type for this column is nominal rather than date/time. First, let’s create a U Chart using the legacy control charts. Go to Analyze->Quality and Process->Legacy Control Charts->U. Specify # defects as Process, Date as Sample Label, and Unit size as Unit Size.

Click OK.

Notice that there was no sorting of the x axis. The 25 rows of the data table are plotted in the order in which they occurred in the data table. This is the proper order for this data. 10FEB80 comes before 12FEB80 and the last x axis label shown is 12MAR80.

Let’s compare this to Control Chart Builder. Select Analyze->Quality and Process->Control Chart->U Control Chart. New for Version 15, the first 9 options under the Control Chart menu are dialogs that launch the Control Chart Builder platform.

Specify # defects as Y, Date as Subgroup, and Unit size as n Trials.

Click OK.

Notice this is not the same control chart that we got with the legacy platform. On the x axis, 01MAR80 comes before 26FEB80. The modeling type of Date was nominal. Therefore, this variable is going to be sorted in alphabetical order. Since 01 comes before 26, a date of 01MAR80 comes before a date for 26FEB80. Looking back at the data table, we see that the data is already sorted in the desired order. This is common for control chart data as sometimes the data comes directly from measuring instruments recording data in a sequence. From the red triangle next to Control Chart Builder, select Sort by row order.

The Sort by row order option gives you the desired control chart that matches the legacy control chart platform. Date is sorted by the order in which it occurred in the data table.

Conclusion

When your data is already sorted in the desired order for a control chart, you can use the "Sort by row order" option to avoid the automatic sorting done by the Control Chart Builder platform.

Modernizing the Distribution platform – Capability analysis

tonya_mauldin — Wed, 04 Dec 2019 15:00:11 GMT

The Distribution platform is one of the most widely used in JMP. This platform is not only used for testing which distribution fits your data. It is also used for data exploration, capability analysis, and more. Distribution has been around since the first version of JMP, and it was time for a modernization. JMP 15 has delivered the modernization of this commonly used platform.

My first blog post was about the new fitters in JMP 15. The second blog post detailed comparing distribution fits. The third blog post covered the new options for fitted distributions. This blog post discusses the new capability options in the Distribution platform.

History

The Process Capability platform was introduced in JMP 12. Parts of this new platform were added to Control Chart Builder in JMP 12 when the platform was introduced. Now, with JMP 15, parts of the platform have been added to the distribution platform. This not only makes the platforms easier to maintain but also consistent with each other.

Launch Changes

There are several differences that happen around the launch of a distribution report. 1) The histogram only shows spec limits if “Show as Graph Reference Lines” is checked in the Spec Limits column properties. 2) The ability to disable the capability analysis even if there are spec limit column properties. 3) If there is a process capability distribution column property, that distribution will be used for the capability analysis.

As an example, open Process Measurements.jmp found in the sample data folder.

dt=Open("$SAMPLE_DATA/Process Measurements.jmp");

Run the following JSL to set up column properties so that we can investigate these new features.

Column(dt, "Process 2")<<Set Property(
			"Spec Limits",
			{LSL( 5 ), USL( 17 ), Target( 10 ), Show Limits( 1 )}
		);
Column(dt, "Process 3")<<Set Property(
			"Process Capability Distribution",
			Process Capability Distribution( Weibull )
		);

For Process 2, the JSL above sets the Spec Limits column property as LSL=5, Target=10, and USL=17 with show limits (Show as Graph Reference Lines) turned on. For Process 3, the JSL above sets the Process Capability Distribution property as Weibull.

Select Analyze->Distribution and Specify Process 1, Process 2, and Process 3 as Y, Columns.

Notice the new check box in the lower left corner (“Create Process Capability”). This check box only appears if at least one Y, Column has a Spec Limits column property. Uncheck this box and click OK.

In previous versions of JMP, which did not have this “Create Process Capability” check box, you would get capability analysis for each of these three processes. Additionally, there would be spec limits drawn in each of the three histograms since each column contains a Spec Limits column property. In JMP 15, you get the following output:

No capability analysis is given for any of the process variables. Only Process 2 contains spec limits on the histogram because it was the only process variable that had “Show as Graph Reference Lines” checked in the Spec Limit column property.

Now, check the box, "Create Process Capability" and compare the output. Go to Analyze->Distribution. Click Recall. Check the box for “Create Process Capability”. Click OK.

Capability analysis is given for all three process variables. In the main histogram (top), Spec limits are shown only for Process 2. The capability analysis for Process 3 is based on the Weibull distribution because Process 3 had a Weibull Process Capability Distribution column property.

Capability Report

The capability report looks the same and has the same options as the report you would get using the Individual Detail Reports option in Process Capability. Not only did this allow us to make things more consistent and easier to maintain, but we were also able to complete some user requests.

Ppk labeling is now the default labeling for the Overall Sigma Capability report (as seen in the previous image).
The report now always shows both the Within and Overall Sigma Capability indices when a normal distribution is assumed (as seen in the previous image).

Capability commands when there are no spec limit column properties

Just as in the older version of the Distribution platform, there are two different ways of accessing capability analysis when there are no spec limit column properties. Open Thickness.jmp found in the Quality Control sample data folder.

Open("$SAMPLE_DATA/Quality Control/Thickness.jmp");

Select Analyze->Distribution. Select Thickness 03 and Thickness 04 as Y, Columns and click OK. Click on the red triangle next to Thickness 03 and choose Process Capability.

These options are available in the Process Capability platform. The dialogs are simply arranged differently.

The other method of obtaining a capability analysis is through a fitted distribution. Click on the red triangle next to Thickness 04 and select Continuous Fit->Fit Beta. Now click on the red triangle next to Fitted Beta Distribution and select Process Capability. The capability dialog looks the same as the above image with one additional section.

These options replace the Quantiles and Set Spec Limits for K Sigma options found in previous versions of the distribution platform. The quantile options enable you to enter the probability associated with the quantile of the fitted distribution to calculate specification limits. The second method allows you to enter a K-Sigma multiplier value that is used to calculate the specification limits. Using the previous example, select the option “Enter Sigma Multiplier to Calculate Quantile Spec Limits”. Click the Calculate Spec Limits button to calculate the spec limits for 3-Sigma. Notice that the 3-Sigma values are filled in for LSL (0.042954) and USL (0.069226). Click OK.

This new method allows easier scripting of the Quantiles option. Previously, you had to fit a distribution, get the quantiles, and pass them back to the platform (requires two calls to Distribution). Now you can do this in a single JSL call to Distribution.

Summary

Ability to create histograms without the spec limits.
Ability to create a distribution report without capability analysis.
The Process Capability Distribution column property is used for capability analysis.
Easier scripting of the Quantiles option for fitted distributions.
Normal and non-normal capability analysis uses similar dialogs and output.
Consistency between platforms.

Did you miss any of the previous posts on the modernized Distribution platform? You can find all posts in the series here.

Modernizing the Distribution platform – Options for fitted distributions

tonya_mauldin — Mon, 02 Dec 2019 12:57:22 GMT

The Distribution platform is one of the most widely used platforms in JMP. This platform is not only used for testing which distribution fits your data, it is used for data exploration, capability analysis, and so much more. This platform has been around since the first version of JMP and it was time for a modernization. JMP 15 has delivered the modernization of this commonly used platform.

My first blog post detailed the new fitters in JMP 15. The second blog post detailed comparing distribution fits. This blog post details the new options available for fitted distributions.

Diagnostic Plots

The diagnostic plot has been replaced with two new plots (QQ and PP plots). The quantile-quantile (QQ) plot shows the relationship between the observations and the quantiles obtained using the estimated parameters. The percentile-percentile (PP) plot shows the relationship between the empirical cumulative distribution function (CDF) and the fitted CDF obtained using the estimated parameters.

To see an example of these plots, open Braces.jmp found in the Quality Control sample data folder.

Open("$SAMPLE_DATA/Quality Control/Braces.jmp");

Select Analyze->Distribution. Specify # defects as Y, Columns and click OK.

From the red triangle next to # defects, select Discrete Fit->Fit Negative Binomial. From the red triangle next to Fitted Negative Binomial Distribution, choose QQ Plot and PP Plot.

For the QQ Plot, most observations fall close to the diagonal line. The last point is questionable. If the two sets (fitted negative binomial quantiles and the # defects) come from the same distribution, the points should fall approximately along the reference line. The PP plot has a similar interpretation. The points in the PP plot fall approximately along the reference line indicating that the Negative Binomial fitted CDF and the empirical CDF come from a population with the same distribution.

Profilers

Two Profilers have been added. The Distribution Profiler is a Prediction Profiler of the cumulative distribution function (CDF). The Quantile Profiler is a prediction profiler of the quantile function.

Using the previous example, choose Distribution Profiler and Quantile Profiler from the red triangle menu next to Fitted Negative Binomial Distribution.

If the data follow the fitted negative binomial distribution, the Distribution Profiler shows that the probability of getting 27 defects or fewer is .896934 (~90%). The Quantile Profiler gives the inverse of the Distribution Profiler. For this example, the probability that we get seven or fewer defects is .1 or 10%. The Profilers are interactive. You can move the slider around to determine what the probability would be for various numbers of defects.

Save Columns

Two new save column features have been added. Save Distribution Formula saves a column to the data table that contains the cumulative distribution function (CDF) formula computed using the estimated parameter values. Save Simulation Formula saves a column to the data table that contains a formula that generates simulated values using the estimated parameters. This column can be used in the Simulate utility.

Goodness of Fit

The goodness of fit test has been standardized to the Anderson-Darling for continuous fits and the Pearson Chi-Squared test for discrete fits. The Pearson Chi-Square test has improved bin creations. JMP now satisfies the rule of thumb that there are at least five expected observations in each bin.

For the previous example, click on the red triangle next to Fitted Negative Binomial Distribution and choose Goodness of Fit.

The p-value is 0.4971. At any meaningful alpha level, we fail to reject the null hypothesis that the data are from the Negative Binomial distribution.

Summary

QQ Plots
PP Plots
Distribution Profiler
Quantile Profiler
Save Distribution Formula
Save Simulation Formula
Anderson-Darling
Pearson Chi-Square improved binning

These three blog posts only scratch the surface of the JMP 15 new features in the Distribution platform. Look for my next blog post in which I detail new capability options in the Distribution platform.

What’s the point? Just pointing at something tells you about what you are pointing at

John_Sall — Thu, 14 Nov 2019 14:02:38 GMT

In magical tales like the Harry Potter series, wizards can just point at something with their wand, speak an incantation, and magic happens. We do something very similar when we point the cursor at something on our screens, and without even an incantation or a click, magic happens. That is our world of tooltips or hover windows.

Identify points

Graphs of points in JMP are incredibly valuable in seeing the distribution of points over two dimensions. But when you see a point that is far apart from the rest of the data – an outlier – you want to identify that point. By hovering the cursor over the point, a hover window appears showing which point that is, plus you can click to select the point in the table to investigate further. From the time JMP was first released, hovering your cursor over a point in a graph has brought up a small text window identifying a point. In the first version of JMP, it just showed the row number. But we’ve continued to add information and functionality with subsequent versions: the row number and any column values given the “Label” attribute; the values corresponding to the coordinates; the ability to “pin” the hover label to the graph persistently; label columns that allow images in the hover label window; an option to use the images as markers. But with JMP 15 there is much more.

What’s new

JMP 15 introduces far more powerful hover windows – graphlets. They are built in to a number of platforms where points represent many values and can be used in a customization setting for almost any graph, providing drill-down features.

Process Screening

In Process Screening, you can see the health of many processes in one Process Performance Plot. In Figure 1, one process is healthy, the others are incapable, and Thickness 4 is unstable as well. To find out more about the instability, hover over that point and a hover window appears, showing a small graph of the process. Hovering is a lot easier than selecting the point and requesting a quick graph or control chart of that process.

Figure 1: Process Screening

Functional Data Explorer

The Functional Data Explorer characterizes many trajectories using functional principal components. Figure 2 is a score plot of the temperature functions of 16 US weather stations. To determine what’s going on with the score space, points in the four areas have their hover windows pinned. The first component on the X axis is basically how extreme the winter is, while the second component, with less than 5% of the variation, involves the shape of the seasons.

Figure 2: Functional Data Explorer

Multivariate Control Chart

In the new Model Driven Multivariate Control Chart (Figure 3), the fact that a point is far out doesn’t explain which process is involved in it being extreme. By hovering over the point, the individual differences are shown – in this case, Process 6 is implicated. Figure 3: Model Driven Multivariate Control Chart

Custom graphlets

Figure 4 shows the mean violent crime rate for each state over years, but it doesn’t show the trajectory. To see that, I made a separate Graph Builder showing the violent rate by year. Next I copied script to clipboard, then went to the original graph and right-clicked on Paste Graphlet. Now when I hover over a point, it makes the graph of that state by year, in this case detailing Virginia. Figure 4: Custom graphlets

Further customization

Once you make a graphlet, you can customize its features in a Hover Label Editor. The Graphlet panel seen in Figure 5 shows the script that is run on the subset represented in the point. Gridlet provides control over which elements are shown in the hover window. Textlet allows you to customize the text, including the markups to stylize the text.

Figure 5: Graphlet Hover Label Editor

Conclusion

Good software can make things so easy that it seems like all you have to do is point a wand (or cursor) at something, and magic happens.

If you want to see JMP 15 and JMP Pro 15 in action, the JMP R&D leadership team and I gave this presentation at Discovery Summit Tucson 2019.

Modernizing the Distribution platform – Comparing distributional fits in JMP 15

tonya_mauldin — Wed, 20 Nov 2019 13:20:39 GMT

Distribution is one of the most widely used platforms in JMP, and it has been around since the first version of JMP. It's useful for all sorts of things, including data exploration, capability analysis – and, of course, testing which distribution fits your data. But it was time to modernize Distribution, and that is what you see in JMP 15, the latest version of our software.

My previous post details the new fitters available in Distribution in JMP 15. This post details how those fits can be compared.

Comparing Distributions

After fits have been selected, a Compare Distributions report is added. In previous versions of JMP, you only got a Compare Distributions report when using the All option under Continuous Fit. Now, you get this report after selecting any distribution. As distributions are added to the report, they are also added to the Compare Distributions report. Additionally, AICc Weight (corrected and weighted Akaike's Information Criterion) and BIC (Bayesian Information Criterion) has been added to this report. These statistics were already present in other parts of JMP such as Model Comparison, and they have now been added to the Distribution platform.

You may notice that there are no longer check boxes next to the distribution names that allow you to choose which distributions to show. In JMP 15, all selected fits are shown. You can remove a fit by double-clicking on the name in the Distribution column. If you are a fan of the check boxes, they will be returning in JMP 15.1.

The Compare Distributions report has been moved so that it is always directly underneath the histogram. This gave us the ability to remove the legend that previously appeared beneath the histogram and move that information to the Compare Distributions report.

CDF Plot

In addition to the Compare Distributions report and the overlaid histograms (available in previous version of JMP), CDF plots can be used to compare distributional fits. Adding the CDF plot to the report superimposes the CDFs for the fitted distributions onto the empirical CDF.

To see an example of these new features, open Washers.jmp found in the Quality Control sample data folder. The # defective column contains counts of the number of defective items in each lot.

Open("$SAMPLE_DATA/Quality Control/Washers.jmp");

Select Analyze->Distribution. Specify # defective as Y, Columns and click OK.

Since we are dealing with count data, let’s fit the Poisson distribution and the ZI Poisson distribution to see how they compare. Hold down the Alt key (Option key on the mac) and click on the red triangle. Check the box for CDF Plot, Fit Poisson, and Fit ZI Poisson. Click OK.

The histogram shows us that the blue line more closely follows the data. In the Compare Distributions report, we see that the blue line is for the ZI Poisson distribution. The Compare Distributions report is initially sorted by AICc in ascending order. The ZI Poisson distribution has a smaller AICc indicating a better fit. The CDF plot shows that the blue line (ZI Poisson) more closely follows the red line (empirical CDF) indicating a better fit.

Summary

Cleaner look by moving the histogram legend into the Compare Distributions report.
Cleaner look by removing the distribution check boxes from Compare Distributions.
AICc weight and BIC have been added to Compare Distributions.
Overlaid CDFs for the fitted distributions in the CDF plot.

These two blog posts only scratch the surface of the JMP 15 new features in the distribution platform. Look for my next blog post in which I detail the new options for fitted distributions.

Modernizing the Distribution platform – New fitters in JMP 15

tonya_mauldin — Wed, 13 Nov 2019 16:08:35 GMT

The Distribution platform is one of the most widely used platforms in JMP. This platform is not only used for testing which distribution fits your data, it is used for data exploration, capability analysis, and so much more. Distribution has been around since the first version of JMP, and it was time for some changes. With JMP 15 comes the modernization of this commonly used platform.

New Fitters

One of the biggest changes in the Distribution platform is related to fitting distributions. The fitters have been modernized to use the same code as the Generalized Regression platform. Sharing code between JMP platforms in this way ensures consistency and requires less maintenance going forward. New Distribution fits were added while we were updating this code.

You can now fit a Negative Binomial distribution (equivalent to the Gamma-Poisson distribution), a Cauchy distribution, a ZI (zero-inflated) Poisson distribution, and a ZI (zero-inflated) Negative Binomial distribution. The Johnson fits are now a single command that fits the best-fitting distribution from the Johnson system of distributions. This Johnson fitting method is the same method (quantile matching) used in the Process Capability platform. This method is more stable and faster than maximum likelihood.

To see an example of the new Johnson fitter, open Airline Delays.jmp found in the sample data folder.

Open("$SAMPLE_DATA/Airline Delays.jmp");

Select Analyze->Distribution. Specify Arrival Delay as Y, Columns and click OK.

Click on the red triangle next to Arrival Delay and select Continuous Fit->Fit Johnson. The best fit from the family of Johnson distributions for this data is the Johnson Su Distribution.

Prior to JMP 15, the only way to control the Kernel Std for the Nonparametric Density was via a slider using the interface. Now, there is the option to specify a specific value for the bandwidth parameter. JMP 15 also gives you the ability to script the bandwidth parameter.

Before JMP 15, no standard errors were given for the parameter estimates for the SHASH distribution. This has been added in JMP 15 to be consistent with the other distributional fits.

Summary

New fitters that match the Generalized Regression platform.
Cauchy Distribution.
Negative Binomial Distribution.
ZI Poisson Distribution.
ZI Negative Binomial Distribution.
Single Johnson fitter that shows the best Johnson fit.
Ability to enter a value for the smooth curve bandwidth parameter (also the ability to script this parameter).
Parameter estimate standard errors for the SHASH distribution.

This blog post only scratches the surface of the JMP 15 new features in the Distribution platform. Look for my next blog post in which I detail the new features for comparing distributional fits.

Graph Builder contour plots in JMP 15

danschikore — Thu, 21 Nov 2019 17:53:32 GMT

In a recent post, gave an overview of the changes for Graph Builder in JMP 15. Several of the updates are related to the contour element, so I thought it would be a good time to take a more in-depth look at the features of the contour element, new and old.

Contour plots are a common technique for visualizing functional data. Contours may take on different names for the various functions that they represent -- isotherms for curves of constant temperature, isobars for curves of constant pressure, and many more.

When data is collected or measured at discrete locations, it can be a challenge to make observations from the points themselves. In the examples below, consider a density function computed from the set of points. The density values could be mapped to color in a 2D plot, or a third axis for a 3D view, but these solutions do not really make it easier to answer some questions about the function. For example, where do the maxima / minima occur? Where is the function changing faster or slower? Contour plots are often shown using a series of curves computed for different isolevels. As shown in the lower-right image, the curves will be closer together at regions of high gradient, and farther apart at areas of low gradient.


Scatterplot of data	Scatterplot with density function shown by color

3D Scatterplot with density as function of two variables	Contour plot of density function

In JMP, contours are used in several platforms, including Bivariate, Contour Plot, Contour Profiler, Graph Builder, and others. The Graph Builder contour element has several new features in JMP 15, including new display options as well as new contour techniques. This blog post includes a grand tour of the six different forms of contours that are available, comparing their use for different situations.

Contours in 1D

The scatterplot is a direct way to visualize continuous data and is often the default way to get a visual representation of continuous data. When the points are very dense, it becomes difficult to characterize the distribution of the points, and especially to make comparisons between multiple groups. In this example using the Titanic sample data in JMP, the fares paid by the survivors of the tragedy are compared to the fares paid by those that perished. Some observations can be made from the scatterplots, such as that the outliers in the survivor group paid roughly twice what the others (in either group) paid.

To compare these two groups further, some additional analysis is necessary to characterize the groups. One widely used technique for summarizing 1D data is the boxplot, shown below. Boxplots will be useful for comparison with the new contour techniques. There are several variations on the visual representation for a boxplot, but they are all based on quartiles. The box extends from the first quartile to the third quartile, so it contains 50% of the observations. A line is drawn at the median, giving additional detail. There are several variations on how to draw the whiskers – in this example they extend to the outermost point that lies within 1.5x the interquartile range from the box. Observations outside of this region are optionally shown as outliers.

The Graph Builder Contour element uses a probability density function for a given kernel bandwidth to illustrate the shape of a 1D point distribution. The default view for 1D contours is the Violin plot, which plots the density function with symmetry about the axis. The magnitude of the curves is determined by a scaling option, with options for Equal Area, Equal Width, or Weighted Area. The different scaling options are helpful for comparing the functions in different ways. For example, Equal Width may be useful if your primary need is to compare the locations of the maximum densities for each population. Weighted Area scaling would be the best option if you want the relative areas of the contours to reflect the population sample sizes.

New in JMP 15 is the ability to control the kernel bandwidth via a smoothness parameter. The parameter a normalized value from [-1,1], with the default value of 0 leading to a default computed bandwidth based on the data characteristics. Negative smoothness values result in less smoothing, and positive smoothness values result in more smoothing, within computed limits.

Also new in JMP 15 is a the HDR (Highest Density Region) plot, a new option for 1D contours. The HDR plot uses the same density function as the violin, but applies concepts similar to the boxplot to reduce the display to a few quantitative measures. A black line is drawn at the maximum point in the density function, and dark rectangles are drawn to represent the 50% highest cumulative probability. Lighter rectangles are drawn to represent the 99% cumulative probability, and any points outside of the 99% probability are considered outliers. The same smoothness parameter is available for the HDR plot and Violin plot.

The fill color or line color can be change from the legend, and colors are also used when adding columns to the Overlay or Color roles. The X Grouping and Y Grouping roles provide additional layout possibilities.

One distinction between box plots and HDR plots is that the 50% and 99% probability regions in an HDR plot are not necessarily continuous. When the underlying data is not unimodal, this can give a better picture of the distribution of the data. Compared to the Violin representation, the mode lines and outliers give a more quantitative, but summarized, presentation of the underlying density function.

Contours in 2D

Scatterplots for two continuous variables have some of the same disadvantages as scatterplots in 1D. When the data is dense, it can be difficult to make observations about the distribution of the points. One common practice is to use transparency for the points in a 2D scatterplot. The regions with the most overstriking will appear darker in the resulting plot.

2D Density contours

The Graph Builder contour element uses a bivariate nonparametric density function to extract density contours and directly display regions of various densities. On the left is the default view using four levels, with a darker color at high density fading to white at lower densities, in addition to transparency that fades the contours at low density. Transparency is very useful for contours when used with the Overlay role. In the image on the right, the K Means Cluster platform was used to segment the data into 11 groups based on four factors. The transparent contours show the overlap of these groups for two of the four factors.

In JMP 15, Graph Builder adds several new options and styling attributes for 2D density contours, including line/fill options (from the property panel), density gradient and transparency gradient options (from the legend), and smoothness. As in the 1D case, there is a single normalized smoothness parameter ranging from [-1,1]. The default values of 0 represents a default kernel bandwidth in each dimension, negative values decrease the smoothing, and positive values increase the smoothing.

2D HDR contours

The concepts of HDR (Highest Density Region) plots for 2D density functions are directly analogous to the 1D case. Using the 2D bivariate density function, a marker (*) is drawn at the point of highest probability, a dark contour is drawn for the area of 50% cumulative probability, and a lighter contour is drawn at 99% cumulative probability. Points outside of the 99% region are optionally shown as outliers.

2D Bagplot

Another new representation in JMP 15 is the Bagplot. The Bagplot was designed as a 2D analog to the 1D Box Plot. In 2D the “order” of the points does not have a clear definition. The Bagplot was designed based on Tukey's location depth (Tukey 1975), which determines an integer depth value for each point, which results in a natural nesting of depth hulls from the outer convex hull (depth 0) to the inner hull at maximum depth.

Given these concepts, the bagplot consists of four elements. A marker (*) is shown at the median of the highest depth points. The two depth hulls that separate the points in half are interpolated to determine a 50% probability region, which is shown in the dark color. This is referred to as the bag. The bag is inflated by 3x relative to the median to determine the loop, which is not shown. The convex hull of all points inside the loop is shown in a lighter color and is called the fence. All points outside the loop are considered outliers.

The bagplot is a very close 2D analog of the 1D box plot, and similar distinctions can be made when comparing the bag plot to the 2D HDR plot. The bagplot is better suited to summarizing the distribution of unimodal data, while the 2D HDR plot has an advantage when giving a summary view of multimodal data.

Value Contours

When an additional color role is specified in addition to two continuous columns for X and Y, the Graph Builder contour element shows value contours, like the Contour Plot platform.

Contours in this case are computed using a piecewise linear interpolation of the input data based on a Delaunay triangulation. The triangulation is not shown within the Graph Builder platform – in the example below the triangulation was added via JSL.

JMP 15 adds new options for showing the boundary, filled contours, and line contours. For value contours, smoothing is done by interpolating the triangular data to a fixed grid and applying gaussian smoothing to the grid. The smoothed contours are then clipped to the boundary of the triangulation.

A Delaunay triangulation always results in a convex region. If the points are not uniformly distributed, this can lead to interpolation between points that are spread over large areas. The Alpha Shape is a subset of the Delaunay triangulation that filters triangles that are very large or very long and skinny. The filter looks at the radius of the circumscribing circle for each triangle, removing those that are larger than some limit. The resulting subset could have holes or multiple components.

Summary

The new HDR methods provide a quantitative view of density-based contours for both one and two continuous variables. All Graph Builder density contours now support a smoothing parameter, and the bagplot extends boxplot concepts to 2D. Alpha shapes control the shape of the domain for triangulation-based contours. All of the contour types have new controls for drawing lines or filled contours, and 2D density contours also have additional control of the color palette and transparency.

If you would like to reproduce these plots in JMP 15, download the file JMP15Contour.jmpapp from recent JMP Discovery talk in Tucson and it will launch the data that you can experiment with:

References

H. Edelsbrunner, D. Kirkpatrick, R. Seidel, On the shape of a set of points in the plane, IEEE Transactions on Information Theory, Vol. 29, Issue 4 (July 1983), 551-559.

Rob J. Hyndman, Computing and Graphing Highest Density Regions, The American Statistician, Vol. 50, No. 2 (May 1996), 120-126.

Peter J. Rousseeux, Ida Ruts, John W. Tukey, The Bagplot: A Bivariate Boxplot, The American Statistician, Vol. 53 (1999), 382-387.

J.W. Tukey, Mathematics and the Picturing of Data, Proceedings of the International Congress of Mathematicians, Vancouver, 1975, 523-531.

Improvements to the Area element in Graph Builder

paul_vezzetti — Wed, 04 Dec 2019 18:31:18 GMT

The Area element in Graph Builder is rewritten in JMP 15 to share functionality with the Line element. The Area and Line elements now support the same features, and moving forward, new features added to one element will be picked up by the other element.

Anything that can be created using the Area element can also be created using the Line element and vice versa. However, each element has a different set of initial defaults that are more appropriate for the given type. More about these differences later.

To demonstrate the changes in JMP 15, I downloaded some data from FluView. FluView is from the Centers for Disease Control and Prevention. The website tracks confirmed cases of positive tests for various strains of influenza. There is more information about FluView available from the CDC website.

The data is categorized by week of the year and by four age groups: 0-4 years, 5-24 years, 25-64 years and 65+ years. It contains counts of confirmed cases for each week and age group for six different virus strains. FluView presents the data in stacked bar charts like this one:

FluView from Centers for Disease Control

Stacked bar charts and area charts are typically used to convey the same types of information, so this blog post uses the data to demonstrate features of the Area element. To make the graphs easier to read, the data has been additionally filtered to just 2019 rather than using the full 52 weeks from 2018-2019.

Connection type

Area now supports all the same connection types as Line which are Line, Curve, Step, Centered Step, Horizontal and Vertical. Here are some examples showing Line, Curve and Step for the first 15 weeks of 2019:

Area Connection Type Examples

The Connection setting is available in the options on the Area element. Row Order is also supported for data with implicit row ordering.

Hover tips

Hover tips are now available on the Area element when over any data point. The tip shows the value for the series as well as the overall stacked value at that data point.

Area Hover Tips

Error bars

When the Area is using the Overlaid or Range area styles, error bars can be displayed just as they would on a Line element.

Error Bars

Displaying lines and changing fills

By default in JMP 15, the lines are displayed with a width of 0 on the Area element. This is a change from JMP 14, which displayed the lines with a width of 2 in the same color and transparency as the filled area. This had the effect of making the lines not visible. The lines could be made visible by changing the line properties through the legend.

In the image below from JMP 14, the line color for each series has been changed to a different color and a Line Width of 3 in order to make the lines visible against the fills. Note that both the line and fill properties are shown together in the legend.

JMP 14 Line and Fill Properties

Unfortunately, the transparency setting controlled both the line and fill transparency. This made it impossible to have transparent fills with fully saturated lines. In the example below, the transparency setting on the legend has been reduced to 0.2. Both the lines and the fills have this transparency applied. The lines are slightly visible because they are drawn on top of the fills, but they are not fully saturated.

JMP 14 Area with Transparency

In JMP 15, the fills and lines are separated and controlled through individual legends. By default, the line legends are not shown since the line width is 0. To display the lines and the line legends, the line legends must first be enabled in the Legend Settings dialog.

JMP 15 Legend Settings

In the Legend Settings dialog, enable the legends for the lines. These are the unselected legends below the fill legends. The preview does not display the lines because the line width is still set to 0. We adjust the line width once the legend is visible.

In this screenshot, the line legends are shown enabled and highlighted in blue:

Turning on Line Legends

Now that the legends are shown, use the right-click context menu on each legend to set a line width. The line is initially not visible because the line width is zero. Setting a non-zero width makes the line visible in the legend and on the Area element.

Setting Line Width

Now the outline of the fill area can be seen and is independent from the properties of the fill.

JMP 15 Lines and Fills

To make the lines more prominent and the fills more subtle, we can adjust the fill transparency (or color) without affecting the line. In this case below, the fill transparency has been reduced from 0.75 to 0.2 as was done previously with the JMP 14 example.

The difference from JMP 14 is very apparent, as the lines are fully saturated while the fills are highly transparent.

Lines with Transparent Fills

Area options in Line Element

The Area element isn’t the only one that got new features in JMP 15. The Line element also picked up features from the Area element. The Line now has an option for stacking the values.

In the example below, the Stack option is enabled on the Line element. The hover tip includes the stacked value as well as the value for the series.

Stacked Lines

By specifying a fill on the Line element, we can recreate what we had with the Area element.

Area using the Line Element

Other Improvements

The Area element now handles hidden row states. The JMP 14 Area element did not take hidden row states into account when displaying the data. The behavior for hidden rows now matches the existing behavior of the Line element.

Support for Missing Values has been added, and the behavior of Missing Factors matches the Line element as well. The Missing Value connection is displayed through the line display. Therefore, the lines need to be visible to see some of the styles such as Connect Faded and Connect Dashed.

Differences

The visual perception of an area chart is typically different than a line chart. When viewing an area, the relative amounts of the fill areas are the primary focus. The viewer is processing the individual fill amounts as well as the cumulative values.

For these reasons, the Area element puts more emphasis on the cumulative values and the filled areas. The Area element uses stacking by default. The fills are highly saturated, and the lines are deemphasized.

In a typical line chart, the focus tends to be more on the changes across the x-axis. The viewer is looking for trends, minima/maxima, or sudden changes.

The line element doesn’t stack values or include any fills by default. When applying fills, the fills are more transparent to give the lines higher visual priority.

Why have both? Which one should I use?

While the Area element and Line element are functionally equivalent now, both are still available to provide easier access to the expected default configuration. This minimizes the number of steps required to achieve the desired graph when you know what you are trying to create.

However, sometimes during data exploration, you may start with one element but then realize that features of the other element would enhance your data story. For example, you may start with a line element, add an overlay variable, and then decide that you want to stack the overlay values to see the cumulative amounts. Previously, this would have required switching the element type and potentially losing some customizations. Now, you can easily stack without switching.

Celebrate 30 years of JMP with a 30% discount on training and books

Larry_LaRusso — Wed, 06 Nov 2019 15:57:58 GMT

Get a discount on books and training to celebrate 30 years of JMP.

Thirty years ago (Oct. 5, 1989, to be exact), we shipped JMP Version 1. It was a big day for us, and for our customers. That first offering of JMP brought interactive data visualization and analysis directly to the desktop, provided "a graph for every statistic," and proved to be a giant leap forward for data explorers around the world.

And for the past 30 years, our users have been doing incredible things with JMP software ... changing the world along the way. You all are, in fact, the real heroes of JMP's success story.

That's why we're hoping you'll help us celebrate this milestone (and expand your JMP knowledge even further) with a 30% discount on JMP training and books. Simply purchase any JMP book (between Nov. 6 and Nov. 13, 2019) and/or any JMP course (between Nov. 6 and Dec. 31, 2019) and enjoy the discount -- a small gift for our amazing users.

The fine print

30% Discount on JMP e-books

Purchase any JMP e-book through the SAS Store and receive a 30% discount at checkout.
Offer is good for any purchase made between Nov. 6 and Nov. 13, 2019.
Enter Promotion Code 30JMP at checkout.
View JMP e-Books

30% Discount on JMP Training

Purchase any JMP course between Nov. 6 and Dec. 31, 2019, and receive a 30% discount at checkout.
Offer is good for any public, live web, or e-learning course or the JMP Learning Subscription.
(Note: The discount is valid only for courses in the U.S. and DOES NOT apply to on-site courses.)
Enter Promo Code 30JMP at checkout.
View the JMP Learning Path and JMP Courses

Questions? Feel free to leave me a comment.

Enjoy!

Standing on the shoulders of giants: The improved D-optimal design construction procedure in JMP 15

calking — Wed, 06 Nov 2019 18:59:07 GMT

I wondered if it was it possible to create an optimal design for any run size?

When I was first learning design of experiments in graduate school, we covered the usual suspects: factorial and fractional factorial designs with two-level factors. These designs are popular precisely because they allow for main effects to be independent of one another; the focus of fractional factorials being to save on runs at the expense of aliasing with interaction effects. While learning about resolution and how to choose one fractional factorial over another, I had come to accept that the number of runs was always going to be some power of 2. When we briefly touched on Plackett-Burman designs, I saw that the run size could be generalized to a multiple of 4. But beyond that, it seemed like a good design wasn’t possible.

Flash forward several years and I’m playing around with the Custom Design platform in JMP for generating experimental designs for clients at Sandia Labs. That was the first time I’d seen where you could specify any run size you liked. Now I tended to stick with run sizes that were, preferably, multiples of the factor level combinations as it just felt right. But it was around that time that a question started to nag me; was it possible to create an optimal design for any run size?

Now, to clarify, the answer is technically yes as JMP, much like other software, uses an algorithm to search for the optimal design given a starting one. But that’s not exactly what I was asking. What I wanted to know was if it was possible to directly construct one, much like the factorial and fractional factorial designs.

For those unaware, factorial and fractional factorial designs are essentially special types of matrices known as Hadamard matrices. What makes these matrices special is that, when you take the transpose of the matrix and multiply it by the original matrix (e.g., X^t*X), you end up with a diagonal matrix. This is precisely why they are used as factorial and fractional factorial designs; the diagonal matrix is equivalent to the covariance matrix and so represents independent main effects.

When I learned this fact, it made me want to explore my question even more. But it wasn’t until my first year at JMP that I was able to do so. I started as I usually do; playing around with the mathematics, slowly gaining insight into possible construction methods. I discussed my work with my colleagues, and , and Ryan sent me an article by Dennis Lin on supersaturated designs that he thought might pique my interest due to some similar themes. While it didn’t directly relate to my work, one of the references did. And it’s there that everything changed.

The Joy of Rediscovery

It felt like I had stumbled upon a hidden door that, upon opening it, revealed a strange and wonderful new world (the similarity to Lucy’s experience in C.S. Lewis’s The Lion, The Witch, and The Wardrobe, while unintended, is quite appropriate). Reading that reference led to another, which led to another, and on and on and on until I had compiled quite the collection of literature. What was so fascinating about these papers? They all discussed direct construction methods for generating D-optimal designs for two-level factors for virtually any run size!!

Now, to avoid hyperbole, I should say that even though such methods exist, they do not exist for every single possible run size and some of the methods are quite complex. But the fact that direct methods did exist excited me! And, as it turns out, for a good majority of practical cases, the construction procedures were quite simple and involved adding or subtracting runs from the well known factorial and fractional factorial designs (it turns out this was what my personal research was going towards; per usual, I find myself often rediscovering the wheel…oh well). In fact, I was surprised I had never heard of some of these methods during my time in graduate school. Then again, it might be something you would learn only after you’ve mastered factorial and fractional factorial designs, which require a significant time to master on their own.

I shared these methods with my colleagues, who immediately saw the impact such methods could have in creating optimal designs. Imagine that, instead of having to implement a search algorithm to hopefully find a D-optimal design, you could build one from scratch and have it mathematically guaranteed to be D-optimal. Now imagine you could do this for a wide range of run sizes; again, not all, but a vast majority that is sufficient to cover practical situations. Well guess what? You don’t have to imagine it anymore!!

The Punchline

That’s right folks. In JMP 15, we now implement the simple construction procedures for generating D-optimal designs right into the Custom Design platform! “That’s nice,” you might say, “but how does this change things?” Well, yes, it’s not as visible or glamorous as, say, creating a new supersaturated design construction platform. But perhaps the following demonstration may help illustrate:

Both of these designs involve the same number of factors (31) and run sizes (34) and both were created to be D-optimal. Note that the run size is not a multiple of 4 (or you can double check for yourself, I won’t judge…just stare condescendingly). In the version on the left, it took nearly 30 seconds for the algorithm to find what it hopes is the D-optimal design. In the version on the right, it created a clearly more D-efficient design almost instantaneously. Now, at this point, I’d like to mention that I come from an accelerated testing background. I say this because, in that context, we’re used to cranking things up to extreme levels. So how about we crank up this here example to an extreme?

In this case, the number of factors is 531 and the number of runs is 1059 (an experimenter’s dream and worst nightmare all in one). In the version on the left, it took nearly three minutes for the algorithm to find what it hopes is the D-optimal design. For the version on the right, it took nearly 3 seconds to report the more D-efficient design. I say report instead of create because the creation step is about as long as it was for the previous example; the rest of the time is spent laying out the design and getting the diagnostics in the final report, just because it’s so massive. If that doesn’t prove to you how amazing this improvement is, I don’t know what will!!

Business as Usual

As you’ll be seeing in several upcoming blog posts, JMP 15 is brimming with new features. And if you attended the JMP Discovery Summit conference in Tucson, you no doubt got a hands-on look. While you might be bouncing off the walls with excitement (or perhaps you’re more subdued in your excitement, no judgement here), for us developers and testers it’s business as usual: making sure our customer’s statistics needs are always being met and even surprising them with what they never knew they needed. So enjoy this new version and all it has to offer and definitely make sure to let us know your feedback!!

Using regular expressions in Recode in JMP 15

ErnestPasour — Mon, 25 Nov 2019 16:06:58 GMT

In a previous blog post, I showed how to use the Extract Segment feature in Recode in JMP 15 to extract a number embedded in a string. Recode also makes it easy to use a regular expression to perform the same task. Depending on the form of the data, it may be easier to use one tool versus the other.

Step 1. We’ll start off the same way, by invoking Recode on the Observation ID column.

Step 2. From the red triangle menu, chose Replace String…

Step 3. We are best off sticking to simple regular expressions. In this case, we are searching for the letters “WK”, followed by one or more digits. \d matches a digit, and the plus sign means “1 or more”.

Step 4. Looking at our replacement values, we can tell that the regular expression found WK1, but replaced it with nothing. We can use \1 as the replacement text to replace the match with the first capture group. We don’t want the entire string, only the numeric portion, so we add parentheses to create a capture group.

Step 5. Now, we have the number (ex. 1) in the place where the matched string was before (ex. “WK1”). However, all we want is the digit. There’s a new checkbox in Recode that replaces the entire string with the replacement value.

Step 6. Now we have our data trimmed down to just the WK number as a string. We can continue by parsing the string into a number as before. If we save the result as a column formula, we’ll get the following formula.

The resulting column looks like this:

Regular expressions are a valuable tool to have in your toolbox. With JMP 15, you can combine regular expressions with other tools in Recode to perform more advanced data preparation.

Using the Extract Segment tool in the Recode platform

ErnestPasour — Tue, 19 Nov 2019 14:31:24 GMT

Recode has always been a great tool for cleaning up messy data, but it’s also useful for extracting data embedded in text columns. JMP 15 adds the flexible Extract Segment tool, and the platform has been enhanced to generate semantic scripts in many cases so that the script or formula can be executed against new data.

For example, let’s take the sample data set Drosophila Aging Distances. Say we want to get the WK number out of the Observation ID column and put it in a separate numeric column. We notice that the field has a regular pattern where the WK number is the third field.

Step 1. Recode the Observation ID column

Step 2. From the red triangle menu, choose Advanced->Extract Segment...

Note that the Text to Capture section shows that there is only one word in our sample text. However, for this purpose, our words are delimited by the underscore character. We’ll add the underscore as a delimiter.

Once we add the delimiter, we can see that the first sample contains four words. I have used the slider to move the selection to the third word. You can see in the Preview area that we are now capturing the WK portion of the string. We use the OK button to finalize this step in our process.

Step 3. Now that we have isolated our desired data, let’s use Extract Segment again to peel off everything but the numeric portion. If we remove all delimiters, then each character is treated like a word. Notice that I have selected the third word. However, I have also switched the end to point to the end of the string (a negative number indicates we are counting from the end of the string). We only have a single digit in this case, but if we had multiple we would be capturing from the third word to the end of the string.

Step 4. We now have the digits as strings. We can go to the red triangle menu again and choose Parse as Numbers to retrieve the numbers.

Step 5. Save to a new column or a script. Note in this script that it captured the three steps performed and can be run against new data. The Recode Column portion is very similar to the Recode command that would be written out to a formula column.

The resulting column looks like this:

Get Drosophila Aging Distances, give it try and let me know what you think.

Structural Equation Modeling: The arrival of a powerful new framework in JMP Pro 15

LauraCS — Mon, 25 Nov 2019 14:20:01 GMT

Ever wished for an intuitive and super flexible approach for specifying and fitting models? Structural equation modeling (SEM) is a framework that provides just that! Think factor analysis merged with regression, together with simultaneous estimation of numerous effects and awesome graphs.

In this post, I discuss the unique advantages of using JMP Pro for fitting structural equations models. If you’re new to SEM, be sure to check out this post, otherwise, let’s dive in!

Ease and Speed

SEM in JMP Pro is built with an intuitive user interface that enables a speedy workflow. We pride ourselves on sitting at the intersection of user-friendly and efficient; fast and replicable as coding-based tools, yet visual and simple. Importantly for SEM enthusiasts, you can do all your data cleaning, data manipulation, and model fitting, in one place.

Model Comparisons

Best practices of SEM involve fitting multiple, competing models to data. Our platform facilitates this task by providing all the information you need from multiple models in one place. You can select which models to look at, or you can see them all at once.

Local Data Filter + SEM = Greatness

Questions such as “How do these models fit for this subset of the sample?” are important and often lead to invaluable insights, but seasoned SEM analysts have learned the pain such questions invoke. The pain comes from recognizing the amount of time needed to create new data sets and re-run all the models. In JMP Pro, the Local Data Filter will transform your pain into eagerness! In a matter of seconds, you can select the level of a variable that denotes the subset of your sample and all your models update instantly! You’ll have answers to all those insightful questions within seconds. If you’re in academia, think how much time that’ll save you in responding to reviewers’ comments. This is one of our favorite features and we can’t wait for you to try it.

High-Quality Path Diagrams

Nobody likes to spend their time building SEM path diagrams in PowerPoint, Vizio, or any other non-statistical software. Similarly, nobody likes to use the subpar diagrams that are often produced by automated algorithms. The SEM platform in JMP Pro has high-quality, presentation-ready, appealing-to-the-eye, path diagrams. These diagrams are interactive* so you can arrange them just the way you want them (although we get them to you in pretty good shape from the get-go). Save them as a picture file and forget about busy work.

Ongoing Error Checking

“The matrix is not positive definite” is one of numerous warning messages that SEM analysts are used to encountering when fitting complex models to data. Many times, issues are due to mistakes that are easily introduced (even by expert users) due to the incredible flexibility afforded by SEM. Unfortunately, these issues are only found after running the faulty model. The SEM platform in JMP Pro performs ongoing checks on the specification of the model to save users time and effort in tracking potential issues. The Details box gives you immediate information on the specified model and our Status tab* will let you know if you’re on the right path, the expert path, or a risky path.

These are my five favorite reasons for using the SEM platform in JMP Pro 15. Give it a try and let us know what your favorites are! Maybe you’ll enjoy visualizing your models’ residuals, sorting results based on parameter estimates or model fit indices, or the model shortcuts. There’s lots to discover and much more to come in the new SEM platform!

*Coming in JMP Pro 15.1: model estimates on path diagram, interactive path diagram, AICc weights for model comparison, and Status tab with checks for identification rules and expanded model details.

How to launch JMP the first time on Mac OS 10.14 Mojave and 10.15 Catalina

dave_matthews1 — Fri, 25 Oct 2019 14:16:35 GMT

This is part 2 of 2: In part one, we covered the download and installation of JMP software on Mac OS 10.14 Mojave and 10.15 Catalina. In the second part, here, we cover launching JMP for the first time.

Apple Mac OS prefers you open applications that come through their Apple App Store. This is especially true in Mac OS 10.14 (Mojave) and 10.15 (Catalina). While Mac OS allows you to install applications you download outside of the App Store, it will give errors and/or warnings about the installation and make you follow a certain process to complete the installation.

The Mac OS application process is designed to protect you from becoming the victim of malware software or accidentally installing something that is not good for your system. However, the process can be confusing and unsettling to new Mac OS users or users who have not encountered the messages before. Fortunately, the process to allow to run is a straightforward one.

Open JMP in System Preferences

If you attempt to open the JMP app after you installed it, Mac OS may stop you and require permission to run JMP.

To successfully launch JMP do the following. Open System Preferences (Apple menu > System Preferences) and select Security & Privacy. On the General tab, you will see a message near the bottom of the window stating that “JMP was blocked from use because it is not from an identified developer”.

Next to this message will be a button, “Open Anyway”. Click it.

You will be asked one more time to confirm your request.

Click Open. and JMP should launch immediately.

This creates an exception for the JMP app. You will be able to open JMP in the future without having to repeat this process.

I hope you have found this article helpful. If you still need assistance, you can search the JMP User Community or request assistance from JMP Technical Support.

How to install JMP on Mac OS 10.14 Mojave and 10.15 Catalina

dave_matthews1 — Fri, 25 Oct 2019 14:17:33 GMT

Part 1 of 2: In this part, we cover the download and installation of JMP software. In the second part, we cover launching JMP for the first time.

Apple Mac OS prefers you install applications through their Apple App Store. This is especially true in Mac OS 10.14 (Mojave) and 10.15 (Catalina). While Mac OS allows you to install applications you download outside of the App Store, it will give errors and/or warnings about the installation and make you follow a certain process to complete the installation.

Downloading JMP

To download JMP, please follow the instructions that are provided. The JMP trial version provides these instructions in the email with your download link and serial number. The single-user and annual license versions provide these instructions in the Software Order Email, sent to your SAS Site Representative. If you are not the Site Representative, please contact that person about how to access the software at your organization.

Installing JMP

Once you have downloaded JMP, you will need to double-click the PKG file to launch JMP.

In Mac OS 10.14 Mojave and 10.15 Catalina, the Mac OS software security utility, Gatekeeper, will force you to verify that you want to install the software. Software that is installed outside of the Apple App Store will typically have to follow this process.

You will typically encounter one of two messages:

The first message states it is unable to scan the JMP software for malicious code.

The other message states the JMP software is from an unidentified developer.

To allow JMP to install, you will need to follow Apple’s process.

To do this, open System Preferences (Apple menu > System Preferences) and select Security & Privacy. On the General tab, you will see a message near the bottom of the window stating that “JMP can’t be opened because Apple cannot check it for malicious software” or “JMP was blocked from use because it is not from an identified developer”.

Next to this message will be a button, “Open Anyway”. Click that button.

You will be asked one more time to confirm your request.

Click Open and the JMP installation process should begin immediately.

Part 2, How to Launch JMP the first time on Mac OS 10.14 Mojave and 10.15 Catalina is covered here.

I hope you have found this article helpful. If you still need assistance, you can search the JMP User Community or request assistance from JMP Technical Support.

Functional data analysis for photovoltaic devices: A ray of hope for more clean energy

gail_massari — Thu, 24 Oct 2019 15:36:19 GMT

The strength of the JMP Pro Functional Data Explorer platform (FDE) is identifying and extracting key features of functional processes for use in further modeling.This is done by fitting a functional model to the data using a B-spline, P-spline, or Fourier basis model; performing a functional principal components analysis on the functional model; saving the PCA results; and using the results for feature extraction and analysis in another modeling platform.

Hadley Myers and Don McCormack , two JMP Systems Engineers, recently contributed to a paper in IEEE describing how they used JMP Pro to better understand how photovoltaic (PV) devices are affected by changes in deposition conditions and the influence this has on device quality. What are photovoltaic devices? They are systems that convert sunlight directly to electricity using photovoltaic cells made of semiconductor materials.

Hadley and Don leveraged the use of Functional Data Explorer (FDE) because of its usefulness for analyzing curves that vary across a continuum, such as those recorded during current voltage (JV) and External Quantum Efficiency (EQE) measurements. NOTE: EQE curves show the ratio of the number of charge carriers collected by the solar cell to the number of photons of a given wavelength.

Diagram of a typical crystalline silicon (c-Si) cell. (Courtesy of Solar Energy Industries Association)

One of their goals was to apply functional data analysis, and specifically to test JMP Pro FDE as a new tool to gain insight into the complex interplay of the various factors.

They followed this sequence:

Characterize the photovoltaic cells by measuring current-density–voltage (JV) characteristics under AM1.5 illumination for the different back contact sets at the various maximum growth temperatures during deposition.
Remove device outliers.
Model performance for remaining devices, enforcing statistical heredity to limit the total number of models.
Perform functional data analysis on the entire JV and EQE curves using the FDE platform in JMP Pro.
Fit models to output of FDE to identify key factors contributing to the functional responses that had been missed in Step 3.

What they found:

FDA helped isolate and quantify the effect of deposition temperature on the EQE of the devices. Specifically, temperature was seen to be the only factor that significantly influenced variation. The curve shapes showed that higher temperatures appeared in general to result in increased bandgaps of the absorber and decreased collection in the entirety of the absorber layer.
Higher growth temperature seems to improve the final diode, but might adversely affect semiconductor material quality.
Effect of back-contact deposition conditions (%O₂ in sputtering chamber and film thickness) on the JV-response that had been missed when only considering the usual device parameters.

The study increased the understanding of photovoltaic devices, uncovered new and previously undetected effects, and is just one example of how to use JMP Pro FDE in the pursuit of better (and hopefully environmentally sound) devices and systems.

The techniques used in this study of Copper Indium Gallium Selenide (CIGS) photovoltaic devices can be applied to the analysis of other types of PV structures, including those constructed using Crystalline Silicon (c-Si).

Get the full details on the study and its results in "Functional Data Analysis of Electrical Measurements on Thin-Film Photovoltaic Devices" by Hadley Franklin Myer, Don McCormack, Muhammad Saifullah, Tobias Bertram, Rutger Schlatmann and Christian A. Kaufmann.

By the way, photovoltaic solar arrays at SAS world headquarters in Cary, NC, generate 3.8 million kilowatt-hours of clean, renewable energy annually.

JMP 15の新機能を使って、ラグビーの得点状況を可視化

naohiro_masu — Fri, 18 Oct 2019 04:45:31 GMT

日本は、丁度ラグビー日本代表の活躍に熱狂しているところです。現在、ラグビーワールドカップ2019が日本で開催されておりますが、日本は予選リーグで4戦全勝し、初めてベスト8に進むことができました。

そんな中、丁度JMP 15がリリースしました。データ分析家の血が騒いだのは言うまでもありません。予選リーグのデータを集め、いくつかのデータを可視化してみることにしました。

ラグビーでは、トライ、コンバージョン、ペナルティキック、ドロップゴールなどさまざまな方法で点数を取ることができ、時間内に点数を積み上げていき、最後に点数が上回ったチームの勝利になります。そこで、2チームの対戦で、時間ごとにどのように得点を積み上げていったかをJMPを使って可視化してみることにしました。そんな時、JMP 15の新機能が役に立ったのです！

次のグラフは、ウェールズ（WALES）vs フィジー（FIJI）について、時間（横軸）ごとの総得点（縦軸）可視化したものになります。

前半の30分ぐらいまでは、フィジーがリードしていましたが、その後ウェールズが逆転し、さらに逆転、逆転で、最終的には、ある程度差をつけてウェールズが勝利したことがわかります。

このグラフは、JMP 15のグラフビルダーで、折れ線グラフの下側を塗りつぶすことにより描くことができます。

グラフビルダーで折れ線を描くと、JMP 15では次のようなオプションを設定でき、「塗りつぶし」のオプションを[下に塗りつぶす] を指定します。ちなみに、この折れ線の「接続」オプションは、[ステップ] に設定して、階段状のグラフにしています。

以下は、JMPを英語表示したときのオプションになり、Fillを [Fill Below] に指定します。

下のグラフは、ベスト8をかけて死闘を繰り広げた　日本(Japan) vs スコットランド(Scotland) の試合です。

日本は先制点を許すも、すぐに逆転し、最後には追い付かれそうになったが、逃げ切った様子が可視化できています。

他の対戦についてのグラフや、他の可視化した結果は、JMP Public に公開しています。データフィルタの機能を使って、予選の組ごとや国ごとに絞って分析結果を見ることができますので、是非ご確認ください。

Optimizing film topology using target functions in functional data analysis

MikeD_Anderson — Tue, 12 Nov 2019 12:13:03 GMT

Earlier in this blog series, I mentioned that functional data analysis (FDA) could be used with experimental design. It’s an interesting idea to consider. What if you wanted to optimize the shape of a curve? Or, understand how to control the periodicity of a response? That’s what I want to discuss in this last entry in this series. To do it, we’re going to use “The Way Back Machine” and go back to 2010 to that year's JMP ENBIS Challenge.

The 2010 ENBIS ChallengeWhat's Golden Curve optimization or analysis?

Let me take you back to 2010. Ian Cox ( Worldwide Technical Director) submitted a simulator for an LPCVD (Low-Pressure Chemical Vapor Deposition) process to the European Network for Business and Industrial Statistics (ENBIS) annual conference for their ENBIS Challenge. The challenge was to find a process recipe that would maximize the simulated profit and reduce defectively on the simulated parts. If you want to play with the simulator, it’s still up.

Part of the challenge was that you had to consider the cost of development in the profitability calculations and only had a certain amount of total material to play withal (for production and development). I stumbled into the simulator a couple of years after the fact while I was looking for an example I could use for some training purposes. It turns out Ian did his homework when developing the simulator, and it does a nice job of replicating the output of the LPCVD process with realistic defect rates, etc. It’s still an informative tool for demonstrating why DOE is the only way to do process development. We’re going to use it here, although maybe somewhat outside its original scope.

Golden Curves Ahead

One of the main benefits of FDA is that it makes it possible to use curves (not individual data points) as factors in a model. One could also argue that we would want to look at how process factors (time, temperature, etc.) affect an observed curve. FDA can do that, too. The next logical step in that line of thinking is: I know the curve I’m after (either through first principles or process constraints). I want to optimize my process conditions to fit the curve I’ve provided as closely as possible. This idea is an extension of what we would do in JMP with the Profiler and the Maximize Desirability routine. In the world of FDA, that is called a “Golden Curve” optimization. This Golden Curve Analysis is what we are going to focus on today.

The Process

First off, I’m not going to give a long-winded treatise of LPCVD. It’s not that important to this discussion. But, just so you’re not flying blind, here’s the elevator speech version. The process is used to put films and coatings on stuff. The films can be functional (corrosion protection) or decorative (like finishes on higher-end bathroom fixtures). In LPCVD, you put a substrate into a chamber (this could be a silicon wafer, a drill bit, or the faucet in your kitchen). The chamber is then evacuated (the low-pressure part) and heated up to some temperature. Then reactive gasses are introduced into the chamber at a controlled rate. The gasses react and attach themselves to the surfaces of the parts. Some times radio frequencies are used to break chemical bonds in the gases to make the process more efficient.

ENBIS Simulator DOE Interface

In the DOE simulator, we have control over a number of the process parameters and gas flow rates. The simulator gives data measured from 49 sites across the substrate arranged as a center point and three concentric rings of measurements. This measurement pattern is relatively standard in the industry. It’s not my favorite by a long shot, but it shows up a lot.

The Problem

There are a couple of problems we have to deal with in the LPCVD simulator. First is the issue of getting the target thickness and film parameters correct. If you didn't care about the profile of the film, this could be done using a standard DOE and focusing on some summary statistic like the average or median film thickness. The second issue is that, because of the geometry of the chamber and the way the gases flow, the film is not uniform. It usually has a toroidal feature in the middle of the substrate. This second problem is why we need to use FDA to deal with the analysis.

The Analysis

Let me skip ahead to the bit where I'm analyzing the DOE results. Doing a graphical analysis of the DOE wafers, we can see that the non-uniformity of the wafer has radial symmetry. We can use this to our advantage and convert the (x,y) coordinates provided by the simulation to calculate the radius for each point using the Pythagorean Theorem. This transform reduces the dimensionality of the data set and makes FDA more straightforward.

Before I do the FDA in the Functional Data Explorer platform in JMP Pro, I need to add a set of data to the DOE analysis results. This data set will be the target curve. It contains the target parameters for a large number of radii between the measured data points. I'm providing a large number of data points to force the optimization algorithm to look for a solution that matches the target curve (in this case, a uniform value across the substrate) closely.

Now, I can do FDA. The setup in JMP Pro is below. I’m focusing on just the thickness in this example. Note that I’m using radius as my “X, Input” value. Recall in an earlier post in this series that the definition of time is somewhat fluid in FDA, so this is OK. Note that my DOE parameters are supplementary variables. Once I’ve hit OK in the launch dialog, I need to do one thing differently than I usually would with FDE. I need to click on the “Load” button in the Target Function part of the FDE interface. Doing this will bring up a dialog that lists the function IDs. I select the one associated with the target function (added when I created it) and click “OK.”

Now, with the preliminaries over, we can create the functional form of the experimental data and pass it to a modeling platform. There’s a good tutorial on how do the mechanics of creating the functional form here. JMP Pro 15 handles all the mechanics of passing the functional forms and the supplementary variables to the Fit Model Platform behind the scenes. So, you only need to select Functional DOE analysis from under the red triangle menu for the model. This will give you access to the Generalized Regression personality in Fit Model and pull up the Profiler automatically from within FDE. (Note that all the reports and data are available here on JMP Public.)

When you’ve provided a target function, the Profiler has two additional rows of graphs. These show the difference between the function produced by the current settings and the target function and the integrated error between the target function and the current function. When you select Maximize Desirability from the Profiler red triangle menu, JMP attempts to minimize the integrated difference between the target and the displayed function. In our case, JMP searches for the setting that produces the flattest thickness curve in the process. From there, you’d need to read off the values in the Profiler and do a couple of verification runs to check the results. (Always verify your DOE predictions, people!!).

Let's Practice What We Preach

So, looking at the FDOE Profiler, we can see that the parameters that really drive this process are Deposition Time, Deposition Temperature, and Deposition Pressure. We are going to presume that we need to run the process at 300℃, deposit for 824 seconds at 261 mTorr. However, there is a great adage involving assuming and what it turns oneself into ... so we're going to do some validation runs with those settings.

Before I do the runs though, I'm going to do a little simulation using the Profiler's simulation capabilities to understand the probable range of the data. I'm going to do that to act as a stand-in for a calculated confidence interval. It's not perfect, but it will at least give me an idea of what the output data should look like for this situation if my model is correct. Here's the FDOE output with the simulator turned on and some best guess values for the process parameters.

And, here are the results of the validation runs with some historical data from the ENBIS challenge for comparison. Note that this control chart is looking at the average film thickness over the entire wafer. If you look at the Profiler above, you can see that the edge and outer region of the wafer have lower film thickness. They also have the majority of the data points in the wafer (look at the sampling plan at the top of the article). This will combine to skew the wafer mean values downward (one of the major reasons I'm not a fan of this sampling plan). But, that little quirk aside, the wafers are now all within spec and the variation lot to lot has also dropped.

Conclusion

Golden Curve Analysis is probably my favorite application of FDA. Trying to optimize a process with a known spatial correlation while ignoring the effects of that correlation has always bugged me. However, it’s quite a challenge to do the analysis other ways -- particularly within the framework of a DOE.

Image and wafer analysis using functional data analysis

MikeD_Anderson — Mon, 04 Nov 2019 23:05:59 GMT

If you're just now tuning in to the series, welcome! This is the third entry in what was supposed to be a simple little blog article to deal with some writer's block on another project. Things escalated quickly…

Here are the other entries in the series:

And now on to the subject of this post.

An Interesting Question

How do you use functional data analysis for images or wafer maps?Almost as soon as the Functional Data Explorer (FDE) came out, I started getting questions about using FDE for images or wafer maps. Conceptually, it made sense to me that this would be useful. And, I was pretty sure the math could handle it. However, Functional Data Explorer in JMP Pro wasn't set up to handle the data structure. So, I found a way around the problem.

This article is an explanation of the thought process and mechanics of how I got image/wafer map data to work in FDE. The trick is to change your point of view. Let me show you what I mean.

Some Motivation

Before we go too far down this path, let's take a second to consider why we would want to analyze wafer maps or images. People are visual creatures. We can easily see patterns in data that (whether they are there or not).

This ability is particularly useful in manufacturing and defect analysis. With wafer maps, it is easy to pick out patterns caused by different processes by eye (like the ones shown above). Having an algorithm that can do something similar that presents the information in a manner that is easy to use in subsequent analyses is priceless. Similarly with image analysis or other data that are presented in the form of heat maps, being able to translate visual information into an analytically friendly format opens many new avenues for analysis.

A Thought Exercise

The biggest problem with getting FDA to work is deciding what the "function" is in the data set. And that's where most people start. In JMP Pro, they will usually sit, staring at the "ID, Function" field trying to figure out what to put in there before they do anything else -- and that is a horrible plan, especially when you're trying to do FDA on images, wafer maps, etc.

Let me give some context to where I'm going with this. The question that got me thinking about this problem was around something called a wafer map. Here's a picture of one:

In the semiconductor industry, we tend to think of wafer data in one of two ways: a collection of individual cells (called die) or as images, where the wafer is the thing we analyze. From a process perspective, that first method doesn't make sense. Since changes to a process affect the wafer globally (it's tough to adjust process conditions to alter one specific die), each wafer is more like an image. And therein lies our problem. How do we encode the spatial information into a functional data analysis?

Now, what if we stop worrying about defining the function for a second. What if we change our perspective a little and determine what the "time" variable is? If we do that, the path forward becomes more apparent by the process of elimination. It turns out each wafer is a snapshot in time. The same goes for pictures -- just like a movie -- each frame of a movie film represents an instant in time. If someone is doing FDA on image data, they are generally looking at the same field of view collected over time. With images, "time" can also be thought of as slices moving through a volume or changes in energies, exposure times, etc. So "time" is a stand-in for an independent variable (i.e., what you alter in your experiment, etc.).

In our semiconductor data set, the wafers all to have a value attached to them called the Wafer ID. These IDs are ordered chronologically. In the 3D scatterplot above, we can use the z-axis (up) as moving forward in time. So, if the wafers (or images) are the "time" variable, what does that leave for us to consider the "function?" The answer is the die or pixels. If we consider the (x,y) coordinate of each die or pixel as an ID of sorts, we can create a unique identifier that we can use as the function ID. Using that logic, we can plot the average response for different families of functions (die) and look at the average number of defects over a series of wafers (the "time" domain). We can see that the edge has a much larger problem with defects than the center of the wafer (to which every semiconductor engineer goes, "Yup, now tell me something I don't know.").

Changing Perspective

It turns out that considering a die or pixel location as the functional ID has some positive and negative consequences. On the plus side, we can do FDA on images and image like structures (wafer maps). On the downside: Within the FDA process, we lose our ability to interpret the results visually. We get it back on the other side, but we're flying blind while we're building the functional forms and have to rely on the numbers.

So, how do we change our perspective? The first step, getting the data into JMP Pro, particularly for images, has been covered by in a couple of Community posts (Image Analyzer presentation and add-in) a few years back. You need a stack of data that has your response of interest, and the (x,y) coordinate pair for each pixel, die, etc. After this, we treat the images (wafer maps) as a time variable and the pixel ID (die ID) as the function ID.

Likewise, we run FDA as usual, but in this case, we're going to ignore most of the graphics -- the Score Plot and the Bar Graph with the FPC rankings being the exception. Looking at the Functional PCA table, I want to see a high score in the cumulative column (indicating that we're capturing most of the information in the data set). I'm also looking for anything unusual in the Score Plot (see the previous entry in the series).

The last step in the FDA process is exporting the FPC values into a new data table with the pixel/die ID column. From there, we can break up the ID column into the x and y components for visualization using column utilities. I am glossing over these steps, but the data sets are available on JMP Public, complete with the formula columns I used to convert the Die/Pixel coordinates into Function IDs and back again.

Example 1: Wafer Map

This first example is looking at defect rates in a semiconductor wafer map. The data set is part of the JMP sample data. Below is an interactive graphic of the FPC's from the study (the other steps are above). Note that FDE was able to tease out two different defect signatures in the data. The first signature is easy to see in FPC1. The second is in FPC 2 and is really weak. It's a more radial signature. FPC2 contains a lot of the random shot noise in the data. The upside of this analysis is that if we want to see what causes the signature represented by FPC1, we can just use FPC in a model/root cause analysis exercise.

Example 2: Image

This image set is a sample data set from the ImageJ package. It is sintered alumina particles, with each image in the original set being taken from a different depth in the thickness of the sample using confocal microscopy. The reconstructed image using the FPCs is shown below. What's interesting about this image is that the FPCs appear to emphasize features at different depths in the sample.

For fun, I took that information and did a 3D reconstruction of the sample using the first three FPCs:

Conclusion

So, it is possible to do some image analysis with FDA in JMP Pro. The key is making sure you consider what the Function and the "Time" domains are in the data set. More broadly, this is really a critical consideration with FDA. It can cause very different results or be the difference between a successful analysis and something nonsensical.

Next week, we're going to cover what the heck you can do with the information you get from analysis like this. We're going to marry a process DOE to functional data analysis. I hope you come back for the next episode!

Graphical tour of JMP 15 Graph Builder enhancements

XanGregg — Tue, 26 Nov 2019 19:06:45 GMT

Here's a quick tour of many of the enhancements made to Graph Builder in JMP 15. Some of these will be explored in depth in upcoming blog posts.

	Heatmaps now support categorical coloring variables. It works best if all the values in a cell have the same category level. But blending can useful when there is some overlap, like the blue-green overlap in this heatmap of the familiar Iris data.
	Previously, box plots could be colored individually. Now, they can be colored systematically with data by putting a variable in the Color role. Read more about box plots in color.
	Shaded error intervals can be enabled directly in the Line and the Points elements, as an alternative of error bars, without the need to layer in a separate Area element.
	The Histogram element can show a choice of count or percentage on the response (or Y) axis.
	The Histogram element has three new drawing modes: frequency polygon, smooth kernel density (shown here) and shadowgram.
	With many histograms in a single frame, the Histogram element has an overlap feature that can be used to create a "ridgeline" plot.
	The Line and Area elements have had their options aligned to match each other. The Area element now has Line features like connection style (step is shown) and missing value treatment. The Line element now has Area features like stacking and filling.
	The Line of Fit element has a new mode called Robust Cauchy, which assumes the residuals follow a Cauchy distribution instead of a Gaussian distribution. As a result of the fatter tails of the Cauchy distribution, the fit is less affected by outliers. Here it is with the classic Anscombe quartet data.
	The Time Series analytics in JMP has been updated for JMP 15, and the basics are available in Graph Builder as another new mode for the Line of Fit element; this allows for seasonal fits with forecast regions.
	Line of Fit and other elements have an "Adapt to Axis Scale" option in their red triangle menus (also new!). This causes the fit to apply to the transformed (e.g., log) values instead of the raw values. It is on by default.
	Several elements have a Save Formula command that will create a new column in the table with the results formula. For elements like Bar that summarize by category, this command enables you to extract the actual results of those summaries as formulas.
	Univariate density contours can be shown as Highest Density Region (HDR) plots in addition to violin plots. HDR shows the 50% and 99% density levels plus the mode and outliers. As a result, it combines the quick overview of a box plot with the multimodality detection of a violin plot.
	Bivariate density contours can be shown as "bag plots," meant to be an extension of box plots into two dimensions. The darker area shows the innermost 50% of the data. The lighter fill is equivalent to the tails or fenced region of a box plot.
	Bivariate density contours can be shown as 2-D HDR plots, useful for quick comparison of possibly-irregular bivariate distributions. The darker area shows the region with the highest density for 50% of the data, and the lighter area shows the 99% region. The regions need not be contiguous for bimodal data.
	The contour element provides a choice of showing lines, fills or both for the contours, and a choice of showing the outer boundary line. And the outer boundary is no longer restricted to being convex by using alpha shapes (as shown here). Read more about contour plots.
	For summarizing elements, such as bar charts, the single summary statistic for percent has been replaced with three more specific options to control the basis for the percent calculations. This is useful when you have multiple Xs and Ys and overlay levels. And this makes it easier to construct 100% stacked bars.
	Axis tick labels can now wrap over multiple lines. This axis feature is available in all JMP graphs, not just those within Graph Builder. This is a great way to avoid having rotated text.

JMP Live tips and tricks

John_Powell_JMP — Mon, 02 Dec 2019 00:08:30 GMT

JMP Live is a new product, but we’ve been using it internally for a while. In this blog I’d like to share some of the tips and tricks we’ve learned to help you get the most out of JMP Live.

Tip #1: Adding a hyperlink in your description

In some cases, it may be useful to provide a link to a web page from your JMP Live post. This can be added to your post description using a little bit of html markup. For example, a link to jmp.com can be added to your description in the Edit Details dialog as follows:

This graph was published using JMP 15. Get your free trial at <a href=”http://www.jmp.com”>jmp.com</a>

Tip #2: Keeping your data private

You may want to share the results of your analyses on JMP Live, but not your data. You can do so by disabling “Publish Data” in the “Select Reports” dialog when you publish:

When you do this, your audience will see the web-friendly user interface with collapsible sections and tables with tooltips, but they won’t be able to select or the interact with the graphs.

If you are publishing multiple reports at once, you can add images. So, another way to share the result of your analyses without sharing your data is to save your reports as images, then add them when publishing. You can save reports as images using File > Save As with ‘Save as type:’ PNG, JPEG, GIF, or TIFF. When publishing multiple reports, you would use “Add Image” in the “Configure Report” dialog under the report previews to choose an image to include. The “Add Image” button can be used multiple times.

Tip #3: Consider the size of devices that will display your report

When sharing JMP scripts with colleagues who have JMP, it’s often safe to assume that if it looks good on your computer, it will probably look good on theirs. If it doesn’t, they can easily resize graphs within JMP. When you publish reports to JMP Live, however, your colleagues could be viewing what you’ve published on mobile devices, which typically have smaller screens.

When the graphs don’t fit on their screen, they can usually zoom to fit, but then the titles and axes might become too small to read. So, just like when you’re preparing a presentation and you want to make sure the text in your reports can be read from someone at the back of the room, you should not create graphs so large that the titles and labels look tiny in comparison. Here’s an example of two graphs shrunk down to fit this blog post. The graph on the right started out smaller, so its text is easier to read than the left.

Tip #4: Why is my report not interactive?

JMP Live uses JMP’s Interactive HTML capability to convert JMP reports into interactive web pages. Not all the features in JMP are supported in Interactive HTML; but rather than producing no result, Interactive HTML creates a web page that is not interactive or partially interactive. To know what feature is not supported, look at JMP’s Log window for a short description of what is not interactive. Here’s an example of what you would see when publishing a 3D Scatterplot (not supported at the time this post was written):

Interactive HTML: Three-dimensional graphs are not interactive.

Tip #5: Downloading an image

When you publish multiple reports at once, you have the option of including an image. What if you want to download such an image when viewing it on JMP Live?

On Windows or Mac using Chrome, you can modify the link to the image as follows:

https://public.jmp.com/packages/JMP-Ball/js-p/PkcPtM3y_0CfhVZwDCjPz-8

to:

https://public.jmp.com/api/packages/JMP-Ball/js-p/PkcPtM3y_0CfhVZwDCjPz-8/indexPage?type=raw

Then hit “Enter” to navigate to this web address.

Right-click and choose "Save image as". Pick a name and location. Then Save.

Tip #6: Building a web page of posts

With a text editor or web page editor, you can create a web page with a title and a few JMP Live post links using “Copy Link”, “Embed Code(card)”, or “Embed Code(iframe)”:

If you don’t see all these options, it could be because the post isn’t enabled to Share with Everyone. This can be set in the Edit details dialog.

With “Copy Link” or “Embed Code(card)”, you will provide a link to your post that your audience can click on to see the report running in JMP Live. “Embed Code(iframe)” lets you embed the interactive report directly in your web page. For each option, some HTML code is copied to the clipboard so you can paste it into a web page editor or text editor.

I’ll only cover using a text editor since the steps using a web page editor would depend on the web page editor you are using.

Start off with the following in a text file:

<!DOCTYPE html>
<html><head><title >Some JMP Live Posts</title></head>
  <body>
    <h1> My Reports </h1>
       <!-- paste links below-->
  </body>
</html>

While viewing the post on JMP Live, click the desired sharing option.
Paste the clipboard contents into your text file under the line 
For “Copy Link”, you’ll need to add an anchor element and text you’d like to display, like the title of the report. For example:

<a href="https://public.jmp.com/packages/Bubble-Plot-of-Protein-by-Meal-Sized-by-/js-p/CFnRvnDdp4q9YkrYFlsx6"> Bubble Plot of Protein by Meal Sized by Fat Across Date ID Meal</a>

The highlighted areas are added around the link. Don’t forget the quote characters.

Repeat steps 2 and 3 for any other JMP Live Posts you’d like to add to your web page. Then Save the file with a .html or .htm file name extension so your operating system will recognize it as a web page.

The following shows the results of the three paste options for the same JMP Live post. This is only for demonstration; you would typically want to display different JMP Live posts with the same paste option.

Please see Mike Anderson’s post for more details on How to embed JMP Public or JMP Live content in a webpage or blog post.

Tip #7: Simulating a dashboard selection filter

Selection filters are a powerful addition to dashboards because they allow you to filter one graph with another. Unfortunately, the functionality is not yet supported by Interactive HTML. However, since JMP Live communicates with JMP running on a server, we can get close to the same functionality in JMP Live using “Exclude and Hide Rows” from the toolbar’s hamburger menu. Instead of building a dashboard with a selection filter, just combine windows with any two graphs like the Mosaic and Heatmap shown in the embedded example below.

To interact with the report like a selection filter:

Select the Mosaic cells you would like to include in the heatmap.
Using the hamburger menu:
a. Invert Row Selection.

b. Exclude and Hide Selected Rows. This will remove the selected rows from the Mosaic and Heatmap.

c. Use Include and Show all Rows to select another Mosaic cell and repeat 2.a. and 2.b.

Try these steps in this embedded example:

Tip #8: Drilling down in nested treemaps

This tip uses the hamburger menu similarly to Tip #7. Publish a treemap with multiple categorical variables as shown in the example below.

To drill down and focus on a part of a treemap in JMP Live:

Select the Treemap cells you would like to focus on. For example, click on “Passenger Class: 2” in the embedded example below.
Using the hamburger menu:
a. Invert Row Selection.

b. Exclude and Hide Selected Rows. This will “drill-down” to show just Passenger Class 2.

c. Use Include and Show all Rows to see the whole treemap again.

Tip #9: Customizing Thumbnails

Thumbnail images are generated when you post to JMP Live. They show up on cards throughout JMP Live. JMP uses the first graph in your report as the thumbnail, but this may not always be what you want to emphasize or use to catch the viewers’ attention. For example, if you build a dashboard, like in Tip #7, you might want to show more than one graph in the thumbnail.

To customize a thumbnail, you first need to capture an image or otherwise obtain an image to use as the thumbnail. Windows 10 includes an application called the “Snipping Tool” that allows you to capture a rectangular area of your screen and save it to an image file. On the Mac, you can use the shortcut (Command+Shift+4) to do the same. The thumbnail display is a bit wider than its height, so try to capture a rectangle that is approximately the same shape. It will be scaled to fit but clipped if it is too tall or wide.

To upload a new thumbnail in JMP Live, open the Edit Details dialog by clicking on the pen icon. Then click on the thumbnail, and you will be provided with a way to find and select the image file.

Cards for folders will use up to four of the child thumbnails to make up the folder thumbnail. You can override this with an image of your own in the same way you update a post’s thumbnail.

Tip #10: Making an interactive legend

This tip was inspired by a discussion on the JMP User Community. Legends are not interactive in Interactive HTML (at the time this post was written). But with this tip, you can build a graph that behaves like an interactive legend. That is, when you click on the item in the legend, the graphics in the main graph are selected.

All you need to do is create a bar chart with the same variable as the legend. Hide the real legends and combine windows to create a dashboard. I also cleared the “Graph Builder” title from the bar chart “Legend” so it wouldn’t be distracting.

Below is an interactive example. You can download the dashboard script from the post details on JMP Public.

Summary

I hope you will find some of these tips and tricks useful and discover some of your own as you work with JMP Live and JMP Public. Please share them here in the JMP User Community.

P.S. If you're hungry for more JMP Live tips and tricks, please see:

Column Switchers in JMP Live

John_Powell_JMP — Mon, 28 Oct 2019 16:42:30 GMT

The Column Switcher in JMP lets you quickly analyze different variables without having to recreate your analysis. When a variable is changed with the Column Switcher, the analyses are recomputed with the new variable.

JMP Live uses Interactive HTML to make JMP reports run interactively in a web browser for users who don’t have JMP. Interactive HTML provides several of the interactive capabilities in JMP, but it does not provide the ability to recalculate platforms. JMP Live enables this capability on the web by providing a mechanism for Interactive HTML reports to communicate with JMP running within JMP Live.

Below is a JMP Live Column Switcher hosted on JMP Public and embedded in the blog post so you can interact with it without having to go to JMP Public. If you do want to see it on JMP Public, this is the link:

Using Column Switchers on web pages can be very practical because they allow you to present many variables in the same space. When publishing content to the internet, conserving screen space may be important because your audience may be viewing the report on devices with smaller screens.

You could publish multiple reports instead, but if you want your audience to be able to change multiple variables in your report, the number of reports you would need to publish to cover every combination might be overwhelming. Many JMP platforms use multiple variables. For example, Graph Builder and Bubble Plot support variables for X, Y, color, size, and more.

Here’s a Graph Builder example using two column switchers:

Notice how the term “Column Switcher” was replaced with “Choose Y” and “Choose X” to provide the audience with some context.

Because the column switching is being accomplished by JMP and not Interactive HTML itself, we can even switch variables in reports not yet supported by Interactive HTML!

If you expect your audience to have a wide screen, you can use two Column Switchers in a way that helps them compare variables side by side. Click on the animated gif image to explore this in a separate window.

Column Switchers can be combined with a local data filter to make a powerful visualization. Here’s an example using the Food Journal sample data. I’ve added a Day column to make it easier to pick a specific day to focus on.

Need to change a variable in more than one role? No problem. The Column Switcher applied to this bubble plot changes both size and color role:

So, go ahead and start publishing to JMP Live with Column Switchers. They provide concise ways to explore the effects of many variables in one web page.

You can visit my Column Switcher gallery for more inspiration.

Why JMP Live?

Eric_Hill — Tue, 22 Oct 2019 15:56:09 GMT

JMP Live has arrived. Here's why we created it.

JMP is an incredibly powerful interactive statistical discovery tool. Many folks reading this have invested a great deal in both JMP software and in building JMP expertise in your organizations. But you likely also have people in your organization who don’t have JMP and don’t really need it, but who could benefit from seeing and interacting with JMP analyses created by your JMP experts.

Early Sharing Efforts

The JMP development team has added several features over the years to improve the ability of JMP users to share their results with colleagues who don’t have JMP. You can save JMP reports as PDF files or export them to PowerPoint. While helpful in some cases, what you get is a static image of the report. Interactivity is the hallmark of JMP. Without it, much of the value of JMP is lost.

Interactive HTML – Getting Closer

Then, along came the Interactive HTML development team. The goal of this team has been to allow the production of web-deliverable JMP content that retains as much interactivity as possible. Interactive HTML brings many JMP data exploration features to people who don’t have the software. For example, users can brush data points or bars in one graph and see where those selected points are in other linked graphs. They can also hover over points to see rich identifying information in hover tips.

Interactive HTML has been a welcome addition, but it also has limits. Important JMP features like Data Filter and Column Switcher cannot be fully supported; they require JMP to run to recompute the analysis. Also, Interactive HTML leaves an HTML file on your local machine, leaving you to figure out how to share it with others.

JMP Live – No Compromises

These opportunities to take Interactive HTML further opened the door for JMP Live. JMP Live is a web application running either on-premise at your company or in your company’s private cloud. JMP Live gives your users a portal to which they can publish interesting JMP reports, in Interactive HTML format, directly from JMP. At publish time, they can decide who should be able to see the report – everybody, a specific group of people, or just themselves. If a colleague bookmarks a link to their report, and then the original publisher decides to update the report, the colleague visiting that bookmarked link sees the updated version.

And now for the pièce de résistance: JMP Live includes a pool of JMP instances standing ready to regenerate an analysis when needed. So, when a user viewing Interactive HTML on JMP Live makes a change to a data filter, JMP Live uses one of those JMP instances to recalculate and update the statistics and graphs in the report, with appropriate points excluded.

Similarly, when the user makes a change to a Column Switcher, a JMP instance is used to regenerate the analysis with the new role assignments. JMP Live supercharges Interactive HTML by giving anyone with a web browser the power to make their own discoveries.

If you are interested in supercharging your JMP Interactive HTML, talk to us about bringing JMP Live to your organization.

All your base are belong to us (in JSL)

paul_vezzetti — Mon, 09 Dec 2019 14:32:49 GMT

Converting a value from one base or radix to another is often required with data in external formats such as sensor or network data. JMP has supported converting from hexadecimal to decimal and back using the Hex and Hex to Number functions. For example:

decimalToHex = Hex(1234, “integer”);
show(decimalToHex);
backToDecimal = HexToNumber(decimalToHex);
show(backToDecimal);

Output:

decimalToHex = "000004D2";
backToDecimal = 1234;

But what if you have binary or octal data?

Not just hex anymore

With JMP 15, these functions have been expanded to support any base from 2 to 36. By specifying the Base option on the Hex function, the decimal number will be converted to that base rather than the default base of 16.

decimalToBinary = Hex(1234, Base(2));
show(decimalToBinary);
decimalToOctal = Hex(1234, Base(8));
show(decimalToOctal);

Output:

decimalToBinary = "10011010010";
decimalToOctal = “2322”

Often with binary data, you want the values to be padded to byte boundaries for consistency. The new Pad To option allows that.

Padded to 2-bytes:

Hex(1234, Base(2), Pad To(16));

"0000010011010010"

Padded to 4-bytes:

Hex(1234, Base(2), Pad To(32));

"00000000000000000000010011010010"

Hex To Number provides the reverse functionality when the Base option is specified.

Convert from binary to decimal:

binaryToDec = Hex To Number("10011010010", Base(2));
show(binaryToDec);
octalToDec = Hex To Number("2322", Base(8));
show(octalToDec);

Output:

binaryToDec = 1234;
octalToDec = 1234;

QR Codes

QR codes (Quick Response codes) are two-dimensional bar codes that store small amounts of binary data that can be read using a scanner or smartphone. These codes were developed for inventory tracking, but they are often used now to encode location data for websites or applications. For example, here is a QR code for the JMP Discovery Summit in Tucson that links to the main web page of the conference.

QR Code for Discovery Summit Tucson 2019

The QR code consists of several fixed patterns (such as the three large square blocks for alignment) and the cell/content area for the data plus the error detection/correction information. QR codes come in a variety of sizes and have varying levels of error correction. Those details are not important for this exercise.

Let’s imagine that you wanted to encode the URL for Discovery Summit Tucson 2019 into the bits needed for a QR code.

We can start with the URL:

https://discoverysummit.jmp/en/2019/usa/home.html

QR codes support four types of encoding: Numeric, Alphanumeric, Byte and Kanji. Since our URL is a combination of letters, numbers and symbols, we can use the Alphanumeric encoding. Alphanumeric coding only supports uppercase letters, but that’s OK since the URL is case-insensitive.

Unfortunately, the QR encodings for characters aren’t just the ASCII values. If they were, we could just use the Hex function to get the encoding. Instead, we will write a short JSL function to do the conversion based on the table below.

Letter	Code	ASCII	Letter	Code	ASCII	Letter	Code	ASCII
0	0	48	F	15	70	U	30	85
1	1	49	G	16	71	V	31	86
2	2	50	H	17	72	W	32	87
3	3	51	I	18	73	X	33	88
4	4	52	J	19	74	Y	34	89
5	5	53	K	20	75	Z	35	90
6	6	54	L	21	76	Space	36	32
7	7	55	M	22	77	$	37	36
8	8	56	N	23	78	%	38	37
9	9	57	O	24	79	*	39	42
A	10	65	P	25	80	+	40	43
B	11	66	Q	26	81	-	41	45
C	12	67	R	27	82	.	42	46
D	13	68	S	28	83	/	43	47
E	14	69	T	29	84	:	44	58

Here is the JSL function to do the conversion from a character to the alphanumeric encoded value.

encodeAlpha = Function( {ch},
    // Convert character to its ASCII number value.
    hexVal = Hex To Number(Hex(ch));
    // Map the ASCII value to the QR encoded value.
    If(hexVal >= 48 & hexVal <= 57, hexVal - 48, // 0-9
       hexVal >= 65 & hexVal <= 90, hexVal - 55, // A-Z
       hexVal >= 97 & hexVal <= 122, hexVal - 87,// a-z
       hexVal == 32, 36,                         //Space
       hexVal >= 36 & hexVal <= 37, hexVal + 1,  // $ %
       hexVal >= 42 & hexVal <= 43, hexVal - 3,  // * +
       hexVal >= 45 & hexVal <= 47, hexVal - 4,  // - . /
       hexVal == 58, 44,                         // :
       0;
    );
);

The If statement maps the ASCII value to the encoded value using ranges. For example, the characters 0-9 are ASCII 48-57 and are encoded as 0-9. Therefore, those can be transformed from ASCII by simply subtracting 48. There are similar ranges for A-Z and a-z. The lowercase a-z are mapped to the values for the uppercase A-Z. The symbols jump around, so most are handled on an individual basis.

To save space, the QR code doesn’t directly write out the encoded value. It breaks the string into pairs of characters. For our URL, it would look like this:

ht tp s: // di sc ov er ys um mi t. jm p/ en /2 01 9/ us a/ ho me .h tm l

For each pair of characters, the encoded value for the first character is multiplied by 45 and added to the encoded value of the second character.

Our first pair is ‘ht’:

H = 17

T = 29

The result would be 17 * 45 + 29 = 794. This is then converted into an 11-bit binary value which is padded with zeros if necessary.

749 = 01011101101

If there is an odd number of characters, as in our case, the last character is just encoded and converted to a 6-bit binary value. In our case, we have a single ‘l’ at the end.

L = 21

21 = 010101

Converting numbers to binary and applying padding is exactly what the Hex function can do now. Let’s start by writing a function to calculate the encoded number for a pair of characters.

encodePair = Function ( {pair},
    char1 = Substr(pair, 1, 1);
    char2 = Substr(pair, 2, 1);
    encode1 = If(Length(pair) == 1, 0, encodeAlpha(char1));
    encode2 = If(Length(pair) == 1, encodeAlpha(char1),
                                    encodeAlpha(char2));
    encode1 * 45 + encode2;	
);

Now we can encode each pair from our URL to create the data for our QR code.

qrData = "";
For( i = 1, i <= Length( url ), i = i + 2,
    charPair = Substr( url, i, 2 );
    encodedValue = encodePair(charPair);
    asBinary = If(Length(charPair) == 1, 
           Hex(encodedValue, Base( 2 ), Pad To (6)),
           Hex(encodedValue, Base( 2 ), Pad To (11));
    );
    qrData = qrData || asBinary || " ";
);

show(qrData);

In this JSL, two characters are taken from the URL with each pass of the For loop and the alphanumeric encoding is calculated on that pair. The Hex function converts the number to its binary representation. The value is padded to either 11-bits or 6-bits depending on whether there are 1 or 2 characters. The binary value is appended to the previous value. We added a space between the pairs to make the output easier to read. The space wouldn’t be included in the actual QR data.

Here is the result for the URL:

qrData = "01100011010 10100110010 10100011000 11110111010 01001011011 10011111000 10001010111 01010010001 11000010110 10101011100 01111110000 10101000011 01101101101 10010010000 01010001101 11110010001 00000000001 00111000000 10101100010 00111101101 01100010101 01111101100 11101110011 10100101111 010101 ";

There is a lot more that goes into creating the final output for a QR code like the encoding mode, the character count indicator, and error correction coding. That’s more than we need to cover to show the new features. Hopefully, this helped explain some of the new Hex and Hex To Number features and how they can be applied.

Box plots - now in COLOR!

paul_vezzetti — Wed, 04 Dec 2019 19:31:27 GMT

JMP 15 adds new ways to control box plot colors and sizes. The color role is now fully supported in Graph Builder, giving you the ability to create more expressive box plots. Color provides additional visual information to guide the data exploration process.

The only way to add color to a box plot in JMP 14 was to use the Customize… option from the right-click context menu on the graph.

Customize Context Menu

The Customize Graph dialog contains an entry for each box and outlier in the graph. The line color and several other properties can be set as shown below. This process works well for highlighting a single box or a small set of boxes but can be tedious if there are lots of boxes. Also, mapping values to a continuous gradient is difficult, and if the data changes, the customizations can be incorrect or, in some cases, completely lost.

Customize Box Plot Line Color

JMP 15

With JMP 15, the Graph Builder box plot now supports the color role for both continuous and categorical variables. I downloaded some of my personal workout data to show off the new features.

The data includes information about when I exercised, the distance, pace, heart rate and weather. Here is a quick sample:Exercise Data

Continuous Color

In this first example below, the box plot is showing the Distance by the Day of Week for my Run activities. It shows, as I would have expected, that Sunday is typically when I do my longer runs. By adding Pace as the color, some other relationships are also visible. My longer Sunday runs are also my slowest. Monday tends to be a slower day as well probably because I’m still tired from Sunday. My fastest day is typically Thursday, which is usually when I will try to do some speed work.

Distance by Day of Week with Pace as ColorIn the example above, the Pace(min/mi) variable is summarized using Median. The statistic could be changed by selecting any of the standard JMP summary statistics from the Summary Statistic item in the right-click context menu on the color role.Selecting Summary Statistic for Color

Categorical Color

The color variable can also be categorical. One way this can be useful is for identifying boxes in a busy graph. Consider the example below from JMP 14. In this case, we are looking at Distance again, but now there is additional nesting on the y-axis by Month and by Day of Week.

Nested Box Plot without Color

Matching a box up with the appropriate Day of Week is challenging because there are so many boxes and the axis is thinned. Your eye must try to create a straight line from the box back to the axis to determine the Day of Week and then the Month.

By adding Day of Week to the color variable in JMP 15, it is now very easy to identify which day the box represents. It also makes it much easier to compare the same Day of Week across the months because we can focus on a single color and don’t have to go back to the axis for reference. For example, by knowing that Sunday is blue, and that the Months are in order from bottom to top, it’s easy to identify and compare the Sunday in June to the Sunday in July without looking at the axis simply by locating the sixth and seventh blue boxes.

Box Plot with Categorical Color

Box Width

In addition to color, the width of the boxes is now adjustable. A new option on the box plot called Width Proportion specifies the width of the box relative to the available width. By default, the option is set to 0, which means that the automatic (pre-JMP 15) width will be used. To change that, you can specify a value between 0 and 1.

Here is an example with very thin boxes with the Width Proportion equal to 0.1.

Width Proportion set to 0.1

Alternatively, for very large boxes with almost no space, the Width Proportion is set to 0.95.

Width Proportion set to 0.95

This is simply an aesthetic change, but it is useful if you are trying to achieve a certain “look” for your box plots.

Summary

This has been a quick look at some new features in JMP 15 for box plots. There are lots of new features in Graph Builder that will be the subject of additional blog posts. Let us know what you think of JMP 15!

Why is my Interactive HTML Profiler not interactive?

josh_markwordt — Mon, 18 Nov 2019 16:08:18 GMT

A common question that comes up with Interactive HTML output is "Why is this report just an image?" We attempt to provide feedback in the Log to give a high-level indication of why. Not familiar with the log? You can access it in Windows by going to View > Log (or pressing CTRL+SHIFT+L), or on the Mac by going to to Window > Log (CMD + 1). The Log can include script output when working with JSL as well as informational output such as the messages output when exporting Interactive HTML. We avoid messages that are so vague that they frustrate power users and statisticians and messages that so specific that they sound like gibberish to less technical users. Hopefully, the messages are informative, and are useful to provide to Technical Support if you are requesting something be implemented in interactive HTML or reporting a problem. Below, I decode the most common causes for Profilers to come out as static images.

Reason 1: Categorical response

I mentioned this above, but currently only continuous responses are supported. If you have tried to model a categorical response and save to Interactive HTML or publish to JMP Live, you will get the message "Interactive HTML: Profiler with categorical response <column name> is not interactive" in the log.

Reason 2: Unsupported JSL commands

The JMP Profiler evaluates a mathematical formula with multiple factors, varying one while keeping the others fixed to produce a curve for each factor. Internally, the formula is a JSL expression, such as what you find when you view the properties of a column with a formula. JSL is a large language with many commands and operations. Some are too programatically intensive to perform in the browser, which might perform badly.

Also, we are trying to implement the most widely used subset and will add features judiciously as we understand the demand for them from you, the user. In JMP 12, the same 50 or so commands implemented in Flash were ported: Add, Subtract, Multiply, Divide, Minus, Power, Root, Abs, Floor, Ceiling, Min, Max, Equals, NotEquals, GreaterThan, LessThan, GreaterThanOrEqualTo, LessThanOrEqualTo, Or, And, Not (negation), Exp, Log, Log10, Sine, Cosine, Tangent, SinH, CosH, TanH, ArcSine, ArcCosine, ArcTangent, ArcSinH, ArcCosH, OpArcTanH, If, Match, Choose, and a number of Financial calculations DDB, FinFV, FinIPMT, FinIRR, FinMIRR, FinNPER, FinNPV, FinPMT, FinPPMT, FinPV, FinRATE, FinSLN,FinSYD.

In JMP 15, six additional ones were added (Is Missing, Zero or Missing, IfMZ, Modulo, Ln and Log1P), mostly because they were heavily used by the Partition, Boosted Tree and Bootstrap Forest platforms. There is no point in doing the work to enable a new embedded platform if the formulas it typically produces are not supported yet due to unimplemented JSL commands, so these had to be implemented first. If a command we haven't gotten to is encountered, the log will display the message "Interactive HTML: Profiler for <column name> is not interactive because of unsupported operations."

Reason 3: Unsupported platforms

If you go to Preferences > Platforms in JMP, you will see more than 100 platforms listed, many of them with Profilers. Not counting the Prediction Profiler platform (found in Graph> Profiler), eight platforms are now supported. Again, we started with what we felt were the most widely used, and we want to hear from you if there is a platform you need that isn't supported. If your platform comes out static, look in the log and it will say "Interactive HTML: Embedded profilers in this platform are not interactive."

If you get this message in the log, there may be a workaround. For many platforms that have a Profiler, the red triangle menu will provide a "Save Columns" submenu with "Save Prediction Formula" as an option:Demonstrating the Save Prediction Formula option in the Partial Least Squares platform.

Using the Body Fat sample data set, run the "Partial Least Squares" script and add the Profiler from the red triangle menu for "NIPALS Fit with 7 Factors". If you were to export this, you would get a static image -- we haven't gotten to enhancing the Partial Least Squares platform yet to export the Profiler to Interactive HTML. However, if you go to that same red triangle menu and select Save Columns > Save Prediction Formula, it will create a new column "Pred Formula Percent body fat".Prediction column saved to the data table. By viewing the column's properties you can see the formula of the profiler.If you view the Column Properties, you can see there is nothing complex about this formula -- simple multiplication and addition. If you open up the Graph > Profiler menu and add this column for Y, you get a standalone Profiler that does export to Interactive HTML or JMP Live. Because of the number of factors, I chose to go to the red triangle menu and make use of the Arrange in Rows feature (found under the Appearance submenu) to make the Profiler wrap into two rows with a value of 7.

Reason 4: Advanced features

For individual embedded Profilers, there could be additional features that aren’t supported specific to that platform:

Non-mixture linear constraints
Random effects
Multiple response variables
Noise factors
Offset variables (this is specific to Generalized Linear Model)

If you encounter one of these, you will get a message in the log that should explain, for example: "Interactive HTML: Profiler with non-mixture linear constraint is not interactive."

We continue to improve the output of the very popular Profiler feature in Interactive HTML. Hopefully, this blog post helps to explain some of the cases that we haven't implemented yet.

We want to hear from you about the features and embedded platforms you would like to see -- leave me a comment!