Hi, I'm Mandy Chambers, and I'm a Principal Test Engineer in the JMP Development team. I want to talk to you today about navigating your data workflow. The Workflow Builder is new for JMP 17, and I think it can grant all your wishes for your data cleanup.
The Workflow Builder is the ability to record your JMP data prep and analysis workflows. It tracks all your changes that you make, cleaning up and reporting. It records your steps and you can use them over and over again. It allows you to save your data and your workflows, package them together nicely, and share them easily with others. It also is just going to save you lots and lots of time in your day.
This is a screenshot of the Workflow Builder. It's listed underneath the File menu system in JMP, File, New, New workflow. You need to open a data table or import data to begin and you will be prompted with a question that says, Do you want to record? When you say yes, then the screen on the right will show you that, like in this case, we opened Big Class and it will be recorded.
I'm going to get into this today and show you as much as I can. You can see here there's lots and lots of stuff happening. There are step settings, there are navigation buttons, there are images, and so much more. I probably can't hit everything today, but I'm going to do my best to try to show you. Let's get right to the demo and I'll get started.
This is a new utility. I do think it saves you time and it allows you to do clicks instead of coding. I think that's really cool and I think for a new JMP user, especially, this is going to really make their life easy.
Great job to JMP for building up ways that we can capture JSL. In 16, JMP introduced the Action Recorder, and now in 17, we give you the Workflow Builder along with even more JSL added to the Action Recorder. The Workflow Builder allows the JSL to be saved and the steps replayed as well as shared. A gain, no coding is really required unless you want to add that.
I'm going to get right into this by showing you a workflow. This is a workflow that I saved as a demo. Most of these examples, I have tried to use sample data so that you can go back yourself and maybe try some of these. I'm using the food journal data table from JMP. I'm recoding a column, I'm changing a column properties order, and then I ran a Text Explorer report.
You see the buttons here at the top, and this is a saved workflow. I'm going to just click this middle button which says to execute the workflow and run it so you can see how quickly it runs.
Notice a couple of things here really quickly. Over here on the right, you see these little images pop up. A ll that is simply, if I click on that, is it's a screenshot of each of the steps. It doesn't really do anything. It's a screenshot. It just shows you what's in there. Again, if I click this one, this is the column where I changed the column property. That's nice to have in case you don't remember what you did.
The little green check is the check that says the step executed properly. If it didn't execute properly, you would get a red X, so you would know something didn't work right.
Now, I'm going to recreate this for us. Looking at this Workflow Builder here, I had created my local data filter is why I changed the order. I wanted it to be in this order. I hung out on this late snack thing because I was looking and thinking, "Good heavens, if I had cappuccino, moko, and chocolate, and candy, and sugar at night, I would never go to sleep." I don't know about you, but anyway.
Let's close this up. That's this little reset button right here, closes everything in the workflow. Then I'm going to go up under the File menu system. I'm going to click New and I'm going to go down here and click New workflow. Here's my new workflow. Let's observe a couple of things here real quick before I get started.
This is the little record button, as I said, so we'll put that... I'm not going to push that. I'm going to show you how that executes. But you have places where you can get data from. The JMP Log history over here, I had reduced it down just because I didn't want to see it while my workflows are running.
But throughout the day, you could open the Workflow Builder and you could just use it all day long and it will stack your statements down here in increments of 10 minutes, an hour, two hours. It'll save things. You'll even have something if you left it up that would show that it was done yesterday.
You could come back to this and you could grab statements and all you have to do is grab them and hit this arrow. You can push them up or you can grab them and drag them or copy and right-click, I believe, and copy and go up and paste them in. All sorts of ways to get things into your Workflow Builder. I'm going to right-click here and delete this. I'm going to cut it out because I want to do this the other way.
Let's get out to my JMP home window and just simply open my food journal. There's the button I told you you would get that says, "Hey, do you want to record this?" I'm going to go yes. Notice that in the Workflow Builder right here, the little red dot is now hollowed out. I am in record mode, so everything I do will be recorded.
I'm going to quickly go in here, so I can make sure I get through a lot of examples. I'm going to recode this and I'm going to go in and I'm going to do it in place because I don't want an extra column. I'm not going to type a whole bunch of stuff this time. I'm just going to convert this to upper case so it makes the change and I'm going to recode it.
Then I'm going to go in and I'm going to go to my column info and I'm going to go down and add value ordering on this. Now I'm going to re order this to where I have breakfast first. I've got an AM snack. I need lunch to move up here. I need a PM snack to be after lunch, then I have dinner, then I have a late snack. There's my ordering I wanted.
You'll see that those steps have been recorded. There's my open, there's my recode, there's my change of column property. Now I'm just going to grab this Graph Builder here is the one I think I read. That's easy. Then I don't have to recreate a lot of stuff. I'm going to add a local data filter and I'm going to use that meal column.
Again, here's the order I wanted. If I click on this, I get a slightly different view. It's not a Text Explorer, but you can see here, here's the column for chocolate and of course, it has the most calories. One of my favorite things is chocolate.
Anyway, notice that this Graph Builder is not recorded yet in the Workflow Builder. Platforms don't get recorded right away, but it records after you close the Graph Builder, or we added a button under here. If you want to save the script, you can save it to the workflow. But for all I'm doing today, I'm just going to close it. There it's been written to the Workflow Builder.
The other place that you can go and grab stuff, and this is one of the things I love about the Action Recorder. If I wanted to, I can go over here and you can click on these steps as well now. Under this red triangle menu, you can save the script and say add it to the workflow. You could always go over there and get something if you needed to.
Let's stop recording. Let's hit the reset button and let's close this thing. Now, let's run it and it should run exactly the same way as the first one. I did a little bit different platform, but you can see that column has been recoded and there's my Graph Builder. Now, we have created your first Workflow Builder. The first is as simple as that.
Moving on to a second example I have. Let's look at this virtual join example I created. I used the Pizzas examples from the sample data library. If you're not familiar with virtual join. I'll give you a crash course here in a minute or less.
But I opened the tables, I created link IDs, which you need with virtual join, and then I created link reference columns and I simply ran a Graph Builder. I'm going to run this and then I'm going to show you some things about it. I'm going to tweak it a little bit, show you how we can clean up the Workflow Builder.
Here's virtual join in 30 seconds or less. A link ID is created in pizza subjects there. A link ID is created in the profiles here. Then I went to pizza references, which is what I like to call my main table with virtual join. A ll my references are set up this way. All of these steps I did do manually. It was all saved in the Workflow Builder and created, and then I simply ran this graph.
That's that. Ways that you could clean this up as a presentation is if I was showing this to somebody, they don't really care about seeing the data tables and all that stuff. They would be more interested in my graph and maybe if I did a distribution or a tabulate, the reporting I would get. I'm going to show you a way to go in here and do some things on the right hand side here of the Workflow Builder.
You open the step settings and this is where all the magic happens. When you're doing these things, the JSL is getting is saved. This is where it's happening and we have some additional buttons over here of things we can do. Like I said, I really don't care very much about the table showing, so there's a nice little option in here called hide tables.
I'm going to click this and I'm going to go down and add hide tables to every one of these so that it quickly hides those three tables and I don't have to worry about that anymore.
The other thing that's nice is that these steps right here are actions are all doing things to the data tables. I'm going to select these and I'm going to right -click and I'm going to say group those selected things together and it will put them in a little group. I'm going to name this something. They're going to always be called group, group one, group two so you might want to name them something that means something to you. I'm going to call them link ID and link reference.
The nice thing about this is as you're building workflow, I can reduce this down. Your workflows could be really long. If I reduce this down, I have a lot more space. You can do groups within groups. That's nice too. Anyway, I'm going to expand this for right now.
Then you probably remember when I ran this, it ran really fast. My final step is a Graph Builder, and I want it to hesitate just a little bit so you can see it hesitates. I'm going to go in here and add a custom action. This is where the coding comes in. If you do know JSL, I'm going to add a wait statement here, and I want the wait to happen before my step, so it's going to be right here.
You do have the ability in this to use the arrows to push things down, to push things up. You have a little bitty trash can right here, so if you add something, you can delete it. You also can simply leave something in and uncheck it, and that actually works really well, too. I'm going to go in here and this is where because I know JSL, I'm going to type a wait statement just as simple as that.
Let's close this up and let's run it and let's watch. There's the hesitation and there's the Graph Builder. Let me close it one more time and get you look right here and look for a little running man when I run this. I'm going to execute it. There's the running man and there's the Graph Builder. You can see that this is much more presentable if I'm doing something. Again, if I had other reports I wanted to show, that would work really well.
A couple more things you can do. Obviously, when you create a workflow, you save it up here by saying File, Save or Save As and give it a name. Every workflow has an ending of a . jmp flow. If you want to create a Journal, the developer was cool in creating something called Add Steps to Journal. You get this right here, which is nice.
Journals are nice, but they're hard sometimes to make. This is nice. Each step is in here. The code is in here. You could run it, you could clear it, you can look at it. I've got a thumbnail down here for my Graph Builder, and if I click this, I get the full -size Graph Builder. Really nice feature to add that in there.
The other thing I wanted to point out is if you go up to the red triangle, you have the ability to save the script to the script window. This is the entire script for doing all of these things we've just done. Even the hide function is added at the top to hide with the tables. The only thing that you need to know is that this will run just like JMP would run prior to JMP 17 without a Workflow Builder.
This script does not rebuild the Workflow Builder window. The only way you can use the window is through the UI. But it's nice if you want the script or you just want to save it that way, then you have that ability to do that.
Moving on. The next Workflow Builder I wanted to show you was an educational type. Let's look at this one. This was done by Peter Hersh. He did something much more complicated and I took it and made one of my own. But basically, it's a workflow that opens a data table, it runs a distribution. It goes out to a particular section like right here, and it's using what we call a show message window, a model window where it stops. I put in a definition for quantiles that I took right out of JMP book because I don't know that stuff. Then I said, okay, and I did a second definition which is showing just a summary of these are the other stats that are listed in the summary statistics.
Kind of a nice thing to do here. Notice a couple of things about this workflow. I'm going to close this up. Notice I don't have a red button. If I go here and say File, New, New workflow again, I don't have a red button and I don't have the JMP Log history. When I set this up, I put this in something called presentation mode. That's first thing under the red triangle. You'll see, if I uncheck that, those objects come back.
But I didn't feel like for this thing, you actually need that in there so I turned it off because it's really something you're using for teaching. The other thing that you can also do in here is you can duplicate a workflow. You can just open a new one and this would be one where you could change something.
Let's make a couple of changes here because I just want to show you how nice this is that this will be able to be changed. I'm going to go and get a different JMP table. I'm going to go get the Cleansing table. Then I'm going to show you how this is set up inside of the report.
The report step code was to run the distribution, which I will have to change this as well to be the same table. Then I need to change the column that in this table is called pH. The dispatch for the quantiles will have to be pH as well. This was assigned, the distribution is run, but then it's been assigned. This is a little JSL where it was assigned to a report.
Then the show message window, which you grab from here, this is how this is added, was added in here. You give it a title, which is the title of the message window is client tiles, and there's where I pasted the definition. Then your next step is to clear. It's another JSL step to select your report and deselect it. Then you go on and select the second section of the report, which is the summary statistics.
I need to change this again to pH. Then at the end, here's my second show message window that was added. You can add as many of these as you want to. It's a model window so that it will stop and wait for you to do something. If I've made these changes correctly, let's run this. There you can see there's the Cleansing table, there's the pH column with the distribution.
I'm selecting the definition of the quantiles, and I'm going on and using the definition for the summary statistics at the bottom. A really nice feature here, really cool thing that you can actually do with teaching, I think. I think there's going to be a lot of people that enjoy using that.
Workflow subset is also a really cool thing. You can create a subset of a workflow by going into here and selecting temporary subset, or you can actually do it on the right hand side where you open a table, you add an action, and you say subset in here, which is what I've already done with this.
Here's my subset. You get this little window whether you do it from the temporary side or whether you do it here and it's stored right here always, just so you know. But we built in some selections here. Your selections are to get all the table, 50 % of the table, 25 % or whatever, and that's what I've checked. These are choices. This is going to make a lot more sense for a table that's really big.
This is great because you can run some analysis on some of the table, maybe that's big. Then when you're all done with capturing everything, you would go back and maybe change it back to all of your data and it should work seamlessly. What I did here was I'm using something everybody's probably really familiar with consumer prefs, and there's about 446 rows, so I'm using 25 %.
I'm going to run this and I just ran a categorical platform. You can see here, this is the output from this on just 112 rows. That's all it used in this case. Now, I'm going to close this up and go over here and say, "Okay, let's just say I'm done with what I want to do and I don't need a subset anymore."
Well, I can go back here and delete this with the trash can, or I could simply just uncheck it this time and basically run it again because I might want to do something later with it. Now when I run it again, you can actually see right here that you get the 448 rows. Okay, that's how many are in there. You get your full platform, you get all the data analysis. I think people are going to get a lot of mileage out of that as well. Being able to do the subset, especially if you're using millions of rows of data, I think that will be very, very helpful.
One more thing I forgot just then. Let me show you quickly how to create a workflow package. If you go in here, last thing in the menu is to create a package. The difference in saving a workflow locally and creating a packages locally, it's for you. So it's going to go wherever you put things on your drive, on your Mac or Windows or wherever. But when you create the package, it creates a little temporary place and you can package it together and send it to somebody and it should work seamlessly for them.
What this is telling me is that I need to have this data source of consumer prefs attached to this for it to work. There's that little button I told you about that said presentation mode. If you were giving it to somebody and they didn't really want them to change anything, you could check that or you don't have to check that.
Let's say, okay, and I'm going to name this workflow package, just let it default. It adds the same . jmp flow to the end. I'm going to save it here. Then what I want to do is go back and hopefully I can find it. There it is. I'm going to open it and I want to show you real quickly how it saves that. It did do presentation mode, but just so you can see, here's the temp consumer prefs.
Just to prove that it runs, there's this and it runs the same way. Really nice feature. Packaging is great for sharing stuff with other coworkers, sending reports and that thing.
The generalized workflow is another really nice utility that I think people are going to use. This particular example starts out by not opening a data table. It was built with a template data table and and I ran a distribution. If there's things you like to do every day, like let's say you run a couple of distributions and maybe you run your favorite Graph Builder and you're always doing the same analysis over and over in a given day on all your data, this is the way to set this up.
The way you set this up is you go under the red triangle menu to References, and you click this little button that says Manage. Now I'm in this Managing window and you can see here this is where I use this ANOVA template JMP table. I typed in this prompt that says select the table that I want to use for analysis. I asked the mode to be every time I run this, I want it to prompt me.
Then I had three columns in that table that I used for my three analysis variables. For every one of them, I'm saying each run, select a column for analysis. I'm going to be prompted for that. Then you save this and this reference is all set up for this particular workflow. Let's run this real quick and let's see. Here's my choose table thing right here. Let's go other.
It's giving me an option to go out here and I'm going to go to my data table and I've got a couple of things I wanted to try to show. I'm going to grab hybrid fuel economy here. Here's my prompts, what columns would I like to use? The first one I want to do is combined in BG and then I want to go and grab engine and then I want to grab the number of cylinders.
Here's my three distributions, and then I've got my favorite box plot. That's just showing one. Just for fun, I wanted to show you a second one. Again, hybrid fuel economy is open, and so I'm going to select other, and I'm going to go to the sample data again, and I'm going to look for Titanic passengers and open this.
Then I'm going to go in here and select I want age, and then I want sex, and then I want passenger class. A gain, here's my favorite box plot and here are my distributions and everything ran exactly like I wanted it to. Really good thing to use reference manager in order to build a template out.
Notice when I close this, this didn't close the tables. The reason it doesn't close those tables is because those tables, there's not a statement in here that says open a table. We opened those tables from the file menu system, so I'll just simply go up and close those myself.
The final example I want to show you is something that is Workflow Builder. We had some comments about people wanting to use workflows for archived projects. This is a project I did back in 2015, 16 when I started with JMP, and I think I showed this another time and mentioned that I had been working six years, and I went back the other day and thought, Wait, that's like eight years. I can't count, but other than that.
But this is something where I went out to a website, I read in data on nesting in North Carolina, and it would continue to change. You would read it throughout the summer and I did some work down at Oak Island, which is a beach where we go often.
Anyway, I had saved all of these scripts in a folder that I worked on. I thought, I'm going to try Workflow Builder with this and see what happens. Basically, instead of opening a table first, what I ended up doing is I went out and I used the custom action field where it says add custom action under here as my very first step.
My first few steps, I went and took my script and I pasted the script I had saved inside of this window. This is what's in here. At the bottom, I'm saving a table. The data that I was actually using in this, all I had to do was go to the same web page and change it to 2020, 2021, 2022. That's what I did. Then there's a turtle species table where you go and do the same thing.
Basically, I read these tables in and then I went to the Workflow Builder and I said, Okay, now what I want to do. Once I got them in, I started building my steps. I concatenated the table. I thought, well, I don't need those original tables, so I closed columns so they get closed. I added some columns. I created a change here. I hid and excluded some totals I didn't need and then I ran my platforms.
I'm going to run this so you can see. The cool thing about the platform runs that I'm going to show you is that these are scripts that I didn't have to change at all. As long as my columns were named the same thing, I just took the scripts and copying and pasted them right into here and they worked beautifully just like they did eight years ago.
Notice in this workflow here at the bottom, these two steps right here are italicized and grayed out. The reason they're in there is because I didn't want to show them today. If I right -click on this, you'll notice that the step enabled button is not checked. I'm going to check it just so you can see that change. Now that changes so that it's dark and it's like everything else.
A nice feature that you can do this because as I was looking at this, I thought, "Well, I don't want to show this today, but I might want to keep them. I don't want to delete them." This turns it off so it won't run, but I don't have to necessarily get rid of it.
Let's run this. Pay attention at the top. This is a good example to watch the little running man that pops up here. It takes a minute. You see him because they're going out to the internet, it's actually getting data. This little part goes a little slower, but then it takes off and goes very fast. But once the data is in here, then everything else runs really fast.
Really nice that script all worked. Here's my data. I have this turtle nesting data showing May, June, July, and August for each of these years, 2020, 20 2 1 and 2022. Nesting just for a case study and point tends to trend up, and then the next year it might go down.
Again, it did that. If you go back and look at data from way back, it's just up and down, up and down. But 2022 was a pretty good year for nesting in North Carolina. Then this was a really cool graph that I had before that I loved. It's a bubble plot, but I went out and grabbed a turtle SVG file and plugged the turtles in for the bubble plot.
This is showing nests totals with false crawls, and just false crawls are when a turtle comes out to lay her eggs and then she's scared or something and she doesn't lay or she changes her mind. They're usually pretty even numbers on a certain beach, they'll be pretty close to each other. That's why this bubble plot is cool the way it works. It's showing the nesting totals with the false crawls.
I'm interested in this turtle right here because that's Oak Island where I said that I go to the beach and I did some work down there with them back in 2015, '16. Kind of a fun graph. Kind of neat that this is an archived project and it actually works now. It's a good way to record and save stuff.
That's all I have today. I just want to say that I want to thank the development staff for working so hard on designing this. Hernes Pessour, David White, and Evan McCorkle, just to name a few. Julian Paris was instrumental during the design phase and advisement of this. I do have a reference down here for the sea turtles if you're actually interested.
In closing, I just want to say I think Workflow Builder is the best new feature for JMP 17, but I'm a little biased, but I do believe it's going to save you time with less coding and more clicking. I think you're going to get more and more out of reusing recorded and repetitive steps. It should simplify your work efforts, and then it will definitely accelerate your daily processes, leaving you much more time in your day.
Thank you for listening today. Thank you for letting me share with you about the Workflow Builder. Please try it out. Please let us know what you think and we'll look forward to hearing your feedback.