As Easy as Uploading a Photo: Building, Sharing, and Updating Add-ins Automatically (2021-US-30MP-910)
Michael Akerman, Senior Data Scientist, Novozymes
Johan Pelck Olsen, Data Scientist, Novozymes
The difficulty introduced by a democratizing software like JMP is handling the vast number of users creating and sharing tools and scripts across a large, global corporation. At Novozymes, we've built an automatic toolset called Y'all to help address that need. Y'all is a Python build script, automated workflow, and JMP add-in package that makes building, sharing, and updating JMP add-ins as easy as uploading a photo to Facebook.
Utilizing the continuous integration/continuous deployment capabilities of GitLab (an open-source competitor to GitHub), Y'all allows users to upload raw JSL scripts saved from any platform in JMP through a web user interface, immediately launching an automated Python build script that structures and packages a JMP add-in on the GitLab server. The next time JMP loads, our tool automatically checks for updates (based on work shared by Jeff Perkinson in the JMP community), sending new or updated scripts to users within seconds.
In addition to these capabilities, Y'all allows users to specify the menu name and tooltip of their launchable script, automatically loads a variety of convenient utility functions for JSL coding, automatically adds imported functions to the Scripting Index, and leverages GitLab to allow parallel development of scripts while retaining a full version history of each JSL script and the add-in package, ensuring that valuable work is archived and protected in a centralized system.
Auto-generated transcript...
|
Speaker |
Transcript |
| Michael Akerman | Christie. |
| Christy Spain | Good morning, how are you Michael. |
| Michael Akerman | How are you. |
| Christy Spain | Great Let me close out my email, so I don't get me. |
| Michael Akerman | yeah I'm a streamlining my computer settings so that nothing, nothing pings. |
| Christy Spain | know what i've hit my toolbar Am I. |
| Think I have everything. | |
| how's your weekend. | |
| Michael Akerman | It was good, how are you how's your weekend. |
| Good. | |
| Christy Spain | So do you live in dark. |
| Michael Akerman | Helen wait for us. |
| Okay yeah so pretty close to headquarters I would be going to the actual discovery summit, it was. | |
| present in person at the. | |
| That was disappointing as well, but. | |
| yeah yo on hey. | |
| Christy Spain | Hello Good morning, how are you doing. |
| Johan | doing good. |
| Michael Akerman | I need to set up a an erudite wall behind me like you've got there Johan. |
| Johan | I did did put in click. |
| This and danny's books in there, so you won't be able to recognize. | |
| Christy Spain | looks good. |
| I apologize for the mix up last week but. | |
| I'm happy to be able to meet you both virtually. | |
| Since I'm part of the premier team who supports Nova zones, I don't they should have assigned me to you guys anyway so. | |
| let's see, I have a few things I need to go through do both of you have your display set to 19 2010 at. | |
| I do. | |
| Johan | Let me just double check. |
| Christy Spain | awesome. |
| and | |
| If your sound is good, I would say nobody's wearing logos. | |
| ensure no items are within view that could cause copyright infringement I don't know that your books back there would be a problem. | |
| yeah let's do that, just in case. | |
| Michael Akerman | One of them is one of them is formed from a former JMP discovery summit keynote speaker. |
| Johan | Who is. |
| Michael Akerman | Oh, I didn't he came and talked about that thing explainer. |
| Christy Spain | Oh yeah I have that book. |
| Michael Akerman | yeah yeah. |
| Johan | I haven't. |
| saved any of the ones from the email, but I can just go ahead and pull that out of the email. | |
| Michael Akerman | I'm with having i've got my webcam on a an arm, where I can move it like toward me and back and up and down and left and right like then it's like fiddling with it eternally because it's like I can move it. |
| Christy Spain | Oh that's a good idea. |
| never, never saw me in the right place yeah. | |
| And then everybody's hitting all their Apps outlook any I am teams. | |
| like that. | |
| Michael Akerman | Close teams shouldn't get any notifications. |
| Christy Spain | And you have 30 minutes Is that correct. |
| Michael Akerman | that's my understanding yep. |
| Christy Spain | And will you guys be. |
| Ideally we'd like to just start and go straight through. | |
| Do you want to do a test share of your screen so whenever you transition. | |
| Michael Akerman | Your because we're going to swap between my screen and you're on screen at one point, so let me share my screen first. |
| don't need to share sound. | |
| So. | |
| Michael Akerman | let's back up to begin here. |
| Christy Spain | hi Alex right. |
| Okay. | |
| Michael Akerman | So you're on we were did it did it did it did it did that that and then we were handed off to you on. |
| Johan | yeah will we be controlling screen sharing ourselves, we can use producer dude for us. |
| Christy Spain | you'll you should do that yes. |
| Oh, your background looks great. | |
| Johan | Thanks. |
| Christy Spain | I should have that. |
| Michael Akerman | That I can be more we can be more uniform I guess. |
| Johan | No, we should have our own individual I didn't. |
| Yes. | |
| Christy Spain | Yes, let's get to Michael. |
| Johan | yeah. |
| I guess I won't be much onscreen link you'll be doing, most of the talking and then we'll be doing doing the screen sharing and then. | |
| Michael Akerman | I'm not sure what will we be visible, while the screens are shared. |
| Christy Spain | Well, so whenever I get the recordings back. |
| I get four different. | |
| versions there's one that includes you when it doesn't and I'm not sure Jeff perkinson will be editing everything, so I don't know what hill which one he'll pick. | |
| I would think that he want to show the speaker so. | |
| Michael Akerman | I see on screen, I think. |
| You can just turn off the icons, by the way, if you right click. | |
| Johan | On manage to confuse window, it looks like. |
| Christy Spain | Well, is that really that hard to do. |
| Johan | Oh yeah. |
| So what right click and view and. | |
| Michael Akerman | yeah he found it okay cool. |
| cuz I don't remember exactly where a. | |
| Guy I never have my icons turn on my desktop I just let stuff installed to it, so if I ever turn that on is just like. | |
| Johan | yeah. |
| Christy Spain | So, Michael when you're finished you'll just say okay turn it over to you on the on your screen share and then we'll just go from there we're not really gonna we're not going to take a break, in other words yeah. |
| Michael Akerman | And you get a yellow filter on my room. |
| Christy Spain | Oh, you have a red light your fancy. |
| Michael Akerman | makes me to blue though yeah. |
| Christy Spain | haven't invested in one of those I don't my desk is so small, and I have a lot of space in here. |
| Michael Akerman | yeah I ended up early independent my my boss kept complaining about my microphone quality and my camera quality so I'm like all right man you're buying me a new microphone camera. |
| Christy Spain | good for you, I didn't get this new headset. |
| Michael Akerman | Let me go close my door real quick so. |
| Christy Spain | Oh okay yeah that's important if you have you know furry friends or kids running around. |
| Johan | yeah good neither so well yeah i'll set the closed door. |
| Christy Spain | and Johan Where are you located. |
| Johan | On the company. |
| Christy Spain | than mine oh Nice. |
| be fully. | |
| Johan | Oh, you been. |
| Christy Spain | I have a friend who married to. |
| A guy from Copenhagen are from Denmark, and so we went over there to visit. | |
| Oh yes, beautiful. | |
| Christy Spain | Okay, so, if I can get you guys to state your name company and your abstract title that would be great. |
| And this is not for the recording This is just for Jeff purposes for like identifying what's your. | |
| Michael Akerman | name is sure sure, my name is Michael increment I'm with novozymes the abstract title is as easy as uploading a photo. |
| something about packaging and bill I forgot the exact. | |
| Christy Spain | that's okay. |
| Michael Akerman | Y'All. |
| Johan | And my name is a UN. |
| Also, no sense. | |
| Christy Spain | And we will be recording this and posting it to the discovery Summit Conference, and it will be available for public viewing do you get permission for this recording in that use. |
| Michael Akerman | We do I do. |
| When you go to. | |
| Johan | Oh yeah sorry sorry I was just my mind. |
| Christy Spain | Okay, so. |
| Let me turn off my camera before for you. | |
| Nobody wants to see me. | |
| Christy Spain | And I think we're ready, are you guys ready. |
| Michael Akerman | I guess we'll start with my screen. |
| Christy Spain | Never you're ready let's see the time is. |
| 909 so technically you have until 939 but. | |
| give or take a fuse fine so whenever you're ready. | |
| Michael Akerman | Hello, everyone, and welcome to our talk today on "Y'All." This is an add-in package manager that we've built in Novozymes to automate JMP add-in sharing and updating and maintenance of scripts across our global company. I'm Michael Akerman, and my co presenter is Johan Olsen. |
| Today, I want to, we want to talk to you about Y'All and introduce this toolkit to you too, so that any of you who are interested in it might... | |
| ...Who might want to use it for personal uses or for your organization know how to get this toolkit and to start using it, and what it is. So we'll start by talking about who we are and why we built Y'All... | |
| ...the key features in Y'All as well as the user experience. So Johan's going to go through what a user sees when they're using this toolkit. | |
| We'll go behind the scenes about what's happening behind the Y'All toolkit when a user interacts with it and then we'll do a summary and recap, and have some tips on adopting Y'All in your organization, if you find it interesting for your purposes. | |
| We are data scientists at a company called Novozymes. My co-presenter is Johan Olsen. He's a data scientist in agricultural and industrial biosolutions application research. | |
| I'm a senior data scientists in the same group in Novozymes. We've included our contact information here if you'd like to reach out about Y'All or anything else. Please feel free to reach out to us. | |
| Johan's email address is jpko@novozymes.com and he is JPKO on the JMP User Community. I am mxak@novozymes.com. | |
| Michael Akerman Michael mxak on the JMP User Community, so please feel free to reach out if you have any questions or follow up or comments in any way about Y'All or ways to improve what we've built and what we're working with. | |
| We work for Novozymes. Novozymes is a global biotechnology company. We build biological solutions for sustainability... | |
| ... and improve performance and reduce costs for all kinds of industries around the world. So we support agriculture and biofuels, household care, food and beverages, animal health and lots more industries... | |
| ...with these biological solutions. We've got more than 6000 employees globally and in that set of employees, we've got hundreds of JMP users. | |
| And I mention that because we've got a really large cohort of JMP users that we support and that we work with who are all working with JMP and building things in JMP, and that's really the inspiration for Y'All. | |
| Our purpose as a company is "Together we find biological answers for better lives in a growing world -- let's rethink tomorrow." | |
| As I mentioned, we've got a very large cohort cohort of JMP users in Novozymes. | |
| So we wanted to make their lives easier to work with JMP scripts and JMP add-ins and make their code more reusable and discoverable. | |
| So first off, we wanted to make updating easy. We had observed that across Novozymes people were emailing scripts back and forth. | |
| There were sending JSL files on personal network drives. Some of them were putting it on flash drives and handing them around. We wanted to make distributing new and updated JMP scripts as effortless as possible. | |
| We wanted to make an archive that safeguards the code. We've got a lot of JMP users, all of this work, represents a lot of value to the organization. | |
| So a central repository where all that valuable code is stored and backed up and archived. | |
| And we wanted to track versions over time, so as scripts change, as they're improved, we wanted to be able to see those changes and track those changes over time. | |
| And we wanted to make it easy to discover and reuse scripts so if you're a JMP developer... | |
| ...if you're a developer work writing JSL scripts, you can find relevant scripts written by other developers and improve on them or... | |
| build off of them, for whatever needs you have. So make it convenient to take those scripts and build off of them, also make it convenient to share and import and reuse functions in JSL that other members of the organization would find useful. | |
| So, to make this a reality, we put together three components. We've got a Python build script, we've got a GitLab interface, and we've got a JSL load and update script all working together in this toolkit. | |
| The Python build script is an automated script that will run in Python 3 or above that automatically parses and structures JSL files into add-in menu structures. | |
| It also writes the add-in definition files. It generates an imported custom functions script that adds all of these reusable utility functions to the JMP scripting index and into the JMP IDE, and it zips it all into a .jmpaddin file for use by users. | |
| GitLab is our interface for users to interact with add-in scripts and working with the Add-in Builder itself. GitLab is the open source alternative or competitor to GitHub. | |
| We have this posted internally within Novozymes, and it provides a really easy point-and-click user interface for uploading scripts into this central repository... | |
| ...where all of these scripts are stored and the build actually happens. | |
| And that CICD pipeline, which stands for continuous integration continuous deployment, automatically executes that Python build script and distributes the add-in through both release and beta version channels so users can have either... | |
| ...steady quality-controlled releases, or if they're a developer, they can get frequent quick releases with the very latest versions of each script used in this toolkit. | |
| And finally, we've got the JSL load and update scripts. When JMP opens up and when this add-in loads,... | |
| ...they can check, it will check for new versions on the selected channel -- the beta or release channels -- and offer to download and update that add-in package. This also imports the custom functions into the scripting index. | |
| We've got super easy script contribution in Y'All from any JMP user. It's as easy as uploading a photo on social media. | |
| You go into the user interface, click the upload button, and in seconds it's available on the beta channel of Y'All. | |
| We've got automatic version control and storage of these scripts, an automatic or manual JMP add-in update, so when JMP opens, Y'All will offer to update... | |
| ...or a user can choose to manually update Y'All through the Y'All menu. | |
| We have beta version channel for rapid build and testing if you're a developer writing scripts to get the very latest version of the add-in as you update your script and make changes to it. And we've got a release channel for controlled deployment and quality control. | |
| This gives us discoverable and reusable JSL files, so that people can look through this file structure of scripts submitted by users across Novozymes, Y'All users, and look in and see... | |
| ...find scripts that meet a purpose that's close to what they want and modify them for their own purposes. | |
| And we also add convenience utility functions into the scripting index; you can see a little screenshot of that... | |
| ...on the left and into the JSL editor, so the build script automatically adds functions that are importable and that are reusable into the JMP scripting index with a nice little doc string if that's available to describe how this function works and how to use it. | |
| I did want to take a second to acknowledge some of the code we've included in Y'All for demonstration purposes. Much of the code used within Novozymes is specific to Novozymes' systems and wouldn't be usable outside the company. | |
| So I've grabbed some scripts that were on the JMP User Community already and included them just as demonstration purposes. You can feel free to keep these or remove them from Y'All... | |
| ...as desired. So we've got a json parser script from Xan Gregg and Craige Hales. We've got a data cleaning script assistant presented at JMP Discovery Summit 2020... | |
| ...by Jordan Hiller and Mia Stephens. We've got image table tools by Heidi Hardner and Serkay Olmez at Discovery Summit 2020. | |
| Color row selection.jsl by Nick Holmes. Multiple Y axis chart by Byron Wingerd, and autovalidation setup, I believe, also presented at JMP Discovery Summit... | |
| ...last year by Mike Anderson, and we've got a suite of importable functions for the Riffyn data capture software, Riffyn Nexus... | |
| ...to work with their API, written by Riffyn incorporated and taken from their Riffyn tools JMP add-in with permission. That's a tool that we use heavily in Novozymes. We included this because if there are other Riffyn users, you can feel free to use this code as well. | |
| So with that out of the way, we wanted to jump into some live demos, starting with the user experience and what the user sees and does as they work with the Y'All toolkit. For that, I'm going to hand control over to my co-presenter Johan. | |
| Johan, you're muted. | |
| Johan | Thank you, Michael. |
| Sorry about that. Let me just share my screen. | |
| Are you seeing this? | |
| Michael | Yes. |
| Johan | All right. |
| So as Michael said, I will be walking you through the user experience of using Y'All. | |
| It's really a three-tiered thing to be a user of Y'All. The vast majority of our users basically use it as if it was a built-in JMP platform. They... | |
| ...point and click at the scripts and run them for the purposes that they were intended. We have a pretty large user group as well who are contributing with scripts to also be going through how... | |
| You can contribute simple scripts. And then we have a small group of developers and maintainers (??) that do more advanced stuff, and we'll try to pull a little bit of that as well. | |
| But first of all, the real simple usage of Y'All. Installing it is as simple as clicking on this link to the build artifact (??) that's created in our GitLab repository and downloading the JMP edit file and running it with JMP. | |
| Now this prompted me for an update, I'm just going to cancel this time around, but you can see here that on my menu bar, and I have the Y'All menu... | |
| ...With some of these demo scripts that Michael mentioned. I won't be going into details with any of these scripts as they're not really part of the Y'All package but are just included for demonstration purposes. But what I will do is I'll try to go in and... | |
| ...oops, enable, sorry this was already, enabled it appears. enable beta testing, so that, now that we start creating some things we can see them right away. | |
| Most users will have beta testing disabled, and they will be prompted with an update when they open JMP after a major release, which is once every few weeks. But for users with beta testing enabled, it might be anytime from several times a day to a couple times a week when the beta branch will be updated. | |
| So that's basically what's in it for a normal user. | |
| If you want to contribute scripts to this, there are a couple ways of doing it. One thing is through the GitLab UI. Now the menu structure that we just saw is mirrored in the... | |
| ...directory structure in the... | |
| ...repository. So creating a new... | |
| ...menu or sub menu is as easy as just creating a new directory. Here, let's just call this... | |
| ..."demos" | |
| johan | and add it. |
| Johan | And if we would like to add a script, we can simply create it in the JMP IDE so you want to make a very simple. |
| Hello world. | |
| This will be good enough. There's actually a little extra detail, which is to name the menu items in a nice way to get a tool tip, we implemented the [???] or code magic, that is, if you start by writing commented out, oops. | |
| And you. | |
| Menu name. | |
| Michael Akerman | Y'All. |
| Menu name. | |
| Johan | Y'All menu name |
| michael Akerman | Old habit. |
| Johan | Let's call it hello. Then Y'all tooltip. |
| All right, let's go ahead and save this. | |
| To wherever, I guess, let's see, desktop. And then upload it here. | |
| Was it test? | |
| Michael Akerman | It was script but should be OK. |
| Johan | Now... |
| Once we've uploaded this file, we've triggered a pipeline in this automated [inaudible]. | |
| [inaudible] system, which is a cloning the repository first and then... | |
| ...afterwards building the add-in, and this should prompt upon a restart of JMP, | |
| johan | and update of the all edit. |
| Johan | And once we've chosen to update, the demos menu is now present, with the test script. |
| So this is the simplest way of contributing scripts to JMP. If you are familiar with using Git, you can also use a more classic Git workflow. Just going to open a batch window here. Let me just... | |
| And we can clone the repository. Let's go back to... Some of the things that we're showing here will be specific to using GitLab, which we use here. | |
| But it will be similar in other remote repository coding, code hosting systems. | |
| So now we are cloning the repository, and we will have a complete mirror of the repository in our local system. And similar to before, we can see the directory structure mirrored here, and let's try to see if we can create a... | |
| Another script. | |
| For this. Y'All menu name improved | |
| hello. | |
| Say we'd like to, maybe this time.. | |
| ...open a prompt instead. | |
| It says hello world. | |
| Now, this time around, we can use one of the built-in functions that were included with JMP. Let's go to the scripting index, we can... | |
| ...search for Y'All utils. | |
| And find the errorModal function, and it comes here with a little doc string, of how to use it. | |
| And it's already been imported so... | |
| Y'All utils errorModal | |
| johan | And we can just say hello world. |
| Johan | let's see this. |
| into our repository... | |
| ...under demos. And push the code. | |
| Now the code is pushed to the remote repository, triggering the CICD pipeline, and in a matter of... | |
| ...seconds, should be able to see the pipeline running and... | |
| ...soon enough. | |
| Hasn't started yet. | |
| Maybe I'm getting a little bit... | |
| ...impatient. There we go. | |
| So sure enough, if we go in and check for updates to Y'All, we will see a new update. Is that it? | |
| We will choose to update... | |
| ...the add-in, and now the demos will have our improved hello world. | |
| All right, so I think that was what I had for the user demo. | |
| Michael Akerman | Thanks, Johan. I'll take back over the screen share then, if that's OK with you. |
| Johan | Yes. |
| Michael Akerman | And I wanted to go in a bit to what's happening behind the scenes, when this user experience is happening. So I'm going to look at this code in Visual Studio Code. |
| You can look at this wherever you want once you download the Zip file, and I'm going to go into Zen mode real quick to make it a little bit easier to see. Here we go. | |
| So here's this GitLab CICD pipeline that's running whenever Johan uploaded that script. So he went into the Git repository... | |
| ...and he hit the plus button uploaded a script, and that fired off this CICD pipeline. So this is a small script that defines how the server should react to that situation. | |
| a pre step and a build step... | |
| ...where it cloned the Git repository and all the files in it, including Johan's new script into a working directory where this... | |
| ...this processing module called a runner can work on that code. And then it built it, so it runs this build stage where it... | |
| CDs -- changes directories -- into that build directory and runs that Python build script. So it's as easy as that really. It's cloning all of these new files into a working folder, moving into that folder and running the Python build script in Python 3. | |
| So let's take a look at that Python build script. I'm going to control tab over to that. This is build_yall.py. | |
| So this is a lot of code, but if you're looking at this and want kind of a guide to how to look through it, scroll all the way down to this process function. It says def process. | |
| This is the main code that builds Y'All and step through it, step by step. So we're going to peek into this as we go. | |
| The first thing that happens is the Python script builds a menu hierarchy. So let's peek into that. What it's doing is making a nested Python dictionary where it looks through that folder structure under src/scripts... | |
| ...and pulls all of the JSL files it finds into a dictionary that contains the name and the tooltip and the file location of that JSL file. So it's building that nested dictionary one folder at a time, basically, and crawling through. | |
| Then we check the command names of each of those JSL files. We try to parse those files and look for that Y'All menu name and Y'All tooltip that you saw Johan put in at the top of the script to... | |
| ...define what the button name should be and what the tooltip should be in the add-in menu in JMP. Whenever Y'All is installing, that Y'all menu shows up, what those should be called and what the tooltip should say, if you hover over it. | |
| If it cannot find that Y'All menu name or Y'All tooltip kind of magic command, it will instead just fall back to the name of the script with no tooltip. So test.jsl would just become test as you saw from Johan. | |
| That dictionary of commands in menus is sorted alphabetically to make it easy to find things, except for the help and settings menu, which is popped up to the top to make that always at the top of the menu. | |
| Because help and settings are important. And then all of that information is written into a JMPcust file. | |
| So a .jmpcust file is essentially a kind of markup language document that defines for JMP how to... | |
| what menu buttons should be, where should the menu itself be, what should the tool tips be, are all defined in this .jmpcust file. | |
| I'm going to control tab over to Y'All, so you can see what this looks like. It's literally a bunch of definitions about what the menus are... | |
| ..and what the menus are called and what they contain. So we can see the help and settings menu contains check for Y'All update, switch beta testing mode on off, and so forth. | |
| So the Python script is going to write that automatically with this write jmpcust file function. | |
| And then we're going to write our add-in definition files. So addin.def and version.txt. Addin.def -- I'm going to switch over to an example of that -- is a little... | |
| ...bit of description that JMP uses to identify the add-in and find the version number, as well as the minimum JMP version supported. | |
| We say 12. That's kind of an arbitrary choice on our part, just to not have to worry about things before 12. You can change this in the Python code, if you start working with this yourself. The add-in version is also defined here, as you can see. | |
| And we write this into a file called version.txt so that same integer number you just saw in addin.def ends up in version.txt as well. | |
| The reason for that is that's a nice small file we can read remotely. We can open in JMP over HTTP in the server... | |
| ...and take a look at that version number very quickly in that version.txt file to compare to our current version. So if our currently installed version doesn't match the version.txt... | |
| ...that means we need to update, or we at least can update. The version number we use is the date time that this ran in the UTC timezone. | |
| And we convert that to basically seconds since 1970. So it's an integer number that's convertible back into a date time format. | |
| So we can compare the two numbers on an integer basis, which is nice. We can easily see if they're greater than or less than each other, and we can still convert that back into a date. So it's a really efficient way to store that version number in JMP for us. | |
| Finally. | |
| The next thing we do -- so we've written those add in depth files -- we create a custom functions script. So this is a little strange and a little hard to explain. I'll do my best to do it quickly and succinctly. | |
| JMP has two ways of defining custom functions. One is with a function called function in the JMP Scripting Language, and what that does -- I'll switch over to utils.jsl and the importables sub sub directory of Y'All here. | |
| Function defines a custom function that can be used and called from other locations. And you can define this with a namespace. So in this case, we've got the namespace Y'All utils... | |
| ...with the function name recode if. And it takes these arguments, and we've got this doc string and this function code written here. | |
| The second way to write a custom function in JSL is with the new custom function function, which is not the same as function. I believe this was added and JMP 14. | |
| What new custom function does is it adds the function to the JMP scripting index and into the JSL IDE to make it very easy to work with that function and to reuse it in other in other places with documentation. | |
| So the advantage of function is that it works all the way back to I don't even know what version of JMP... | |
| ...before I was using JMP. The advantage of new custom function is it is a bit more robust, and you get the documentation and the JMP scripting index and the ability to colorize it in the JSL IDE and get... | |
| ...tips on it as you're writing code. So we like new custom function, but we want to support JMP before JMP 14. So the build scripts... | |
| ...here, will essentially parse through the importables directory, looking for any function definitions matching that functions function function function... | |
| ...in JSL with a doc string. So it will try to parse out that doc string between the slash asterisk demarcations and pull that into a new custom function. So you can see an example here. | |
| We take that recode if function and now rewrite it automatically into a new custom function in the Y'All utils namespace called recode if. | |
| And we've got a prototype and a description and a scripting index category where this all shows up. So that's all available to JMP developers automatically. So you can write things using the older function function in JSL and still get this JMP scripting index... | |
| ...ability, by using the Y'All build script automatically. Scroll back down to my process code here. | |
| All of this being written, we now build the add-in. JMP add-ins are using a Zip format that's been modified. So we Zip it together using Python 3, rename it to YALL.jmpaddin and that's now available in our working directory where the CICD runner is working. | |
| So the final step of this CICD pipeline after this build script is complete is to move into that working directory and upload artifacts. | |
| Build artifacts are a GitLab CICD concept, where we take these two files YALL.jmpaddin and version.txt and upload them to the server and make them available with a specific URL. | |
| So that URL is always consistent, and it's tied to the branch that we're running this on in GitLab, so if we're on the beta version versus the main release version. | |
| Now, in JMP -- I'm going to jump over, ha!... | |
| ...to autoUpgrade.jsl. This is where we check for new versions of the add-in. When JMP loads, Y'All is going to run this auto-upgrade script,... | |
| ...which looks for those new versions in the GitLab repository. You can see here we load our beta flag, whether on beta mode or main mode. | |
| And we look in the specific channel for that mode. So we're pulling... | |
| ...from a specific URL based on that channel selection. So if we're in beta mode, it's going to look the beta channel. If we're in main mode, it's going to look at the main channel. And we look for the YALL.jmpaddin file and the version.txt | |
| michael Akerman | And now we're going to compare that version number to our installed version number, and if it's different, request an update from the user. Pop up a dialog box that you saw in Johan's screen... |
| Michael Akerman | ...that asks them if they want the update and tells them what that version number is. And they can choose to or choose not to. |
| So. | |
| That's all there is to it really. All that happens behind the scenes, for the user. They upload a script through that GitLab UI using that plus button to upload a file and build that new... | |
| ...JMP add-in, that Y'All JMP add-in, and make it available to other users who are connected to the same repository. | |
| All that happens behind the scenes. It's as easy for the user as saving a JMP script from the red hot spot in JMP and uploading it to the GitLab repository. | |
| So I wanted to recap on the features of Y'All. | |
| We have very easy script contribution. As you've seen, you've got a nice user interface to upload files, upload JSL scripts, and make them available. | |
| And with that comes automatic version control and code archiving. So since this is all happening through a central GitLab repository stored on Novozymes servers, all of that is tracked and stored... | |
| ...in perpetuity. Automatic updates so developers can release scripts to their users very easily, and they can always have the most recent version... | |
| ...of those scripts that work properly and are quality-controlled. And we've got beta and release channels. | |
| So the beta channel is going to be essentially instantly built, available within a couple of seconds of a new script hitting Y'All and updated frequently. And the release channel is going to be a more slowly controlled release structure. I wanted to touch on how we do that. | |
| You can see here the branch structure in our GitLab repository. We've got a beta branch with a lot of updates. We've got a release branch. We've got a main branch. What we do in Novozymes -- this is totally up to user choice -- | |
| michael Akerman | ...we take this beta branch when it's in a state where we want to release it. We open a merge request between beta and release, and move it to the release branch. That's going to essentially freeze the screen release branch... |
| Michael Akerman | ...at the code version, where we did that, and that locks the code down on this release branch. |
| Beta can still change. People can still add scripts and change scripts on beta. But release is now frozen. We quality-check that, and when we're prepared, when we're happy with the state of it... | |
| ...we release it to main and merge it into there, and now that is available to main branch users of Y'All. | |
| We have discoverable code. Everything's available in a file structure on a central repository. Developers can look through the scripts and... | |
| ...find similar ones to build off of and improve on. And we've got utility functions automatically added into the scripting index, automatically available to developers, so that they can reuse blocks of code like that recode if... | |
| ...or some database queries and things like that that we've included as demonstrations as well. So, if you look through the Y'All... | |
| ...namespace in the JMP scripting index with Y'All installed. You'll see that we've got some some postgres database queries and postgres database tooling that we've included to demonstrate how we use that as well. | |
| All of this is happening by starting in JMP with a user-created JSL script that they want to share with somebody else in the Y'All user base. | |
| They go into GitLab. And in the user interface upload that into a subdirectory matching the menu location where they want that in Y'All so if they want it in demos demos test... | |
| ...they would put that in the src slash scripts slash demos slash demos subdirectory. Once they do that, an automatic build pipeline fires off that launches the Python build script, which parses structures and builds the add on.... | |
| ...the add in and re-uploads that to the GitLab server. JMP then finds that new update and automatically offers to download it and update the Y'All add-in in JMP. And that script is available within just a couple of seconds to users on that beta channel. | |
| So we hope you're excited about Y'All.We hope you're interested in using Y'All yourself or in your organization. We wanted to give some tips on how to get started, if you are. | |
| The Zip file that's posted in the JMP User Community if you're watching this video in the JMP User Community -- it should be on the same page -- has all the necessary elements to get started. Some of those will need modification. Some of those are demo code for you to play with. | |
| But feel free to download that and take a look. The Python build script in that Zip file is built on native Python 3 libraries. There's nothing installed but Python on our build system. And it should run on any standard Python 3 installation. | |
| The central repository storage and automation that we do with GitLab for CICD for continuous integration and deployment, that's available through other providers. GitLab, GitHub, Azure, AWS all have similar implementations. | |
| There's a huge ecosystem of options out there, so the implementation details will be specific to your organization. | |
| The unfortunate side effect of that is that CICD pipeline file gitlab-ci.yml will require a modification, if you want to run this in an automated fashion on your own systems. It will not work as it is on your systems, unless of course you're a Novozymes employee. | |
| Any connection strings in our utility functions that that we have that reflected our internal beta database have been replaced with ODBC my underscore connection underscore string. | |
| If you want to use any of those functions, you will have to go into the code in the working directories and change those strings to reflect your database. | |
| So, once again, we hope you're excited about Y'All. We hope you're interested in Y'All. If you have any feedback, questions, comments, please feel free to reach out to me or Johan... | |
| ...with them. We would also love any contributions you have, if you take this code and improve on it or take this toolkit and use it in an interesting way. Thank you. | |
| Christy Spain | Okay, that was great. |
| Michael Akerman | Time. |
| Christy Spain | Reasonable perfect yeah yep. |
| Michael Akerman | Does it. |
| doesn't matter that Johan said no JMP wants. | |
| Christy Spain | You know. |
| Michael Akerman | I feel, like most people probably won't catch it. |
| Christy Spain | As a new word I like it. |
| Johan | So we were told to not use that so we had to rename the thing and I I got thrown off by not having deleted the version I had. |
| Previously installed. | |
| So when I realized that I was already on beta it kind of threw me off, and then I just. | |
| anyways. | |
| Christy Spain | yeah I think that's fine, but now that was great very interesting I think people are gonna love to see this, so we really appreciate you guys participating. |
| As I said, I'm going to upload the videos for Jeff to edit I'm not sure when that's going to be completed, but. | |
| Anyway, you guys will see all that October, but if anything comes up i'll certainly let you know but. | |
| thanks again and if. | |
| Johan | You have a new breed of be about the normal JMP. |
| Michael Akerman | About you saying you're somewhere else it just kinda. |
| yeah. | |
| Johan | Y'All directory. |
| Christy Spain | Yes, thank you so much, I know it's. |
| A lot of work to disease, and we do appreciate it. | |
| Michael Akerman | Our pleasure. |
| Thanks for. | |
| Michael Akerman | moving quickly after after the next up. |
| Christy Spain | Yes, man, it was of course great to meet you both and. |
| Hopefully next time. | |
| At least with Michael it'll be in person, Johan i'd like over there, meet you as well, but. | |
| yeah. | |
| Michael Akerman | it's always you know if if he's, especially if he's presenting and just every summit, or something that's a in the past in person it's been the same week as all things open in raleigh so you can do like a whole week of. |
| computational conference. | |
| and see a bunch of local local employees. | |
| Johan | might be easier to argue the travel budget, then yeah. |
| Christy Spain | Two for one right. |
| yeah. | |
| Johan | All right. |
| Christy Spain | Okay well you guys have a great rest of your Monday great week thanks. |
| You, too, can talk to you again soon. | |
| Johan | bye. |
| bye. |
