Extending Hamcrest Automated Testing of JSL Applications for Continuous Improvement (2021-EU-45MP-783)
Vince Faller, Chief Software Engineer, Predictum
Wayne Levin, President, Predictum
This session will be of interest to users who work with JMP Scripting Language (JSL). Software engineers at Predictum use a continuous integration/continuous delivery (CI/CD) pipeline to manage their workflow in developing analytical applications that use JSL. The CI/CD pipeline extends the use of Hamcrest to perform hundreds of automated tests concurrently on multiple levels, which factor in different types of operating systems, software versions and other interoperability requirements.
In this presentation, Vince will demonstrate the key components of Predictum’s DevOps environment and how they extend Hamcrest’s automated testing capabilities for continuous improvement in developing robust, reliable and sustainable applications that use JSL:
- Visual Studio Code with JSL extension – a single code editor to edit and run JSL commands and scripts in addition to other programming languages.
- GitLab – a management hub for code repositories, project management, and automation for testing and deployment.
- Continuous integration/continuous delivery (CI/CD) pipeline – a workflow for managing hundreds of automated tests using Hamcrest that are conducted on multiple operating systems, software versions and other interoperability requirements.
- Predictum System Framework (PSF) 2.0 – our library of functions used by all client projects, including custom platforms, integration with GitLab and CI/CD pipeline, helper functions, and JSL workarounds.
Auto-generated transcript...
Speaker | Transcript |
| Wayne Levin | Welcome to our session here on extending Hamcrest automated testing of JSL applications for continuous improvement. |
| What we're going to show you here, our promise to you, is we're going to show you how you too can build a productive cost-effective | |
| high quality assurance, highly reliable and supportable JMP-based mission-critical integrated analytical systems. Yeah that's a lot to say but that's | |
| that's what we're doing in this in this environment. We're quite pleased with it. We're really honored to be able to share it with you. | |
| So here's the agenda we'll just follow here. A little introduction, my self, I'll do that in a moment, and just a little bit about Predictum, because you may not know too much about us, | |
| our background, background of our JSL development, infrastructure, a little bit of history involved with that. | |
| And then the results of the changes that we've been putting in place that we're here to share with you. Then we're going to do a demonstration and talk about what's next, what we have planned for going forward, and then we'll open it up, finally, for any questions that that you may have. | |
| So I'm Wayne Levin, so that's me over here on the right. I'm the president of Predictum and I'm joined | |
| with Vince Faller. Vince is our chief software engineer who's been leading this very important initiative. So just a little bit about us, right there. We're a JMP partner. | |
| We launched in 1992, so 29 years old. | |
| We do training in statistical methods and so on, using JMP, consulting in those areas and we spend an awful lot of time building and | |
| deploying integrated analytical applications and systems, hence why this effort was very important to us. We first delivered JMP application with | |
| JMP 4.0 in the year 2000, yeah, indeed over 20 years ago, and we've been building larger systems. Of course, since back then, it was too small little tools, but we started, I think, around JMP 8 or 9 building larger systems. | |
| So we've got quite a bit of history on this, over 10 years easily. So just a little bit of background...until about the second half of 2019, our development environment was really disparate, it was piecemeal. | |
| Project management | |
| was there, but again, everything was kind of broken up. We had different applications for version control and for managing time, you know, our developer time, and so on, and just project management generally. | |
| Developers were easily spending, and we'll talk about this, | |
| about half their time just doing routine mechanical things, like encrypting and packaging JMP add-ins. You know, maintaining configuration packages and, | |
| you know, and separating the repositories or what we generally call repo's, you know, for encrypted and unencrypted script. It was...there was a lot we hade to think about that wasn't really development work. It was really work that | |
| developer talent is...was wasted on. We also had, like I said, we've been doing it a long time, even at 2019, we had easily 10 years, so over 10 years of legacy framework going all the way back even to JMP 5, you know, with, you know, it was getting bloated and slow. | |
| And we know JMP has come a long way over the years. I mean in JMP 9, we got namespaces and JMP 14 introduced classes and that's when Hamcrest | |
| began. And it was Hamcrest that really allowed us to go this this...with this major initiative. So we began this major initiative back in August of 2019. | |
| And that's when we are acquired our first Gitlab licenses and that's the development of our new...the development of our new development architecture, there you go, started to take shape and it's been improving ever since. | |
| Every month, basically, we've been adding and building on our capabilities to become more and more productive, as we go forward. | |
| And and that's continuing, so we actually consider this, if you will, a Lean type of effort. It really does follow Lean principles and it's accelerated our development. | |
| We have automated testing, thanks to this system, and Vince is going to show us that. And we have this little model here, test early and test often | |
| And that's what we do. It supports reusing code and we've redeveloped our Predictum system framework. It's now 2.0. We've learned a lot from our earlier effort. All that's gone, pretty much all of its gone, and it's been replaced and expanded. | |
| And Vince will tell us more about that. Easily, easily we have over 50% increase in productivity, and I'm just going to say the developers are much happier. They're less frustrated. | |
| They're more focused on their work, I mean the real work that developers should be doing, not the tedious sort of stuff. | |
| There's still room for improvement, I'm going to say, so we're not done and Vince will tell us more about that. | |
| We have development standards now, so we have style guides for functions and all of our development is functionally based, you might say. | |
| Each function requires at least one Hamcrest test, and there are code reviews that the developers, they're sharing with one another | |
| to ensure that we're following our standards. And it raises questions about how to enhance those standards, make them better. | |
| We also have these, sort of, fun sessions, where developers are encouraged to break code, | |
| right, so they're called like, these break code challenges, or what have you. So it's become part of our modus operandi and it all fits right in with this development environment. It leads to, for example, further tests, further Hamcrest tests to be added. We have one small, fairly small project | |
| that we did just over a year ago. We're going into a new phase of it. It's got well over... | |
| well over 100 Hamcrest tests are built into it and they get run over and over and over again through the development process. | |
| So some other benefits is it allows us to assign and track our resource allocation, like what developers are doing what. Everyone knows what everyone else is doing, | |
| continuous integration, continuous deployment, something like that), | |
| there's...code collisions are detected early so if we have... and we do, we have multiple people working on some projects, so, you know, somebody's changing a function over here and it's going to collide with something that someone else is doing. We're going to find out much sooner. | |
| It also allows us to improve supportability across multiple staff. We can't have code dependent on a particular developer; we have to have code that any developer or support staff can support | |
| ging forward. So that's was an important objective of ours as well. And it does advance the whole quality assurance area just generally, | |
| including supporting, you know, FDA requirements, concerning QA, you know, things like validation, the IQ OQ PQ. So it's...we're automating or semi automating | |
| those tasks as well through this infrastructure. We do use it internally and externally, so you may know, we have | |
| some products out there, (???)Kobe sash lab but new ones spam well Kobe send spam(???) are talked about also elsewhere | |
| in the JMP Discovery European Conference in 2021. You might want to go check them out, but they're fairly large | |
| code bases and they're all developed, in other words, we eat our own dog food, if you know that expression, but we also use it with all of our client | |
| development, so this is something that's important to our clients, so because we're building applications that they're going to be dependent on. | |
| And so we, we need to...we need to have the infrastructure that allows us to be dependable, and anyway, that's a big part of this. | |
| I mentioned the Predictum system framework. You can see some snippets of it here. It's right within the scripting index, and you know, we see the arguments and the | |
| examples and all that. We built all that in and 95%, over 95% of them have Hamcrest tests associated with them. Of course, our goal is to make sure that all of them do and we're we're getting there. | |
| We're getting there. Have...these framework...this framework is actually part of our infrastructure here. That's one of the important elements of it. Another is just that...Hamcrest... | |
| the ability to do the unit testing. And I'm going to have...there's a slide at the...at the end, which will give you a link into the Community where you can learn more about Hamcrest. This is a | |
| development that was brought to us by by JMP, back in JMP 14, as I mentioned a few minutes ago. Gitlab is a big part of this; that gives us the project management repository, the CI/CD pipeline, etc. | |
| And also there's a visual...visual studio code extension for JSL that we created and we'd...you see five stars there because it was given five stars on the | |
| on the visual studio. I'm not sure what we call that. Vince, maybe you can tell us, the store, what have you. It's been downloaded hundreds of times and we've been | |
| updating it regularly. So that's something you can go and look for as well. I think we have a link for that as well in the resource slide at the end. | |
| So what I'm going to do now is I'm going to pass this over to Vince Faller. Vince is, again, our chief software engineer. Vince led this initiative, starting in August 2019, as I said. It was a lot of hard work and | |
| the hard work continues. We're all, in the company, very grateful for Vince and his leadership here. So with that said, Vince, why don't you take it from here? I'm gonna... | |
| I'm... | |
| Vince Faller | Sharing. |
| So Wayne said Hamcrest a bunch of times. For people that don't know what Hamcrest is, it is an add-in created by | |
| JMP. | |
| Justin Chilton and Evan McCorkle were leading it. It's just a unit testing library that lets you run, test, and get results of it in an automated way. | |
| It really started the ball rolling of us being able to even do this, hence why it's called extending. | |
| I'm going to be showing some stuff | |
| with my screen. I work pretty much exclusively in the VSCode extension that we built. This is VSCode. | |
| We do this because it has a lot of built-in functionality or extendable functionality that we don't have to write, like get integration, get lab integration. | |
| Here you can see this is a JSL script and it reads it just fine. | |
| If you want to get it, if you're...if you're familiar with VSCode, it's just a lightweight text editor. You just type in JMP | |
| and you'll see it. It's the only one. | |
| But we'll go to what we're doing. | |
| So. | |
| For any code change we make, | |
| there is a pipeline run. | |
| We'll just kind of show what it does. So if I change the README file to this is a demo for Discovery. | |
| 2021. | |
| And I'm just going to commit that. If you don't know get, committing as just | |
| saying I want a snapshot of exactly where we are at the moment, and then you push it to the repo and it saved on the server. | |
| Happy day. | |
| Commit message. More readme info. | |
| And I can just do get push, because VSCode is awesome. | |
| Pipeline demo. | |
| So now I've pushed it. There is going to be a pipeline running. I can just go down here and click this and it will give me my merge request. So now | |
| pipeline started running. | |
| I can check the status of the pipeline. | |
| What it's doing right now is it's going to go through and check that it has the required Hamcrest files. We have some requirements that | |
| we enforce so that it can... | |
| we can make sure that we're | |
| doing our jobs well. | |
| And then it's done. I'm going to press encrypt. | |
| Now encrypt is going to take the whole package and encrypt it. | |
| If we go over here, | |
| this is just a vm somewhere. It should | |
| start running | |
| in a second. | |
| So it's just going through all the code. | |
| Writing all the encrypted passwords, going through, clicking all that stuff. If you've ever tried to encrypt multiple scripts at the same time, | |
| you'll probably know that that's a pain, so we automated it so that we don't have to do this because, as Wayne said, it was taking a lot of our time to do these. | |
| Like, if we have 100 scripts to go through and encrypt every single one of them every time we want to do any release, it was awful. | |
| Because we have to have our | |
| code encrypted because, yeah sorry, opinion, all right, I can stop sharing that. | |
| Ah. | |
| So that's gonna run. | |
| It should finish pretty soon. | |
| Then it will go through and stage it and then the staging basically takes all of the sources of information we want, as our as in our documentation, as in | |
| anything else we've written, and it renders them into the form that we want in the add-in, because much like the rest of github, gitlab, most of our documentation is written in markdown and then we render it into whatever. | |
| I don't need to show the rest of this but yeah. So it's passing. It's going to go. | |
| We'll go back to VSCode. | |
| So. | |
| If we were to change, so this is just a single function. | |
| If I go in here like, if I were to run this... | |
| JSL, run current selection. | |
| So. | |
| You can see that it came back...all that it's trying to do is open Big Class, run a fit line, and get the equation. It's returning the equation. | |
| And you can actually see it ran over here as well. | |
| But. | |
| So this could use some more documentation. | |
| And we're like, oh, we don't actually want this data table open. | |
| But let's let's just run this real quick. | |
| And say, | |
| no. This isn't a good return, it turns the equation in all caps apparently. | |
| So if I stage that. | |
| Better documentation. | |
| Push. | |
| Again back to here. | |
| So, again it's pushing. This is another pipeline. | |
| It's just running a bunch of power shell scripts in order, depending on however we set it up. | |
| But you'll notice this pipeline has more stages. | |
| So when we | |
| in an effort to help be able to scale this, we only test the JSL | |
| minimally at first, and then, as it passes, we allow to test further. | |
| And we only tested if there are JSL files that have changed. | |
| But we can go through this. | |
| It will run and it will tell us where it is in the the testing, just in case the testing freezes. You know, if you have a modal dialog box that just won't close, obviously JMP isn't going to keep doing anything after that. But you can see, it did a bunch of stuff, yeah, awesome. I'm done. | |
| Exciting. | |
| Refresh that. Get a little green checkmark. | |
| And we could go, okay, run everything now. It would go through, test everything, then encrypt it, then test the encrypted, basically the actual thing that we're going to make the add-in of, and then stage it again, package it for us, create the actual add-in that we would give to a customer. | |
| I'm not going to do that real quick because it takes a minute. | |
| But | |
| let's say we go in here and we're, like, oh, | |
| well, I really want to close this data table. I don't know why I commented out in the first place. I don't think it should be open, because I'm not using it anymore, we don't...we don't want that. | |
| We'll say okay. | |
| Close the dt. | |
| Again push. Now, this could all be done manually on my computer with Hamcrest. | |
| But you know, sometimes a developer will push stuff and not run all of their Hamcrest for everything on their computer, and this is...the entire purpose of it is to catch it. It forced us to do our jobs a little better. | |
| And | |
| yeah. | |
| Keep clicking this button. I could just open that, but it's fine. | |
| So now you'll see it's running the pipeline again. Go to the pipeline. | |
| And I'm just going to keep saying this for repetition. We're just going through, testing, and encrypting, then testing | |
| because sometimes encryption enters its own | |
| world of problems, if anybody's ever done encrypting. | |
| Run, run, run, run, run. | |
| And then, oh, we got a throw. Would you look at that? | |
| I'm not trying to be deadpan, but you know. | |
| So if we were to mark this as ready and say, yeah we're done, we'd see, | |
| oh, well, that test didn't pass. | |
| Now | |
| we could download | |
| why the test didn't pass in the artifacts. | |
| And this will open a J unit | |
| file that I'm just going to pull out here. | |
| It will also render it in getlab, which might be easier, but for now we'll just do this. | |
| Eventually. Minimize everything. | |
| Now come on. | |
| So, we could see that something happened with R squared and it failed. | |
| Inside | |
| of blue. | |
| So we can come here. | |
| Say, why is there something in boo that is causing this to fail? | |
| We see, oh, somebody called our equation | |
| and then they just assumed that the data table was there. So because something I changed broke somebody else's code, as if that would ever happen. | |
| So we're having that problem. | |
| Where did you go? Here we go. | |
| So that's the main purpose of everything we're doing here, is to be able to catch the fact that I changed something and I broke somebody else's stuff. | |
| So I could go through. | |
| Look at what boo does. | |
| Say, oh well, maybe I should just open Big Class | |
| myself. | |
| Yeah, cool. Well, if I save that, | |
| I should probably make it better. | |
| Open Big Class myself. | |
| I'll stage that. | |
| Open Big Class.Get push. | |
| And again, just show | |
| the old pipeline. | |
| Now this should take | |
| not...not too long. | |
| So we're going to go in here. We're...we only test on one...to... | |
| JMP version, but you can see | |
| automatically, we only test on one. | |
| Then it waits for the developer to say, yeah, I'm done and everything looks good. Continue. We do that for resource reasons, | |
| because these are running on vms that are automatically just chugging all the time, and we have multiple developers, who are all using these systems. | |
| We're also... | |
| You can see, this one is actually a docker system, we're containerizing these. | |
| Well, we're in the process of containerizing these. We have them working, but we don't have all the versions yet. | |
| But we run 14.3, at least for this project, we run 14.3, 15, 15.1, | |
| and that should work. | |
| Let's just revert things. | |
| Because that you know works. | |
| Probably should have done a classic...but it's fine. | |
| So yeah. | |
| We're going to test. | |
| I feel like I keep saying this over and over. We're going to test everything. | |
| We'll actually let this one run to show you kind of the end result of what we get. | |
| It should only take. | |
| a little bit. | |
| And so we'll test this, make sure it's going, and you can see the logs. We're getting decent information out of what is happening, on where it is, like it'll tell you the runner that is running. | |
| I'm only running on Windows right now. Again, this is a demo and all that but | |
| we should be able to run | |
| more. | |
| While that's running, I'll just talk about VSCode some more. | |
| In VSCode, we also...there's also snippets and things, so if you want to make a function, | |
| it will create you all over the the function information. We use natural docs | |
| again, that was stolen from the Hamcrest team, | |
| as our | |
| development documentation. | |
| So it'll just put everything in a natural docs form. So it just, again, the idea is helping us do our jobs and forcing us to do our jobs | |
| a little better, with a little more gusto. | |
| Wayne Levin | For the documentation? |
| Vince Faller | So that's for the documentation, yeah. |
| Wayne Levin | As we're developing, we're documenting at the same time. |
| Vince Faller | Yep. Absolutely. |
| You know, it also has for loops, while loops. | |
| For with an associate row, stuff like that. | |
| Are we...is this...is this done yet? It's probably done, yep. | |
| So we get our Green checkmark. Now it's going to run on all of the systems. | |
| If we can go back to here, you'll just see it. | |
| Open JMP. It'll run some tests, | |
| probably will open Big Class. | |
| Then close all...close itself all down. | |
| Wayne Levin | So we're doing this, largely because |
| many of our clients have different versions of JMP deployed and they want a particular add-in but they're | |
| running it, they have, you know, just different versions out there in the field. We also test against the early adopter versions of JMP, which is a service to JMP because we report bugs. | |
| But also for the clients, it's helpful because then they know that they can...they can upgrade to the new version of JMP. They know that the applications that we built for them have been tested. | |
| And that's just routine for us. | |
| Good. | |
| Vince Faller | You're done. You're done. You're done. |
| Change to... | |
| I can talk about... | |
| And this is just going to run, we can | |
| movie magic this if you want to, Meg. | |
| Just to make it run faster. | |
| Basically, I just want to get to staging but it takes a second. | |
| Is there anything else you have to say, Wayne, about it? | |
| Cool. | |
| I'll put that... | |
| Something I can say, | |
| when we're staging, | |
| we also have our documentation in mk docs. So it'll actually run the mk doc's version, render it, put the help into the help files, and basically be able to create a release for us, so that we don't have to | |
| deal with it. | |
| Because creating releases is just a lot of effort. | |
| Encrypting. It's almost done. | |
| Probably should just have had one pre loaded. | |
| Live demos, what are you gonna do. | |
| Some. | |
| Run. | |
| Oh, one thing I definitely want to do. | |
| So, the last thing that the pipeline actually does is checks that we actually spent our time, because, you know, if | |
| we don't actually record our time spent, we don't get paid, so forcing us to do it. | |
| Great, great time. | |
| Vinde Faller | So |
| Vince Faller | the job would have failed without that. |
| I can just show some jobs. Trying. | |
| That's the docker one. We don't want that. | |
| So | |
| You can see that gave us our successes. | |
| No failures. No unexpected throws. That's all stuff from Hamcrest. | |
| Come on. One more. | |
| Okay got to staging. | |
| One thing that it does it creates the repositories. | |
| It creates them fresh every time. | |
| So it's like, it tries to keep it in a sort of stateless | |
| way. Okay, we can download the artifacts now. | |
| And now we should have this pipeline demo. | |
| I really wish it would have just went there. | |
| What. | |
| Why is Internet Explorer up? So now you'll see pipeline demo | |
| is a JMP add-in. | |
| If we unzip it. | |
| If you didn't know, a JMP add-in is just a zip file. | |
| If we look at that now, | |
| you can see that has all of our scripts in it, it has our foo, it has our bar. If we open those, | |
| open, | |
| you can see it's an encrypted file. | |
| So this is basically what we would be able to give to the customer | |
| and not have so much mechanical work. Wayne mentioned that | |
| it's less frustrated developers, and personally, I think that's an understatement, because doing this over and over was very frustrating before we got this in place, and this has helped a bunch. | |
| That. | |
| Wayne Levin | Now, about the encryption, when you're delivering an add-in |
| for use by users within a company, you typically don't want, for security reasons and so on, you don't want them to anyone to be able to go in and | |
| deal with the code. You know, that sort of thing, so we may deliver a code unencrypted just for, you know, so the client has their code on encrypted, but for delivery to the end user, you typically want everything encrypted, just so it can't be tampered with. Just one of those sort of things. | |
| Vince Faller | Yep, and that is the end of my demo. Wayne, if you want to take it back for the wrap up. |
| Wayne Levin | Yeah, terrific. |
| Sure, thanks, very much for that, Vince. So there's a lot of moving parts in this whole system so it's, you know, basically, making sure that we've got, | |
| you know, code being developed by multiple users that are not colliding with one another. We're building in the documentation at the same time. And actually, the documentation gets deployed with the application and we don't have to weave that in. It's... | |
| We set the infrastructure up so that it's automatically taken care of. We can update that, along with the code comprehensively, simultaneously, | |
| if you will. The Hamcrest tests that are going on, each one of those functions that are written, | |
| there are expected results, if you will. So they get compared and so we saw, briefly, there was, I guess, some problem with that equation there. An R square or whatever came back with a different value, | |
| so it broke, in other words, to say hey, something's not right here; I was expecting this output from the function | |
| for a use case. So that's one of the things that we get from clients, so you know, we build up a pool of use cases that get turned into Hamcrest tests and away we go. There are some other slides here that | |
| are available to you, like, when you, when you, if you go and download the slides. So I'll leave that available for you and here's a little picture of of the pipeline | |
| that that we're employing and a little bit about code review activity for developers too. If you want to to go back and forth with it. Vince, do you want to add anything here about how code review and approval takes place? | |
| Vince Faller | Yeah, so inside of the merge request |
| it will have the JSL code on the diffs of the code. And again, a big | |
| thank you to | |
| the people who did Hamcrest, as well, because they also started a lexer(?) for GitHub and GitLab to be able to read JSL, so actually this is inside of getlab. | |
| And they can also read the JSL. It doesn't execute it, but it has nice formatting. It's not all just white text, it's it's beautiful there. We will just go in, | |
| like in this screenshot, you click a line, you put in a comment that you want, and it becomes a reviewable task. So we try to do as much inside of GitLab as we can | |
| for transparency reasons, and | |
| once everything is closed out, you can say yep, my merge request is ready to go. Let's put it into the master branch, main branch. | |
| Wayne Levin | Awesom. So you're really it's...it's helping, you know, we're really defining coding standards, if you will, and |
| I don't like the word enforcement but that's what it what it amounts to. And it reduces variation. It makes it easier for multiple developers, if you will, to understand what | |
| what others have done. And as we bring new developers on board, they come to understand the standard and they know what to look for, they know what to do. | |
| So it...it makes onboarding a lot easier, and again it deals with...everything's attached to everything here, so you know supportability and so on. | |
| This is the slide I mentioned earlier, just for some resources so we're using GitLab. I suppose the same principles applied to any git generally so like GitHub or what have you. Here's the community link for Hamcrest. There was a talk at | |
| in tucson, that was in 2019, in the old days when we used to travel and get together. That was a lot of fun. And here's the the marketplace link for the visuals...visual code studio. | |
| Visual studio code, what have you. | |
| So as Vince said, yeah we make a lot of use of that editor, as opposed to using the built-in JMP editor just because it's all integrated. It's... | |
| it's just all part of one big application development environment. And with that, Vince and I, on behalf of Vince and myself, I want to thank you for your your interest in this, and want to, again we really want to thank the JMP team. Justin Chilton and company, | |
| I'll call out to you. If not for Hamcrest, we would not be on this. That was | |
| the missing piece or was the enabling piece that really allowed us to to take JSL development to, | |
| basically, the kinds of standards you expect in code development, generally, in industry. So we're really grateful for it, and I know our... | |
| you know, that that is propagated out with each application we've deployed. And at this point, Vince and I are happy to take any questions that... | |
| info@predictum.com and it'll get forwarded to us and we'll get back to you. But at this point, we'll open it up to Q&A. |
