Structuring Curve and Spectral Instrument Data for Exploration and Modeling (2026-US-30MP-2923)

Hello everyone, I'm Ryan Cooper, the JMP Systems Engineer for Southern California, and I'm presenting today with Bill Worley. He's a Principal Systems Engineer here at JMP, who works closely with many of our larger chemical customers.

One thing we've run into quite a bit working with customers is that instrument software is usually great at collecting data, but it's not always great at helping you get the most value out of it due to analysis limitations. There's often a lot more information that you can obtain from spectra, chromatography, and other curve-based measurement outputs.

The main thing we want to show today is that getting the data into the right structure can unlock some really powerful visualization, modeling, and advanced analytical techniques in JMP and JMP Pro. Once you've built a repeatable workflow, those techniques become even more accessible and practical to apply on a regular basis. We'll show a couple of examples using NMR and EQE data, but the ideas apply to a wide range of instrument outputs. With that, I'll hand it over to Bill to get us started with a few key takeaways to keep in mind as we go through the presentation.

Thanks, Ryan. To start off with, the reason we're sharing this is because, as Ryan said, you may have different endpoints for importing data, visualizing, and then sharing or modeling data. We wanted to show you how to build this structure and then go from there. Importing raw files is one of the important parts of everything. This is an Excel file that has been imported directly, and we've got the NMR data in columns. Each spectra has its own column right now, and then the PPM data or the chemical shift data is what we're calling NMR ID right now.

Then if you look over here to the right, we've taken this data and put it into a structure where we're looking at individual spectra, and if you want to then you can also stack that data where you're seeing all the spectra together. Then from there, you're going to keep cleaning and reshaping the data to your needs. Then you can look at, in this case, we've got a Trellis plot, where we've got all the spectra in their individual, we'll say, spots, and then we'll go from there. Then you can take that same data and then, again, look at individual spectra or combine several spectra together, and look at those in a way that makes sense to you. In this case, we've got three spectra stacked together right here.

Then going on from there, we've got the cleaning and reshaping. In this case, we've taken the data that was in a stack format, and now we've taken it and put it into a wide format. As you can guess, that gets pretty big. We've got over 550 columns of chemical shift data for each spectra, right? Each spectra is one line, one single line.

Now we've got that wide format. This allows us to get into modeling the data, and this structure leads to something what we call Functional Data Explorer. If you haven't heard of it, this is one way that you can really learn from the shapes, the features of your data. As you see the GIF down below, we've taken this data for the propanol, butanol, and pentanol, added that to the Functional Data Explorer, and now as you change the amount of any of these alcohols, you can see the structure of the spectra change. That's very helpful in understanding how… Just building a predictive model that you might need.

The nice thing about this is that, and Ryan did a wonderful job on this, is he take all that information and put it into one single workflow, Workflow Builder. It takes the raw files, brings those in, cleans and reshapes them, then you can do the visualization, then you can do the modeling. This allows for a repeatable analysis, and it beats manual preparation, doing manual preparation time and time again. What you want to look at is, and Ryan is going to show this as we go forward, but these Workflow Builders are going to be very important to your analysis as you move forward.

Our JMP analytic workflow is the basis for our Workflow Builder. If you look at bringing data in or finding your data sources, bringing the data in, cleaning it and blending it, and then doing some exploration and visualization, you've got all these tools at your fingertips. You can take that from there and then go to advanced modeling and then automation. All of that is what this whole structure of the Workflow Builder is built on. You can use any of these different platforms that we're talking about in there, but for the most part, what you see highlighted today is what we'll be talking about.

Like we've been saying, curves should be analyzed as curves. Shapes contain information. Instead of analyzing individual points, which you can do in JMP through partial least squares or principal component analysis, you're going to take these and look at the individual curves or all the curves together, model them, learn from them, learn from the shapes and features, and then be able to predict outcomes from entire curves. Now I'm going to turn it back over to Ryan. He's going to share some examples of the NMR and the EQE Workflow Builder.

Yes. Thanks, Bill. Before we start the interactive demonstration, I wanted to go over quickly what both of these examples are. The main example I'm going to focus on today is NMR, nuclear magnetic resonance. It focuses on the atomic nuclei inside a molecule applying radio waves in order for the nuclei to absorb energy and resonate where the frequency matches their specific chemical environment. Then over here on the right, we've got EQE, which is external quantum efficiency. It's popular in solar cells and photonics. What we're doing is converting incoming light particles into usable electric current.

Just a couple of examples. Of course, you may have some of your own as well in how curve data, spectral data works. All right. Let's go ahead and go to JMP. The first thing we're going to look at is the data in its raw form. This is an Excel file, and this format actually comes from a customer of one of ours. You can see that there are responses down here to each of the NMR IDs in each column here. Then there's also concentrations up here, propanol, butanol, and pentanol, for each of the NMR IDs. What we've got to do is find a way to import the data that's down here, the response data, as well as for further analysis, combine that with our propanol, butanol, and pentanol variables.

Let's take a look at how we might import this in JMP. I'm going to go to File, Open, and I'm going to find that file. I actually need to close out of it, because if I have it open, JMP won't recognize the import. I closed out of it, and now I'm going to open it up. You'll see the default settings here in the Excel Import Wizard. You'll see what you saw before with propanol, butanol, and pentanol up here on the top. What I'm going to do is select the spectral data worksheet, and I'm going to say that the data starts on this row. Data starts on this row. I can click the plus sign here in order to get rid of that initial portion that we're going to use later.

You'll see there're some options for using all worksheets if you'd like. Using these settings on all worksheets, you don't necessarily have to do that. If you have multiple worksheets, you can concatenate them as well. There're some other settings over here on the next screen as well in order to shape this data. I'm going to go ahead and click Import. We now have our Excel data imported into JMP. I'm going to make a couple of changes here. I'm going to call the NMR ID X instead, since the NMR ID is describing what's going on in the columns. X, I'm just going to describe what's going on with our responses.

The other thing that I notice when I import this is this column got imported as a category, but this is actually continuous, and it's numerical, just like the other ones. I need to go in here and change the modeling type as well. I can do that by right-clicking and going to Column Info and changing to Numeric and Continuous. Now we've got this into the format that we'd like.

From here, we can start making some graphs. Let me do that. I'm going to go to Graph and Graph Builder. The first graph I'm going to make is I'm just going to pull my X down here to X, and just one of these NMR ID, I'm going to pull over to the Y. We'll get the default output from Graph Builder, but I can change this into a nice line graph here so you can see all the peaks and everything. I can click done here and just see one of the NMR IDs, but I can also go to the red triangle, Redo and Column Switcher.

I can select all of my columns here. I'm replacing the Y-axis with the rest of these. Now what I can do is toggle through each of my NMR IDs and see what that looks like interactively here.

Another graph that I can make, if I go to Graph and Graph Builder, I could pull all of my NMR IDs over here to the Y-axis and my X to the X and do the same thing with the line plot, and click done. From here, if I click on any of the items in the legend, it doesn't really light anything up. If I highlight any of these points, we really can't do a whole lot with this. There's a limitation with how the data is currently set up and what I can do to visualize it.

At this point, what I need to do is transform the data in a way where we can get more out of our visualizations. Let's go ahead and do that. I'm going to take my table and use a function called Stack. I'm going to put all of my NMR ID columns into the Stack. You can see the output in the preview here. I'm going to call my label column the NMR ID, and the data column my response. We'll go ahead and keep this output table name, but this is now in a good format for me to do some other things with it. Let's click Okay.

Another thing I want to do is just change the name here. I don't want these parentheses here. From here, what we can do is go to Graph, and Graph Builder, and we can make some of the same… We can make some plots that look very similar, but we're going to be able to do some more with it. If we look at Graph Builder here, I'm going to pull my X into my X just like I had before, but this time I'm going to pull response into Y. What this is doing is it's plotting everything. If I'd like to split things up by NMR ID, I could perhaps pull NMR ID over into the wrap portion of Graph Builder. You get the Trellis plot that Bill mentioned earlier. I can also change this to a line graph, so now you can see if I click done, all of these NMR ID responses on the same chart here.

Another graph that I can do from this particular format is if I go to Graph Builder, and I pull my X down here and my response over here, change back to that line, what it's doing right now is it's taking the mean response of each of my X's here for each of the NMR IDs. If I want to split this up and filter each one of those, what I'm going to need to do is set up a local data filter using my NMR ID, where I can now flip through each of my NMR IDs just like I did with the Column Switcher. This time it's with the local data filter because I have all my NMR IDs in one column here.

If I'd like to color these, I could select two at once. What it's doing right now is it's just taking the mean of each of the two NMR IDs, but if I pull my NMR ID over here into overlay, I can now select each one individually, and we can see a bit of an overlay of each of the NMR IDs. I can click done here, and you can see a nice little graph of each of these overlaid. As I continue holding the Control key down, you can see how more are added to the graph.

That's an example of a few of the things that you could do from this particular format of the data. Something else that we might want to do is get this into a format where all of our NMR IDs are on the left side, we have 23 of those. But we also have those process variables, those mixture variables that we might want to merge in and analyze all of that data together as well.

If I want to do that, I need to transpose this data table. There's a function for that as well under Tables. Go to Tables and Transpose, and what we're going to do is… Actually, sorry, we're going to do this from another data table, not this one. We're going to go back to the one that we used before. This is the one that imported from Excel. We're going to do it from there. We go to Tables, Transpose, and my transpose columns are going to be all of my NMR columns, and my label column is going to be my X. Then I want to rename my label column to NMR ID.

Now we've got this in a format where each of my responses is right here, and I've got each of my NMR IDs on a row with everything else following it. Now we're going to have a good format where once we import the process variables, we can join these in. Let's go ahead and import the… Actually, let's group these columns first. We can actually go over here into the columns and group everything together and rename this. I'm just going to call it X's.

From here, let me go ahead and import our process variables. We're going to do this from the exact same file that we used before. I'll restore the default settings so that we can see propanol, butanol, and pentanol at the top. I'm going to say that my data ends on this row, and that selection is over here on Next. Ends on row. I'm going to take in just the worksheet here and say Import.

From here, I just need to do a quick transpose again, like what we did before with the other table. I need to label with NMR ID once again. I'm going to call this DoE Alcohol Concentrations. I'm going to pull this from somewhere else. My label is actually going to be my NMR ID. You'll see row one, two, and three change over here.

Now I have this set up in a way where I have a couple of tables. Was that this one? I didn't need that. But now what I can do is join my NMR IDs together with the propanol, butanol, and pentanol. Let's do that from this table, and we'll use a function called Join to do that. I'm going to select my DoE Alcohol Concentrations as the table I'm joining with, and what I can do now is match on my NMR ID, pull these into my selected columns for my Join table, we'll start to see an auto-refresh here, and then I want to pull all my X's after that.

You'll see in the preview here, now we've got the process variables next to the responses. If I click Okay here, now I've got my final data table. From here, we can do some really, really neat analysis. First thing I'm going to do is I can create a ternary plot just to show the DOE conditions that we had before analyzing the response. I can just pull the propanol, butanol, and pentanol over here, and you can see where these fall on a ternary plot if you want to see that. But the other thing that I can do, most importantly, is I can use some specialized modeling, and Bill mentioned it earlier, called Functional Data Explorer.

From here to set up Functional Data Explorer, what I need to do is go to my columns as my function, actually, my rows as my functions. I'm going to pull my Y column group as my X's, my ID as my NMR ID, and then my supplementary variables that help me predict what's going on with the spectra, those go in here. I can click OK. Now I have the functional data output here. From here, Bill is going to take it and explain what some of this output means and what you can do with it.

Ryan did all that work and put it into a really nice Workflow Builder, right? What I've done is I've imported that Workflow Builder into a project, and I've run it. You can see that every step has been run. I don't have to worry about doing any of the preparation that I would have if I had to do it from start or do it from scratch. Anyway, what it all comes down to is that we've got the analysis. I think I've got one more. I can get one more.

There we've got the Functional Data Explorer that Ryan had basically shown you the process for getting there, and then we're building a model from that. What we're doing is looking at a different type of modeling. We're looking at models that could be splines, it could be Fourier bases. What we're looking at right now are called wavelets. Wavelets are a special set of curve types or functions that allow you to model curve data.

From there, we've gone down here, and the best model overall is something called Haar, and it's got the lowest AICc or BIC. Now you can see down below here that each one of the spectra has been modeled and did very well. This model is doing very well with the data overall. Based on all of that, we can look at things called Shape Functions and see how the model or the different spectra are being modeled. This is a lot like principal components. You can say, how many Functional or Shape Functions are we using to get the best data out of this?

If you use that same rule of thumb that you can use in principal components, you say that 13 functional principal components or Shape Functions could give you a pretty good outcome when you're building your models. From there, we're going to take this down here and look at the score plot here. We can see that we've got more or less a ternary plot like Ryan showed before. Then we've also got that functional DOE Profiler. If we move this over here to one of the peaks, now we can start saying, "Okay, how do these peak shapes change as we're analyzing the curves?"

As we change the amount of propanol or butanol, and you can see the values change down here below as well because this is set up as a mixture. You can do that. Then overall, analyzing your curve data becomes very, very easy and very understandable, and you can build more generalized models off this as well. One of the things I've looked at doing is building a multivariate curve resolution model where you can actually determine the percentages of the different alcohols in the model directly within Functional Data Explorer. It's very helpful with mixtures. From that, I'll turn that back over to Ryan. Stop my share.

All right, let me share mine. Bill briefly showed this. This is the full workflow of the NMR DoE workflow, and you'll see how this is grouped. I've got that first section of graphs that I looked at followed by the second section. I basically recorded this with the record button within Workflow Builder as I did all of my steps. Now I can play it back. If I click the play button here, you'll see what's going on. It pulled the data in from the Excel file, and it's doing all of the things that we just did. It creates a nice repeatable workflow for you.

Moving on to the next example. We're going to look at the EQE data now. The difference between this example and the last one, the biggest difference is going to be the way that the files are imported. We're going to review that and some of the data preparation steps, and then largely from there, everything remains very similar to the way that we were looking at the NMR workflow. This particular data is actually located in a folder here called EQE files. There's .dat files for each one of my IDs.

In the previous example, we were looking at NMR IDs. Here we're looking at EQE IDs that are embedded in the file name here. If we look at one of those files, it looks a little something like this, where we have a wavelength in one column, and we have EQE in the other. But there's no indication of what the ID is, only in the file name here. We're going to need to find a way to merge that together so that we can analyze all of these files together, because it's really powerful when you're able to do that.

Let's take a look at how to import multiple files. If you go to File and Import Multiple Files, and you direct your folder to the folder where all your files are, we can see all of our files located here, and we can select files by name or extension. In this case, there's just the .dat files, but if you have something else located in here, you can add the extension so that it just pulls those.

There's also a selection to import files manually. I don't necessarily recommend doing that when you have a folder that you're consistently updating with new data. If I had more IDs that I was running and storing those in this folder and I use the filter, then I can automatically pull those into JMP and analyze all of those together without having to tell JMP that I want to add a certain file. I highly recommend keeping it on use filters for something like this.

I'm also going to add a file name column to make sure that I get this file name, and I'm also going to stack similar files. The other thing to note about this particular file is that we have tabular data. It's not comma-delimited. If I go to the settings here, you'll notice the end of field is defaulted to Comma. In this case, it's actually Tab. This will allow my columns to show up properly when I'm importing into JMP.

Let's go ahead and import this. You'll notice the wavelength, the EQE, and then the file name as well. If I open up the distribution here, you can see how many file names there are with all the wavelength and the EQE data stacked on top of each other.

From here, there's a few things that I might want to do. There's a function for the file name here called Text to Columns that I'm going to use to split this up. If I go to Columns and I go to Utilities and Text to Columns, I can specify a delimiter to split the file name on. I'm going to do that with Space for now, but you might be familiar with some file names that have underscores as well. That's probably the most common one that I've seen, where you can split up some of that metadata in the file.

We're going to use a space for this one, but keep in mind, you can do any other delimiter as well. Clicking OK, you see where we split on the space for each of these names. This is really, really helpful to get our ID out of here. Let's delete some of the ones that really aren't that helpful to me right now. The file name 1, 2, 5, looks like 7 as well, are the same all the way across here. I'm going to leave ID just so I can concatenate these together. I can delete these right now. The next thing I want to do is concatenate my file name 3 and 4 so that I have an ID number.

I can do that really, really easily just by right-clicking and saying New Formula Column and then going to Character and Concatenate with a Space. I'm going to do that. Now I've got my ID here. The other thing I want to do is concatenate it with this other column here, too. You'll see why here in a second. Let's do these three columns and do the exact same thing.

At this point, we have an ID column, and we also have an ID and cell column. At this point, I could get rid of the ones that I'm not using. I can delete the file name 3 and file name 4 and file name 6. JMP gives me an alert here that says that these columns are used in the file name here, or in the new columns. These references are used in the formulas. I can remove the formula so that now these columns are just used for their values, then I can delete the file name as well.

Now this is in a format where I can continue using it for graphing. I can create that Trellis plot that we looked at before, I can go attach some process conditions and look at it in Functional Data Explorer as well. I'm actually going to close out of this so that we can look at the workflow just all together.

If I click play here, we're going to go through to import multiple files, all the data prep steps, and watch it go into Functional Data Explorer as the process conditions are attached to it as well, in a similar way as what we saw before. You can see all this happening behind the scenes, and then we get to the point where we've got our process conditions attached to the result here.

What I like to do on this one is scan the wavelength here for anything that looks like it's influencing it. Right now, there's nothing really influencing it. All the bars on the other processes are pretty horizontal. But once I get to this drop off right here, it looks like this part of the curve is heavily influenced by the CIGS Temp over here. I could also vary this and see what different CIGS Temps could contribute towards basically different slopes of this line here, but the entire curve is involved in the modeling process here.

To close out, let's take a look at PowerPoint again, and some key takeaways. To close, the examples we showed today really come back to three ideas: how data structure enables analysis, workflows make advanced analytics more accessible, and how curves should be analyzed as curves. Through organizing data appropriately and creating repeatable workflows, it's possible to apply some really powerful analytic techniques to instrument data that might otherwise be difficult to use. As we saw in the examples, treating spectra and other curves as complete observations can reveal patterns that traditional point-by-point methods might miss. With that, let's finish up with a few supplementary extensions here. I'll pass it on to Bill for these.

Thanks, Ryan. A few things that we wanted to share with you. These are things that you can find on our JMP Marketplace. But being able to import files can be difficult, we've got a couple of extensions or add-ins that have been developed by our coworkers. One is called Can Open-er, File Importer, which allows you to, like, SPC files or SPC, and then you've got this Spectral Importer that was developed as well for pulling in chromatographic data or spectral data.

There's a newer one out there called FDE Model Screen. In FDE, you have a choice of models to build. Now you can let this Model Screen add-in go through all the different models to see which one is the best for you. We've got a few other add-ins. This spectral graphic one, Graphical Spectral Data, pyBaselines working with Python-type data, and then a couple other add-ins from our friends at Waters.

Now these are just things that you can pull in through the JMP Marketplace. They're free as of right now, but go ahead and use those to your heart's desire and learn from them and build on your analysis capabilities. With that, we'll say thank you. Ryan?

Yeah. Thank you for viewing our presentation and let us know if you have any questions down the line.

Thanks.