best way to solve a complex time series analysis?

[#1]

Hi there,

here is what I want to do (my JMP version is 19.0.3):

• I have a data set with >10 million rows, time series data
• I need to carry out a segmented fit for >1000 segments
• I know that I can do this in R, but not how to do this in JMP
• my aim is now:
  • have the data in JMP
  • carry out the segemented fitting in R
  • summarise the data into a new table
  • transfer the data back to JMP

The alternative is to work fully outside of JMP for the summary tables - but if this is possible I would like to try this within JMP and its R integration.

What would help me are: training resources / reading materials for R in JMP, and hints if you have done something like this already.

Thank you!

OverviewZoomDeep Zoom

[#2]

If I remember correctly, in JMP19, R integration is currently done via Python Integration. Here is one presentation of thatScripters Club Recording: R is Back Again in JMP 19 - Exploring R and Python Integration 

[#3]

Thank you @jthi, that's a good starting point!

[#4]

Hi @PurpleBat ,

To understand a bit more about what you're trying to achieve, what is the segmentation doing/what is the end goal? I segment the zones where there is activity (ie increasing Phil T1 Ave)?

If you can comment on the process in R or the package, it may be possible just to use JMP or even JSL to achieve the same thing.

Thanks,

Ben

[#5]

Hi Ben,

I want to extract from the segments the min and max, as well as the risetime (fitting as a first approximation a 1-exp(-t/tau)), and then use the parameter tau to describe the rise.

Once I have that, I want to look at the evolution of tau over time.

Does this help?

[#6]

Hi @PurpleBat ,

How are you defining the segments? Are they equal sizes (length of X)? Are they generated via a set package in R? I'm sure this can be done in JMP but need more detail.

For R specifically, the scripting index (Help > Scripting Index) has full details on the R integration and components needed to run it (see screenshot below)

Thanks,
Ben

[#7]

Hi @PurpleBat ,

There's also this page to working with R in JMP: https://www.jmp.com/support/help/en/19.0/index.shtml#page/jmp/work-with-r.shtml#ww1284119

Thanks,

Ben

[#8]

Once you have the segments labeled individually, it will be possible to extract the rise times using JMP's Fit Curve, for example. The tricky part is automatically detecting the segments. Please let us know how you define the start and end of each segment. For now, I'll assume that each segment begins/ends at each local maximum or minimum. 

Here's how you can tell JMP to automatically find a mark all the local max and mins, and label them in a new column. From there, the next step will be a script that uses those markers to uniquely label each segment. 

If you would like to upload your data table (or a representative sample data table), I could illustrate how to apply this approach directly to your data. 

[#9]

Thanks both @christian-z  and @Ben_BarrIngh ,

the segments are not regular, but they do start on a local minimum or maximum. I have taken a small example from the much larger data set for illustration purposes (reduced the size from 1.7GB to ~500kB). 
I will continue to work on this reduced dataset myself as well and then see how it ports to the larger one.

Many thanks for your great hints and help so far!

[#10]

@PurpleBat here is an updated version of your example table to get started. I added some formula columns and saved some scripts to the data table. Hopefully it is self-explanatory, but let me know if you have questions. 

I recommend using the data filter to work with a small subset of your table while debugging. Then, when you are ready to run the full analysis, be aware it may take a few moments for the full data set with many thousands of curves. 

Some ideas for next steps would be to filter for only the downward sloping curves (assuming those are the interesting ones), "aligning" the data so that it all starts at time=0, normalizing the curves if needed, etc.