topic Re: Functional Data: missing values imputation? in Discussions

Functional Data: missing values imputation?

matteo_patelmo — Tue, 05 Nov 2019 17:28:19 GMT

Hello, I have a collection of X,Y curves I would like to analyize with Functional Data Explorer.

The problem I'm facing is that some of these curves are truncated at a certain value of X because for greater values a physical failure occurs during data collection. Some other curves reach the maximum possible X with no failure.

FDE is appearently not managing very well the presence of curves with very different X range, the missing portion is fit with very unrealistic splines.

I do not want to truncate all curves to the same X range of the "short" ones, because this physical failure is part of the information I want to keep.

Of course I can manually (or by JSL) impute some predefined value to replace the missing ones, but I was expecting FDE data preparation to offer some handy way to manage "informative" missing portions of curves. I could not find any suggestion in the user manual (JMP14 or even JMP15).

Any idea how to to address this issue? Do I need to wait JMP16 :) ?

thanks a lot for your help!

Matteo

Re: Functional Data: missing values imputation?

Mark_Bailey — Tue, 05 Nov 2019 18:31:20 GMT

You should be able to work with data that is not 'dense' (all functions observed over the same, regular grid of domain values). You can have missing values at the end of the functions or in the middle. You should not have to impute data.

How much of a difference is there when you say "very different X range?"

Are you using P-splines? They often fit well but they can have trouble with either large gaps in X or with very different ranges. Using one of the transforms can help P-splines. Also, it is generally not necessary to use a P-spline model with a degree above 1. Higher degrees can cause problem with such data, too.

So you are using JMP Pro 15?

Re: Functional Data: missing values imputation?

matteo_patelmo — Tue, 05 Nov 2019 20:07:03 GMT

Hi Mark,

very different could mean that some curves are truncated at 60% of the total range. I'm using B-splines.

Here below an example of fitting one such truncated curves, I obtain an unnatural spike where data points are missing.

thanks
Matteo

Re: Functional Data: missing values imputation?

Mark_Bailey — Wed, 06 Nov 2019 20:47:42 GMT

I would say that this data will not be satisfactorily modeled and analyzed with function data analysis techniques. You might try some of the Processing commands to determine if the help (e.g., Transform, Align) but I think that the available models will not be able to handle the short-comings of the data.

Also, JMP R&D suggested using lower order splines. Part of the problem is that the number of knots applies to every function. The number and placement might not be good for every function in the sample. You can manually drag the diamond at the top of the data plot to adjust the location. The also suggest trying step function (degree=0) P-splines.

Do you know about the model controls? Are you using only the default set of B-spline models?

Re: Functional Data: missing values imputation?

matteo_patelmo — Wed, 06 Nov 2019 21:38:39 GMT

Thanks. I will try as you suggest but I stll think that an option allowing to impute data easily would be very useful.

Matteo

Re: Functional Data: missing values imputation?

Mark_Bailey — Thu, 07 Nov 2019 12:55:32 GMT

Imputing data is difficult and works best, if at all, under specific conditions. It requires sufficient non-missing data to support replacement by one method or another. It always comes down to interpolation within the the ranges and categories that already exist. Your case of incomplete functions would require extrapolation of a large portion of the function. That is more like forecasting. The imputed data would incur very high uncertainty, expecially near the end of the function.

So I think it is best to work with the data you have and try some of the processing options and the low order splines with carefully placed knots. The default model parameters alone might not satisfy your needs.