topic Re: Fast way to check for "wrong sensordata" in Discussions

Fast way to check for "wrong sensordata"

MWalther — Fri, 09 Jun 2023 22:11:28 GMT

I have many different sensors (>>100) with many Data (continuous) each (>1mio) and update these each week.

Sometimes these sensors "glitch" and send wrong data (below/above a low/high threshold)

I want to calculate the number of "glitches" per week (KW=categorial) and need a fast way by JSL code.

Example: negative data for this sensor should be detected (low threshold 0)

My Solutions are too slow:
1.) I set the threshold limits as "specification" in column properties and then perform process capability
"Observed outside" Spec per KW is the desired value, but it is quite slow

2.) Brute force: each row, each column, manual check, adding each cell (Excel style) takes forever.

Is there a more efficient way to do this?

Re: Fast way to check for "wrong sensordata"

txnelson — Tue, 20 Apr 2021 13:07:07 GMT

I suggest that you use the Range Check column property. It is quite efficient.

Re: Fast way to check for "wrong sensordata"

P_Bartell — Tue, 20 Apr 2021 14:03:20 GMT

In addition to @txnelson 's suggestion, have you tried out the Explore Outliers utility as a means to finding the rows containing the suspicious observations? Here's a link to the utility in JMP documentation: Explore Outliers Utility

Re: Fast way to check for "wrong sensordata"

MWalther — Fri, 23 Apr 2021 09:16:45 GMT

Thanks, sounds feasible.
So the way to go is use range check to set all data to missing, which do not comply, and the do a tabulate to "count" the missing values per calenderweek.

The only drawback ist, that all "wrong" data then are lost in the original data table.
I will need to use a copy, or is there a way to retrieve the (then) missing data in the original table?

Yours, Marten

Re: Fast way to check for "wrong sensordata"

ih — Tue, 27 Apr 2021 18:51:19 GMT

Yet another idea: If you know the bad data is always a negative number, or below some specific value, you could make a new column which is 1 if the values is 'bad' and then use tabulate to show the number of bad values for each level of KW.

Re: Fast way to check for "wrong sensordata"

txnelson — Wed, 28 Apr 2021 03:42:04 GMT

You will need to create a complete Subset
Tables=>Subset