Automatically selecting outliers

[#1]

I have 7000 rows, that consists of multiple batches.   I am doing a distribution by batch and using the outlier box plot to look for outliers.  But I have 100+ batches, is there a way to have JMP automatically select which rows are outliers?

[#2]

when I ran this, it made a subset table with the first batch only. I
changed this to do my open table:

dt = currentdatatable();
byvarlist = Associative Array( dt:Batch ) << get keys;
For( i = 1, i <= N Items( byvarlist ), i++,
dt << select where( dt:batch == byvarlist[i] );
subdt = dt << Subset( Selected Rows( 1 ) );
dt << clear select;
subdt:Out << set formula(
If(
:y > Col Quantile( :y, 0.75 ) + (Col Quantile( :y, 0.75 )
-Col Quantile( :y, 0.25 )) * 1.5, 1,
:y < Col Quantile( :y, 0.25 ) - (Col Quantile( :y, 0.75 )
-Col Quantile( :y, 0.25 )) * 1.5, 1,
0
)
);
subdt << run formulas;
Column( subdt, "Out" ) << delete formula;
keep = {"it", "Out"};
todelete = subdt << get column names( "string" );
For( icol = 1, icol <= N Items( keep ), icol++,
Remove From( todelete, Contains( todelete, Keep[icol] ), 1 )
);
subdt << delete columns( todelete );
dt << Update( With( subdt ), Match Columns( :it = :it ) );
Close( subdt, No Save );
);
selected = dt << select where( :Out == 1 );
selected << Markers( 11 );
selected << Marker Size( 8 );
selected << Row Colors( red );
dt << clear select;

[#3]

the column name is case sensitive, be consistent when using :Batch or :batch

[#4]

My batch column is capitilzed:

[image: image.png]

[#5]

in the for loop it isn't

[#6]

I am not sure if you got my last message I just sent, I got a weird
automated email back. I fixed the second one and it is still doing the
same thing.

[#7]

this worked for me.

at the end you should have 3 tables. original with all rows, one with just outliers and the last without outliers. let me know if it works.

Names Default To Here( 1 );

// make a sample data table
dt = New Table( "Data",
    add rows( 10000 ),
    New Column( "it", Numeric, Continues, Format( "Best", 8 ), Formula( Row() ) ),
    New Column( "Batch", Numeric, Ordinal, Format( "Best", 8 ), formula( Random Integer( 110 ) ) ),
    New Column( "y", Numeric, Continues, Format( "Best", 8 ), formula( Random Lognormal( :Batch / 10, 5 ) ) ),
    New Column( "Out", Numeric, Ordinal, Format( "Best", 8 ), formula( 0 ) ), 

);

dt << run formulas;
Column( dt, "it" ) << delete formula;
Column( dt, "Batch" ) << delete formula;
Column( dt, "y" ) << delete formula;
Column( dt, "Out" ) << delete formula;

// now we get to work
dt = Current Data Table();
byvarlist = Associative Array( dt:Batch ) << get keys;
For( i = 1, i <= N Items( byvarlist ), i++,
    dt << select where( dt:Batch == byvarlist[i] );
    subdt = dt << Subset( Selected Rows( 1 ) );
    dt << clear select;
    subdt:Out << set formula(
        If(
            :y > Col Quantile( :y, 0.75 ) + (Col Quantile( :y, 0.75 ) - Col Quantile( :y, 0.25 )) * 1.5, 1,
            :y < Col Quantile( :y, 0.25 ) - (Col Quantile( :y, 0.75 ) - Col Quantile( :y, 0.25 )) * 1.5, 1,
            0
        )
    );
    subdt << run formulas;
    Column( subdt, "Out" ) << delete formula;
    keep = {"it", "Out"};
    todelete = subdt << get column names( "string" );
    For( icol = 1, icol <= N Items( keep ), icol++,
        Remove From( todelete, Contains( todelete, Keep[icol] ), 1 )
    );
    subdt << delete columns( todelete );
    dt << Update( With( subdt ), Match Columns( :it = :it ) );
    Close( subdt, No Save );
    Wait( 0.00001 );
);
selected = dt << select where( :Out == 1 );
selected << Markers( 11 );
selected << Marker Size( 8 );
selected << Row Colors( red );
dt << clear select;

dt << select where( dt:Out == 1 );
Outdt = dt << Subset( Selected Rows( 1 ) );
dt << clear select;
    
Wait( 0.00001 );
    
dt << select where( dt:Out == 0 );
indt = dt << Subset( Selected Rows( 1 ) );
dt << clear select;

[#8]

I copy everything from //now we get to work on down. When I run this in
my own table, it just creates one table with one batch only. I must say
one thing, is that the number of points per batch are not consistent. I do
not know if that matters.

[#9]

if you run the whole script can you see that you get what you wanted (just not on your data)?

[#10]

my formula is not sensitive to batch size. furthermore, it will work even if batch column is character (and not numeric).
can you post a sample, even with just 2 or three different batches?

[#11]

When I run it, I get one table, and here is a snapshot of it, this will
work since I can select rows where Out = 1, if I want to select all of the
outliers.

[image: image.png]