topic Re: Replace missing values with 0 in Discussions

Replace missing values with 0

nshivan — Fri, 27 Jul 2018 22:11:47 GMT

Hello,

I'm trying to replace all missing values in a data table with 0. The below script doesn't work. I use JMP 13. Is there any other way to do this?

dt = Current Data Table();
nc = dt << get column names( Numeric );
For( i = 1, i <= N Items( nc ), i++,
  nc[i][dt << get rows where( Is Missing( nc[i][] ) )] = 0
);

Thanks!

Re: Replace missing values with 0

txnelson — Sat, 28 Jul 2018 00:25:16 GMT

Here is the correct syntax for what you are trying to do

dt = Current Data Table();
nc = dt << get column names( Numeric );
For( i = 1, i <= N Items( nc ), i++,
	Column( nc[i] )[dt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0
);

Re: Replace missing values with 0

nshivan — Thu, 30 Jan 2020 19:33:05 GMT

Hi Jim,

I tried the above, but getting the folliowing error.

could not find column in access or evaluation of 'Column' , Column/*###*/(nc[i])

Any idea what's going on? I use JMP 13, I'm not sure if the syntax isn't working for this version.

Thanks,

Re: Replace missing values with 0

cwillden — Mon, 30 Jul 2018 15:59:50 GMT

Hi @nshivan,

I ran Jim's code in versions 13 and 14 and got no errors. Maybe restart JMP and run it again?

Re: Replace missing values with 0

nshivan — Mon, 30 Jul 2018 16:32:36 GMT

Hi Cameron,

I restarted JMP and the script works fine on the specific data table. The missing values get replaced with 0s. But when I integrate to the rest of my script, it gives me the below error:

could not find column in access or evaluation of 'Column' , Column/*###*/(nc[i])

Prior to this piece of script, I have an original data table which I subset based on selected columns, stack the data, summarize anf then tabulate into the final data table. I'm trying to replace missing values in the final data table with 0s.

I'm fairly new to JMP scripts, not sure why the integration is giving problems. Any idea why this is happening?

Re: Replace missing values with 0

cwillden — Mon, 30 Jul 2018 19:48:52 GMT

Something like this is hard to diagnose from a description. You'd probably have to post your script (or at least enough of it to replicate the problem) for us to help with it.

Re: Replace missing values with 0

Byron_JMP — Wed, 01 Aug 2018 20:31:22 GMT

try being a little more specific in your script.

give the data table that has the specific column in it a handle. Like dt=table(); or something like that.

then refer to the column in the table, dt::nameOFCol

It is likely that your script is trying to modify a column that doesn't exist in the table that is current when this step runs.

Re: Replace missing values with 0

Evan_Morris — Fri, 04 Oct 2019 14:40:02 GMT

I use this script constantly. Can't thank you enough for providing it.

Re: Replace missing values with 0

geoffrey_mann — Fri, 08 Jan 2021 03:40:45 GMT

Hi all,

I have been working on this exact problem with a hundreds of columns with hundreds of missing rows. I wasn't happy with the performance and tried many approaches. It seems that Recode solves this problem about 150 times faster than the current solution. Sharing the code with anyone else that might care about speed. Please let me know if you find something faster. Every millisecond counts! :)

replaceMissingStart = HP Time();

// replace missings with zeros
nc = dmdt << get column names( Numeric );

For( i = 1, i <= N Items( nc ), i++,
	col = Column(dt,nc[i]);
	colname = col << getname();

   	Eval(Parse(EvalInsert("\[
		dmdt << Recode Column(
			:Name("^colname^"),
			{Map Value( _rcOrig, {., 0}, Unmatched( _rcNow ) )},
			Update Properties( 1 ),
			Target Column( :Name("^colname^") )
		);
	]\")));
	
	//Code from the Community
	//Column( nc[i] )[dmdt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0;

	// standardize column attributes, change all to nominal
	col << Set Modeling Type( "Nominal" ) 
);

replaceMissingStop = HP Time();
totalReplaceTime = replaceMissingStop-replaceMissingStart;
show(totalReplaceTime/1000000);

Thanks to Rebecca Lyzinski for the idea!!!

Re: Replace missing values with 0

ian_jmp — Fri, 08 Jan 2021 12:23:35 GMT

FWIW, this sort of approach might be better. For 10,000 by 10,000 it took about 10 seconds on my laptop (including making the starting table):

NamesDefaultToHere(1);

// Size the problem . . .
nr = 10000;		// Number of rows
nc = 10000;		// Number of columns
nm = 0.1;		// Proportion of missing values

// Make a table and inject some missing values
mat = J(nr*nc, 1, RandomNormal());
mat[RandomIndex(nr*nc, nr*nc*nm)] = .;
dt = AsTable(Shape(mat, nr, nc));
dt << setName("Data with some missingness");

// Start here: Get values into a matrix (assumes contiguous numeric columns)
mat = dt << getAsMatrix;
// Reshape as a vector
mat = Shape(mat, nr*nc, 1);
// Replace missing
mat[Loc(mat, .)] = 0;
// Reshape back to a matrix
mat = Shape(mat, nc, nc);
// Make a new table
dt2 = AsTable(Shape(mat, nr, nc));
dt2 << setName("Data with missing set to zero values");

Re: Replace missing values with 0

Craige_Hales — Fri, 08 Jan 2021 14:15:20 GMT

Combining solutions, with data table subscripting...

nr = 10000;		// Number of rows
nc = 10000;		// Number of columns
nm = 0.9;		// Proportion of missing values

// Make a table and inject some missing values
mat = J(nr*nc, 1, RandomNormal());
mat[RandomIndex(nr*nc, nr*nc*nm)] = .;
dt = AsTable(Shape(mat, nr, nc));
dt << setName("Data with some missingness");


replaceMissingStart = HP Time();

nc = dt << get column names( Numeric );
mat = dt[0,nc];
mat[Loc(mat, .)] = 0;
dt[0,nc] = mat;


replaceMissingStop = HP Time();
totalReplaceTime = replaceMissingStop-replaceMissingStart;
show(totalReplaceTime/1000000);

About 10 seconds.

Re: Replace missing values with 0

geoffrey_mann — Fri, 08 Jan 2021 16:18:31 GMT

Dear Ian and Craige,

Thank you so much for your submissions. I ran some preliminary tests and Ian's solution is more than 4 times faster than Recode for this and Craige's solution is almost 10 times faster. If anyone would like to compare all 3 solutions, try the following:

//Recode solution

nr = 10000;		// Number of rows
nc = 10000;		// Number of columns
nm = 0.9;		// Proportion of missing values

// Make a table and inject some missing values
mat = J(nr*nc, 1, RandomNormal());
mat[RandomIndex(nr*nc, nr*nc*nm)] = .;
dt = AsTable(Shape(mat, nr, nc));
dt << setName("Data with some missingness");

replaceMissingStart = HP Time();

// replace missings with zeros
nc = dt << get column names( Numeric );

For( i = 1, i <= N Items( nc ), i++,
	col = Column(dt,nc[i]);
	colname = col << getname();

   	Eval(Parse(EvalInsert("\[
		dt << Recode Column(
			:Name("^colname^"),
			{Map Value( _rcOrig, {., 0}, Unmatched( _rcNow ) )},
			Update Properties( 1 ),
			Target Column( :Name("^colname^") )
		);
	]\")));
	
	//Code from the Community
	//Column( nc[i] )[dmdt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0;

	// standardize column attributes, change all to nominal
	//col << Set Modeling Type( "Nominal" ) 
);

replaceMissingStop = HP Time();
totalReplaceTime = replaceMissingStop-replaceMissingStart;
show(totalReplaceTime/1000000);

Russ Wolfinger had mentioned he thought matrices would solve the problem more quickly, I guess that was what you two were thinking as well!

Re: Replace missing values with 0

Goblin_King — Wed, 11 Jan 2023 13:10:59 GMT

in Can I replace missing values in Numeric column with "N/A" string?

Re: Replace missing values with 0

mikedriscoll — Wed, 11 Jan 2023 14:09:11 GMT

>> Can I replace missing values in Numeric column with "N/A" string?

Use value labels to do that. Right click on column header, go to column properties, select Value Labels. Then put a period in the value and N/A in the label field as shown below. Then click add, and then ok.

To script it, follow the log recorder of above steps...

Data Table( "DT name" ):Column 904 << Value Labels( {"." = "N/A"} ) << Use Value Labels( 1 );