topic Is it possible to iterate through all elements in an associative array? in Discussions

Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 10:51:38 GMT

I would like to use a loop to iterate through all values in an associative array. For the life of me, I cannot find the interator function for associative arrays. Is it simply impossible to step through all values in an associative array if they are not at pre-known keys?!

In my associative array, the keys are strings and the values are lists. I need to iterate through the entire associative array, and for each key, read something from the value.

This is my (incomplete) script. I am a semi-veteran of some programming languages but I am a noob at JMP script.

Where I put comments inside /* */ is where there is code that I don't know how to do. I am stuck at /*For all parts in lastTests*/ near the bottom, though I don't know if the rest of my script works because I haven't tried running any of it yet.

By the way, the JMP scripting guide has a very brief section on associative arrays and it does not tell you how to iterate through elements within an associative array :(

Please help!

dt = Current Data Table();


lastTests = Associative Array();


For Each Row (
 
 partKey = Char(:Name("214:ReadOTP:OTP_Row_40[1]") + 256*:Name("215:ReadOTP:OTP_Row_41[1]") + 256*256*:Name("223:ReadOTP:OTP_Row_49[1]") + 256*256*256*:Name("224:ReadOTP:OTP_Row_4a[1]") + 256*256*256*256*:Name("123:LOG_SETPOINT_SPEA:SPEA_Setpoint[1]"));
 
 rowValue = {:Name("start_t"), :Name("part_id"), Row()};
 
 If (Is List(lastTests[partKey]) == 1,
 If ( rowValue[1] > lastTests[partKey][1], //if start_t is later
 lastTests[partKey] = rowValue,
 
 //else if
 (rowValue[1] == lastTests[partKey][1]) & (rowValue[2] > lastTests[partKey][2]), //if start_t is the same but part_id is larger
 lastTests[partKey] = rowValue;
 
 
 ),
 //else
 lastTests[partKey] = rowValue;
 
 );
 
 
);

listOfRowsToSelect = {};

/*For all parts in lastTests*/(
 selectedPart = /*get present item*/;
 Insert Into(listOfRowsToSelect, selectedPart[3]);

);

dt << Select Rows( listOfRowsToSelect );

/*Invert Selection*/;

/*Hide and Exclude selected rows*/;

dt << Clear Select;

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 06:05:55 GMT

As an example, this is how you iterate through all values in a table (named "t") in Lua, a common scripting language:

for key, val in pairs (t) do

<stuff>

end

What is the equivalent in JMP scripting language for associative arrays?

Re: Is it possible to iterate through all elements in an associative array?

txnelson — Wed, 14 Mar 2018 09:17:58 GMT

To find the functions and messages available to an Associative Array, go to

Help==>Scripting Index==>Associative Array

There you will find several ways of looping through the keys, or values by retrieving them into a list and then using a For() loop, etc. Below is an example taken directly from the examples given in the Scripting Index using "First" and "Next" to loop through an Associative Array

Names Default To Here( 1 );
Local(
	{aa = [1 => "bun",
	2 => "shoe",
	3 => "tree",
	4 => "door"], x, words = ""},
	x = aa << First;
	While( !Is Empty( x ),
		words = words || aa[x];
		x = aa << Next( x );
	);
	words;
);

Also, starting on Page 207 in the JSL Scripting Guide there are several pages of documentation on how to Create and Use Associative Arrays in JMP

Help==>Books==>Scripting Guide

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 15:59:33 GMT

Thank you! I did not know about the Help->Scripting Index!

It looks like in my case, I need to use the Get Values function (I don't need the keys).

Re: Is it possible to iterate through all elements in an associative array?

Mark_Bailey — Wed, 14 Mar 2018 17:34:31 GMT

Be aware that there is no ordering of the contents of the associative array. Indexing is out. That is the reason for the First and Next messages to iterate over the array. Using the Get Values message returns an ordered result, but the contents are not necessarily in this order.

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 18:19:35 GMT

Thanks Mark, IMO, this is how it should operate and it's the same as how similar data structures in other languages operate.

How does one test if an associative array has a value assigned to specific key? Since I am assigning specific "Char" (the same as string?) key values to lists, I first tried the Is List function, but it gave me an error. This was my script:

dt = Current Data Table();


lastTests = Associative Array();


For Each Row (
	
	partKey = Char(:Name("214:ReadOTP:OTP_Row_40[1]") + 256*:Name("215:ReadOTP:OTP_Row_41[1]") + 256*256*:Name("223:ReadOTP:OTP_Row_49[1]") + 256*256*256*:Name("224:ReadOTP:OTP_Row_4a[1]") + 256*256*256*256*Abs(:Name("123:LOG_SETPOINT_SPEA:SPEA_Setpoint[1]")));
	
	rowValue = {:Name("start_t"), :Name("part_id"), Row()};
	
	If (Is List(lastTests[partKey]) ==  1,
		If ( rowValue[1] > lastTests[partKey][1], //if start_t is later
			lastTests[partKey] = rowValue,
		
		//else if
		(rowValue[1] == lastTests[partKey][1]) & (rowValue[2] > lastTests[partKey][2]), //if start_t is the same but part_id is larger
			lastTests[partKey] = rowValue;
			
		
		),
	//else
		lastTests[partKey] = rowValue;
	
	);
	
	
);

listOfRowsToSelect = {};

listOfVals = lastTests << Get Values;

For(i = 1, i <= nrows(listOfVals), i++,
	Insert Into(listOfRowsToSelect, listOfVals[i][3]);

);

dt << Select Rows( listOfRowsToSelect );

And this is the error I see in the log:

subscripted key not in map["192344950106"]{1} in access or evaluation of 'partKey' , partKey/*###*/

In the following script, error marked by /*###*/
dt = Current Data Table();
lastTests = Associative Array();
For Each Row(
	partKey = Char(
		:Name( "214:ReadOTP:OTP_Row_40[1]" ) + 256 *
		:Name( "215:ReadOTP:OTP_Row_41[1]" ) + 256 * 256 *
		:Name( "223:ReadOTP:OTP_Row_49[1]" ) + 256 * 256 * 256 *
		:Name( "224:ReadOTP:OTP_Row_4a[1]" ) + 256 * 256 * 256 * 256 *
		Abs( :Name( "123:LOG_SETPOINT_SPEA:SPEA_Setpoint[1]" ) )
	);
	rowValue = {:start_t, :part_id, Row()};
	If( Is List( lastTests[partKey/*###*/] ) == 1,
		If(
			rowValue[1] > lastTests[partKey][1], lastTests[partKey] = rowValue,
			rowValue[1] == lastTests[partKey][1] & rowValue[2] > lastTests[partKey][
			2], lastTests[partKey] = rowValue
		),
		lastTests[partKey] = rowValue
	);
);
listOfRowsToSelect = {};
listOfVals = lastTests << Get Values;
For( i = 1, i <= N Rows( listOfVals ), i++,
	Insert Into( listOfRowsToSelect, listOfVals[i][3] )
);
dt << Select Rows( listOfRowsToSelect );

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 18:20:30 GMT

I also tried the Is Empty function (switching the 1 to a 0 in the comparison as well), and it gave me the same error.

Re: Is it possible to iterate through all elements in an associative array?

pmroz — Wed, 14 Mar 2018 18:37:52 GMT

The getkeys method returns all of the keys for an associative array.

aa = associative array();
aa["One"] = {1, 2, 3};
aa["Two"] = {"a", "b", "c"};
key_list = aa << getkeys();

// Results shown in the log:
{"One", "Two"}

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 18:46:03 GMT

Yes, but this script and its derivatives I plan to make need to operate on a data table with well over three hundred thousand rows, each with hundreds of columns (if my computer can't handle that; if not, then I'll have to break it up some, but the table size for each database fragment will still be HUGE).

So, it is critically important to optimize this code so that it runs as efficiently as possible. Running Get Keys on the associative array and then seaching the huge list of tens or hundreds of thousands of entries simply to see if the returned list contains a value seems HORRIBLY inefficient when all I need to do is see if lastTests[partKey] has a value assigned to it or not!

Re: Is it possible to iterate through all elements in an associative array?

pmroz — Wed, 14 Mar 2018 19:09:06 GMT

Understood. I did a test with an associative array with 20,000 keys. I used a contains search to look for each of the 20,000 keys; took 12 seconds. Looking for an individual key takes a fraction of a second.

// Look for one key
start = hp time();
zlist = aa << getkeys();	// Contains 20,000 keys
a     = "xyz9999";
flag  = contains(zlist, a);
end   = hp time();
show((end - start)/1000000);
// .012 seconds

// Search for all keys using CONTAINS
start = hp time();
for (i = 1, i <= nitems(xlist), i++,   // xlist contains the keys in order of creation
	x = xlist[i];
	b = contains(zlist, x);        // zlist comes from getkeys and is not in order of creation
);
end = hp time();
show((end - start)/1000000);
// 12.2 seconds

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 19:43:17 GMT

So, do you know for certain that there is no way to test an associative array to see if a key exists without using the Get Keys function? Does an associative array always generate an error when you attempt to access a key that doesn't exist yet?

12 ms is going to be very problematic, that is a tremendously long time just to determine if a value exists at a certain key. I'm pretty certain other scripting languages do this in microsecond time scales; I know that Lua does it incredibly fast (compared to 12ms, at least). If I'm dealing with a data table of 300k rows, then actually that could be about 300k keys. So the execution time could increase to (300k/20k)*12 ms = 180 ms. BUT, that would then have to run about 300k times, so the total execution time becomes 15 hours (180 ms * 3e5 = 15 hours). Actually, it would probably be about half of that, since the associative array would be growing as the loop runs. But 7.5 hours is still too long.

However, perhaps it is not the Contains function which chews up the majority of the execution time (within the 12 ms) but the Get Keys function? In my script, I wouldn't even have to run Get Keys at all if I just keep (in parallel) a separate List of keys that I had added into the associative array.

Re: Is it possible to iterate through all elements in an associative array?

pmroz — Wed, 14 Mar 2018 19:52:28 GMT

You could wrap a simple test for the key value in a try() section. Not sure of the performance vs contains.

found = 0;
try(
	b = aa["junk"];   // This key is not in the associative array
	found = 1;
);
show(found);   // Returns 1 if found, 0 if not found.  1 is the same as TRUE, 0 is the same as FALSE

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Wed, 14 Mar 2018 19:57:33 GMT

Cool. I was hoping JMP had something like Try. I will try Try :) and compare its execution time vs Contains when I get the time later. Thank you very much for the help. I will post updates when I have them.

Re: Is it possible to iterate through all elements in an associative array?

tatarjj — Thu, 15 Mar 2018 00:42:36 GMT

I think I got it working finally. I wish I could assign values to a list in the same line that I constructed it, and get rid of all the Insert Into commands. Do I really have to do this:

	rowValue = {};
	Insert Into(rowValue, :Name("start_t"));
	Insert Into(rowValue, :Name("part_id"));
	Insert Into(rowValue, Row());

When I really just want to do this?

rowValue = {:Name("start_t"), :Name("part_id"), Row()};

With the latter, however, the value of the list at index 3 is literally just the text "Row()"! Shouldn't it run the Row() function and return the value?!

Is there a known issue in Windows 10 with the Script window being incredibly sluggish?

Anyway, this is the script in the form where it finally APPEARS to be working (need to fully verify):

dt = Current Data Table();
dt << Clear Select;

lastTests = Associative Array();



//For(i = 1, i <= N Rows(dt), i++,

For Each Row (
	//Row() = i;
	
	partKey = Char(:Name("214:ReadOTP:OTP_Row_40[1]") + 256*:Name("215:ReadOTP:OTP_Row_41[1]") + 256*256*:Name("223:ReadOTP:OTP_Row_49[1]") + 256*256*256*:Name("224:ReadOTP:OTP_Row_4a[1]") + 256*256*256*256*Abs(:Name("123:LOG_SETPOINT_SPEA:SPEA_Setpoint[1]")));
	
	rowValue = {};
	Insert Into(rowValue, :Name("start_t"));
	Insert Into(rowValue, :Name("part_id"));
	Insert Into(rowValue, Row());
	
	//rowValue = {:Name("start_t"), :Name("part_id"), i};
	
	hasValue = 0;
	try(
		junk = lastTests[partKey];
		hasValue = 1;
	);

	If ( hasValue ==  1,
		If ( rowValue[1] > lastTests[partKey][1], //if start_t is later
			lastTests[partKey] = rowValue,
		
		//else if
		(rowValue[1] == lastTests[partKey][1]) & (rowValue[2] > lastTests[partKey][2]), //if start_t is the same but part_id is larger
			lastTests[partKey] = rowValue;
			
		
		),
	//else
		lastTests[partKey] = rowValue;
	
	);
	
	
);

listOfRowsToSelect = {};


listOfVals = lastTests << Get Values;
//Show(N Items(listOfVals));

For(i = 1, i <= N Items(listOfVals), i++,
	Insert Into(listOfRowsToSelect, listOfVals[i][3]);

//Print(listOfVals[i][3]);
);

dt << Select Rows( listOfRowsToSelect );

Re: Is it possible to iterate through all elements in an associative array?

pmroz — Thu, 15 Mar 2018 13:07:42 GMT

Still tricky. Use evallist() to do it in one step. Also, instead of saying if (hasvalue == 1, just say if (hasvalue,. A positive integer is True in JSL, 0 is false. Here are some ways to do what you want. One way loops over all the rows explicitly, the other uses for each row.

dt = open("$sample_data\Big Class.jmp");
for (i = 1, i <= nrows(dt), i++,
	rowvalue = evallist({column(dt, "age")[i], column(dt, "height")[i], i});
	print(rowvalue);
);

current data table(dt);
for each row(
	rowvalue = evallist({as column(dt, "age"), as column(dt, "height"), row()});
// This works too
//	rowvalue = evallist({:name("age"), :name("height"), row()});
	print(rowvalue);
);