Transpose List of Lists without a for loop

[nikles]

I would like to have a method to transpose a list of lists without using a for loop.  That is, some function (call it "Listtranspose" for instance) that flips the "rows" and "columns" of a list of lists, like this:

listlist = {{"cat", "dog", "mouse"}, {"meow", "bark", "squeak"}};
tranlist = Listtranspose(listlist);
Show(tranlist);

/*Result:
{{"cat", "meow"}, {"dog", "bark"}, {"mouse", "squeak}}

An alternate implementation could be something like:

listlist = {{"cat", "dog", "mouse"}, {"meow", "bark", "squeak"}};
nr = NItems(listlist);
col1 = listlist[1::nr][1];
col2 = listlist[1::nr][2];
col3 = listlist[1::nr][3];
translist = EvalList({col1, col2, col3});

I kind of like the latter implementation better as I might only need to transpose just a subset of "columns", not the entire list.

Of course, this can be done using a For Loop, but the reason for asking for this is that For Loops can be slow.  I work with large blocks of text and I believe this would help speed things up greatly for me.

Credit to @vince_faller for asking this originally. Transpose List of Lists without a for loop 

[vince_faller]

I swear up and down I saw someone do something along the lines of 

{"cat", "dog", "mouse", "meow", "bark", "squeak"}[[1 1, 1 1]]
// and had it return
{{"cat", "cat"}, {"cat", "cat"}}

during a talk at discovery but I've never been able to recreate it.  It could have all been a dream ?

If I remember right, they somehow shaped a new list from a flat list in one line like that.  I probably don't remember right. 

That being said you can probably do parallel assign really fast

names default to here(1);
listlist = {{"cat", "dog", "mouse"}, {"meow", "bark", "squeak"}};
m = J(nc, nr, -1);
::trans_list = Aslist(m);
Parallel Assign({l = listlist}, 
	m[a, b] = (
		::trans_list[a][b] = l[b][a];
	)
);
show(::trans_list);

[nikles]

@vince_faller 

Mind == Blown.

Never heard of the Parallel Assign function before.  I tried this, and yup, it works.  Even though I'm working with character strings and this is technically a matrix function, it somehow doesn't care.  It will be interesting to play with this and see if it's faster than a for loop.  Will try to update here once I know.  

Thanks for the idea!

[vince_faller]

@Craige_Hales wrote a great explanation that I use regularly. Multithreading troubleshooting using Parallel Assign 

[Craige_Hales]

Yes, it would be nice to have more list manipulation functions. I can't come up with a nice or fast way, but after playing a bit came up with this. It needs JMP 15, fails in 14.

list = {{"lion", "tiger", "puma"}, {"lamp", "torch", "pharos"}, {"lark", "toucan", "pigeon"}, {"lawn", "terrace", "patio"}};
jlist = As JSON Expr( list );
dt1 = JSON To Data Table( jlist, invisible(1) );
dt1 << New Column( "row", numeric, setvalues( 1 + Floor( (0 :: (N Rows( dt1 ) - 1)) / N Items( list[1] ) ) ) );
dt2 = dt1 << Split( Split By( :row ), Split( column(dt1,1) ), Sort by Column Property, invisible(1) );
Close( dt1, nosave );
transposed = dt2[0, 0];
Close( dt2, nosave );
Show( list, transposed );

list = {{"lion", "tiger", "puma"}, {"lamp", "torch", "pharos"}, {"lark", "toucan", "pigeon"}, {"lawn", "terrace", "patio"}};
transposed = {{"lion", "lamp", "lark", "lawn"}, {"tiger", "torch", "toucan", "terrace"}, {"puma", "pharos", "pigeon", "patio"}};

Unhide the intermediate tables to see what it is doing. A real solution should deal with rows of unequal length.

[Craige_Hales]

But, after some testing, I think a simple nested loop will be faster than any other JSL-based approach.

	result = {};
	for( c = 1, c <= nc, c += 1,
		thisrow = {}; // build a row from a column
		for( r = 1, r <= nr, r += 1,
			thisrow[r] = testList[r][c];
		);
		result[c] = thisrow;
	);

@DonMcCormack - Challenge!

The blue-line for-loop is consistently better. The purple JSON-table might overtake it, but I'm running out of memory. The parallel-assign trick begins to fail for large numbers.

The parallel assign is probably blocking threads against each other, all the time. It beats the json data table at first because of different overheads between the two.

[nikles]

@Craige_Hales Thanks for the tremendous insight Craig.  I agree, I did a speed test of the for loop vs parallel-assign method, and parallel-assign was way slower.  My list array was 718r x 14182c, so >10million elements.  I ran the script for ~10min and the operation did not complete.  This seems to be a little slower than your data above predicts, but too many variables to consider (e.g. computer system, ratio of columns to rows?).  For loop is much faster at ~10.1s for the same table. 

Still slower than I'd like though.  Would be great to have a JMP function that can do this internally.

[vince_faller]

Thinking about this some more.  I still can't beat the for loop, but figured I'd write this down in the event it helps anyone. 

// make a blank data table with the right shape
dt = As Table(J(nr, nc, .), <<private);
// switch the columns to character
(dt << Get Column References) << Data Type(Character);
// put them in
dt[0, 0] = list;
// native transpose
dt_t = dt << Transpose(columns(), private);
close(dt, no save);
// pull them back out. 
trans_table = dt_t[0, 2::nr+1];
close(dt_t, no save);

[hogi]

2024 - JMP18
what is the best/fastest way to transpose a list of lists?

[vince_faller]

@hogi you could run the scripts above and let us know?  (lazy/selfish)

[hogi]

run the scripts: this approach just works if the easiest/fastest/best approach of 2024 can already be found in the list of possibilities of 2021  ; )

Maybe there is now an ultra-fast super-easy listtranspose(list of lists)which just got possible by the improvements of JMP17 or JMP18?