Pi Import Options - the Max count option

[#1]

I haven't seen any recent posts about the Pi Importer options.  This was supposed to be an upgrade in 18, but I'm struggling with the Max Count setting especially.  I'm not sure there is really a question here, but sharing what I've learned and wondering if anyone else has other insight to share here. I focus on the Raw Data Import because of the structure of our data and trying to remove any layers of added confusion.

The basics:  The default for Max Count is 5000, with an absolute max of 150,000.  It is the maximum number of values that JMP will fetch for the request you give.  If you select 4 pi tags and leave the default at 150,000, you will get an error message attached, indicating you are over the max allowed (150,000 / 4 tags = 37500 max). It appears if some tags are from different "assets", the limits/division might be "by asset" - I haven't fully figured that out yet.  Only the description is in the JMP help menu.  These limits/definitions should be added.

The error message has helped me figure out what is going on, although it seems very odd that this isn't a more integrated setting showing total # of points vs points per tag.  What I find very confusing is the output.  Attached is an example of 4 Pi tags pulled.  At first I thought the cut-off of data in each plot was tied to the order they showed up in the pull (alphabetical), or the order I added them to the import window (purple, green, red, blue), but I believe it is actually the # of outputs of each tag.  Blue and purple have fewer data points, so they don't hit the 37,500 max during the 1 day timeframe I'm pulling. I thought red and green had identical data, but digging deeper, there are 4000 points interspersed in the green that aren't in the red.  That means that green hits 37,500 points earlier in the timestamps than red does.  I confirmed there are exactly 37,500 values in both the red and green column, with the remaining cells having "missing" data.  There is NO ERROR MESSAGE or any indication from JMP that additional data is in Pi during the timeframe requested and it is just not being pulled.  I know JMP is pulling what I asked it to pull, but I don't know how much data is there, and it is imposing this arbitrary restriction. This particular data pull was for 1 day and only gave me 42,526 rows in the data table (many timestamps have only 1 or 2 pi tag results).  When one tag cuts out before another and some tags cover the FULL time range, my first assumption is that the tag stopped producing data or there was a disconnect! The confusion is even greater given the layout of the data table.  The red and green columns are between the blue and purple in the output.  Having them just stop at random points as we scroll down the time-ordered table doesn't look like "the data limit was hit" but rather that the data was broken for these tags.  I now understand that is not the case.

It also means when I'm trying to do the work-around and pull smaller amounts of data to combined with longer timeframes and other tags with code, I have no idea if I go over the limit and am suddenly missing a bunch of data.  With the realization that I can look at the number of values outputted, I'll need to add this error-proofing to my code also.

In summary, if I have one tag that only reports out every 8 hours and one tag that has data every 1 second, and I try to pull these together so the timestamps are all aligned, it appears I can't pull 150k total outputs, and the more frequent tag will be the limiting factor.  I do know how to solve this using a Pi import one at a time and looping them through combining tables, but it is a pain and not something I'm going to be able to get many other users on board with here.  This JMP interface has a lot of potential, but this one piece adds a lot of confusion!

I will send this to Tech Support as well to see if they can add more info or get this into the help information, but wondering what other users out there are seeing!

[#2]

although not convenient, perhaps you could import each tag separately, then join them by time  (or concatenate, then split by tag) , then fill in the missing values with something like this :

Names Default to Here(1);
dt = Current Data Table();
rows = N Row( dt );

dtcollist=dt<<get column names;

dlg = Dialog("Select Column",
vlist(
selection=listBox(dtcollist),
button("OK")));
Remove From( dlg, 2 );
Eval List( dlg );

columns=nitems(selection);

For( i = 1, i <= columns, i++,    

	Eval(
		Substitute(
				Expr(

	If( Is Missing( Column( expr(rrr) )[1] ),
		firstone = {none};
		For( k = 2, N Items( firstone ) == 1, k++,
			If( Not( Is Missing( Column(  expr(rrr) )[k] ) ),
				Insert Into( firstone, Column( expr(rrr)  )[k] )
			)
		);

		Column(  expr(rrr)  )[1] = firstone[2]);
			
		For( k = 2, k <= rows, k++,
			If( Is Missing( Column(  expr(rrr)  )[k] ),
				Column(  expr(rrr)  )[k] = Eval( Column(  expr(rrr)  )[k - 1] )
			)
		)

				),
			Expr( rrr ), selection[i]
		)
	)
);

[#3]

Yes, I am already pulling one tag at a time and combining.  However, this feature of "pull multiple tags" that was added in JMP 18 should make it easier for the average JMP user who isn't going to write that complex code to pull the tags - even manually using the box where you can click to select the tags.  Overall I like the interface better than other options I've used to pull Pi data, but this dropping of data with no warning is a key concern.  I shouldn't have had to spend this much time trying to understand why some tags were missing before reverting to the much more difficult and manual method.  I cannot recommend JMP for this use to my coworkers with this gap.