topic Re: Looking to generate two columns based on some pattern from other columns in Discussions

Looking to generate two columns based on some pattern from other columns

HSS — Fri, 09 Jun 2023 18:07:38 GMT

Hi All,

I am looking to generate two new columns based on patterns in other two columns ( "Required column1" and "Required Column2"). Please see the attached image -

Any help/suggestion ?

Thanks.

Re: Looking to generate two columns based on some pattern from other columns

txnelson — Thu, 16 Dec 2021 22:46:10 GMT

Here is a formula for the column you refer to Required Column 2. When you study the code, you will see that all it is doing is applying the logic rules you have stated that need to be checked when determining the Start and End values. It also contains the setting of the value for the column Required Column 1. To run this formula, simply create the Required Column 1 and Required Column 2. Set Required Column 2 to be a character column. Then open the formula for Required Column 2 and copy the formula below into the column.

RCValue = "";
:Required Column 1[Row()] = .;
If( :Sensor != "Primary",
	other = "Yes"
);
If( Row() == 1 | :ID != Lag( :ID ) | :"Sub-set"n != Lag( :"Sub-set"n ),
	counter = 0;
	withinStart = "No";
	other = "Yes";
);
If(
	:Sensor == "Primary" & withinStart == "Yes",
		RCValue = "End";
		withinStart = "No";
		other = "No";,
	:Sensor == "Primary" & Lag( :Sensor, -1 ) != "Primary" & withinStart == "No" & other == "Yes" & Row() !=
	N Rows( Current Data Table() ),
		withinStart = "Yes";
		counter = counter + 1;
		RCValue = "Start";
);
If( withinStart == "Yes" | RCValue == "End",
	:Required Column 1[Row()] = counter
);
RCValue;

Re: Looking to generate two columns based on some pattern from other columns

HSS — Fri, 17 Dec 2021 02:21:02 GMT

Hi @txnelson,

It almost worked with few false entries. I believe we must add extra condition on :Occurrence column. That is :Occurrence must change to set :Required Column 2 == "Start" (and may be option - it should be same when :Required Column 2 == "End").

Something like - Dif(:Occurrence) !=0 then only :Required Column 2 == "Start"

For my sample data (95 rows) was fine with your formula but my actual data which has more than 300k rows, it is giving many wrong entries like below - (I think it should be fine once additional requirement on :Occurrence will be added.) This is just one off screenshot of the wrong entry.

Hopefully!! I can modify the formula/code and can add requirement of :Occurrence. on top of your formula.

Thanks again for your support.

Re: Looking to generate two columns based on some pattern from other columns

HSS — Tue, 21 Dec 2021 08:43:07 GMT

Hi @txnelson and All,

I was able to modify your code based perfectly for my requirement. But on the 2nd stage troubleshooting I need something like this --

Mark :Sensor = "Primary" (1st) around consecutive change in :Occurrence. In the given screenshot -

required values in between --- Last "Primary" in Occurrence = 2 and First "Primary" in Occurrence = 3 [consecutive change in Occurrence]

Finding first Primary is not that difficult, not sure how to find last primary ? Thinking to somehow invert (descending order) the subset data for Occurrence = 2, find the value and then revert it back. Not able to put in a formula!! Any other way to do that ?

Any help?
Thanks.

Re: Looking to generate two columns based on some pattern from other columns

txnelson — Tue, 21 Dec 2021 10:58:08 GMT

You have modified the code I previously provided to get the results that you provide in your most recent response. Please provide the JSL you are using to get the results you are showing in your sample data table.

Re: Looking to generate two columns based on some pattern from other columns

HSS — Tue, 21 Dec 2021 14:20:11 GMT

Hi @txnelson , thanks for your response.

I am not able to get the correct (updated requirement) start point. In the Data Table ("Sample Data Updated") and image, I had entered corrected data columns manually.

The formula. I used to get "Required Column 1 Old" and " Required Column 2 Old" is this (It is just like you give me, I added occurrence condition on it) -

RCValue = "";
:Required Column 1[Row()] = .;
If( :Occurrence != Lag( Occurrence, -1 ),
	other = "Yes"
);
If( Row() == 1 | :ID != Lag( :ID ) | :"Sub-set"n != Lag( :"Sub-set"n ),
	counter = 0;
	withinStart = "No";
	other = "Yes";
);
If(
	:Sensor == "Primary" & :Occurrence == Lag( :Occurrence, -1 ) & Dif( :Occurrence ) == 0 &
	withinStart == "Yes",
		RCValue = "End";
		withinStart = "No";
		other = "No";,
	:Occurrence != Lag( Occurrence, -1 ) & Lag( :Sensor, -1 ) != "Primary" & withinStart == "No"
	 & other == "Yes" & Row() != N Rows( Current Data Table() ),
		withinStart = "Yes";
		counter = counter + 1;
		RCValue = "Start";
);
If( withinStart == "Yes" | RCValue == "End",
	:Required Column 1[Row()] = counter
);
RCValue;

Many thanks.

Re: Looking to generate two columns based on some pattern from other columns

txnelson — Tue, 21 Dec 2021 17:43:01 GMT

i am confused. From what I can see, the script that I wrote seems to work, except that after line 44 in the data table, you have changed the rules, and now want the End to be on the row prior to the next value of Primary. Which is what you want?

Re: Looking to generate two columns based on some pattern from other columns

HSS — Tue, 21 Dec 2021 18:54:42 GMT

Hi Jim,

Sorry for the confusion.

Your original formula (I believe, I have not messed it up) worked for most of the cases but also captured few wrong points and missed few on full set of data. In the attached subset of the data (only for one "ID" and one value of "Sub-set") there is a column based on your formula which has few wrong points.

Rows: 647-649 wrong entry (Red Colour)
Rows: 868-937 not captured (Red Colour)
Rows: 1307-1311 wrong entry (Red Colour)

And this was fixed when I put condition on :Occurrence.

Now I am looking for -

1. Last Primary Sensor in occurrence = n (say n = 2), mark it as "Start"

2. First Primary Sensor in Occurrence = n+1 (= 3), and if there is no primary sensor for n = 3, it should stope/mark the value in Occurrence = n+2 == 4. Mark it as "End"

3. And last fill the numbers between "Start" and "End" in a new column.

Alternate way to think of (may be) --
If we can mark - in any single occurrence n (n = 2) First Primary Sensor as END and in last primary sensor as Start and then fill a values between Start and End in a new column.

I hope this will help and make things clear.

Thanks,
I think this can be, if I can find a

Re: Looking to generate two columns based on some pattern from other columns

txnelson — Wed, 22 Dec 2021 13:37:41 GMT

The issue appears to be with the logic being applied to the JSL variable "other". I do not have time to work on the issue today, but tomorrow afternoon, I should be able to do some investigation.

Re: Looking to generate two columns based on some pattern from other columns

HSS — Thu, 23 Dec 2021 08:18:40 GMT

Thank you @txnelson ,

I did some progress - and identify correctly the start and end point of each set. I used a very simple logic, you can see in the formula column.

But now I am struggling to put numbers between each pair of "Start" and "End" point. I getting issue in stopping the counter at "End". My issue might have the same complicacy but at least now I have clearing marking/tag on my start and end point.

Any help with this ?
Thanks.

Thanks.

Re: Looking to generate two columns based on some pattern from other columns

jthi — Thu, 23 Dec 2021 10:05:27 GMT

I gave a go with this. I try to avoid using Lag() whenever possible and replace it with Col functions. The most difficult formula seems to be Required Column 1 and I couldn't quickly figure out good formula without using Lag() (there is most likely a solution with Col Cumulative Sum somehow). One idea would be to fill places between Start and End with "Ongoing" or something similar, then you could maybe use Col Cumulative Sum easier, but then the most difficult formula would most likely just move to Start and End.

Names Default To Here(1);

dt = Current Data Table();
//Numbering of Occurrence
dt << New Column("NumberOfOcc", Numeric, Nominal, Formula(
	If(:Sensor == "Primary",
		Col Cumulative Sum(:Sensor == "Primary", :ID, :"Sub-set"n, :Occurrence),
		.
	);
));

//Start and End
dt << New column("StartEnd", Character, Nominal, Formula(
	If(:NumberOfOcc == 1, 
		"End"
	, Col Max(:NumberOfOcc, :ID,:"Sub-set"n, :Occurrence) == :NumberOfOcc,
		"Start"
	, //else
		""
	)
));

//Required Column 1, this could most likely be made somehow with Col Cumulative Sum
dt << New column("Req1", Numeric, Ordinal, Formula(
	If(Row() == 1,
		val = .
	);
	If(:StartEnd == "Start",
		val = 1
	, IsMissing(:StartEnd) | :StartEnd == "End",
		val += 1;
	);
	If(:Lag(:StartEnd) == "End" | (Row() == Col Max(Row()) & :StartEnd == "Start") | (:StartEnd == "Start" & Lag(:StartEnd, -1) == "End"), //more missing checks can be added here
		val = .,
		val;
	)
));

//Required Column 2
dt << New column("Req2", Numeric, Nominal, Formula(
	If(!IsMissing(:Req1),
		Col Cumulative Sum(:StartEnd == "Start",
			:ID, :"Sub-set"n
		)
	,
		.
	)
));

If you have time, I would suggest testing both ways of solving this, the one @txnelson has provided with more heavy use of Lag and comparisons, and the one I have with Col formulas. Both definitely have their use cases and understanding them will help you in a long run when writing formulas/calculations in JMP. And in some cases when you understand how to combine both, you can solve quite difficult tasks with just formulas.

Also after you get the logic working, I would most likely remove Formulas from the datatable (or replace Formula() with << Set Each value in script), because this will most likely break if order is changed in datatable.

Re: Looking to generate two columns based on some pattern from other columns

HSS — Thu, 23 Dec 2021 19:40:41 GMT

Dear @jthi (Jarmo) and Dear @txnelson (Jim),

I am really thankful for your support and kindness.

I will certainly find one working way by either of these two methods and some combination of these two.

Many thanks again to both of you.
Merry Christmas and Happy New Year. Enjoy your time.

Best Regards,
Harry

Re: Looking to generate two columns based on some pattern from other columns

HSS — Mon, 27 Dec 2021 11:49:04 GMT

Hi All,

Have another query, actually two -

1. What is the difference, If I add more than one condition in single "If" statement vs. Adding more than one "If" statement with single condition. See the attached image - . First case is not giving correct results while the 2nd one is working fine. Results can be seen in the attached sample data. I am having difficulty to understand the differences here.

To me it seems for each row - in first case, if condition is true, output is given. This ends the results.
While in 2nd case, it will run all 3 if statements sequentially, irrespective of results of previous one and out put will be based on the last true condition. I am not sure if I am able to explain it correctly. Again, I guess in 2nd case (multiple if statement), there is a sort of overwrite on the previous If statement !

2. There is a condition used in the above formula -- " Row() == Col Max(Row()) & :StartEnd == "Start" " How is it working? It works well, means - it is well written. But I am not able to visualize how the variable/counter "val" stops ?

Any help with it ?

Thanks,

Re: Looking to generate two columns based on some pattern from other columns

jthi — Mon, 27 Dec 2021 19:37:43 GMT

1. Maybe this image gives explanation why the first idea won't work (if I remember correctly the second solution relies on that if you calculate . + 1 it will be .:(

I did take the different comparisons from the If statement and just create formulas out of them so it is easier to see what returns True and when.

You could most likely get the first solution working by modifying the comparisons a bit ( Is Missing(:StartEnd) & Lag(:StartEnd) != "End" | :StartEnd == "End", val += 1). But with this change you will have to make small change to second comparison also (add Row() != Col Max(Row()) check).

2. Idea for Row() == Col Max(Row()) & :StartEnd == "Start" is just to to check if we are on last row of the table and :StartEnd value is "Start" we won't set the value as 1 based on previous if statement. Row() will tell you row number and Col Max(Row()) will return you maximum row number in the datatable. Might be a bit unnecessary check because it is last row of table (like the one after it with Start/End), but as it was in the results column, I did implement them.

The general idea with the last if-else was following:

1. Lag(:StartEnd) == "End" -> this is general "stop" after End is countered

2. Row() == Col Max(Row()) & :StartEnd == "Start" -> to prevent returning 1 on last row if it is Start

3. :StartEnd == "Start" & Lag(:StartEnd, -1) == "End" -> to prevent returning 1 and 2 if there are no rows between Start and End

You can use just one if with multiple comparisons by moving some of the comparisons around a bit (this seemed to return same result as Req1):

If(
	Row() == 1, val = .,
	Row() != Col Max(Row()) & :StartEnd == "Start" & Lag(:StartEnd, -1) != "End", val = 1,
	Is Missing(:StartEnd) & Lag(:StartEnd) != "End" | :StartEnd == "End", val += 1,
	val =.
)

Hopefully the explanation isn't too messy

Re: Looking to generate two columns based on some pattern from other columns

HSS — Tue, 28 Dec 2021 05:12:34 GMT

Thanks @jthi for nice explanation.