Bug using Concatenate/Append on tables having columns with List Check

[#1]

// This code shows the bug.  Concatenate with Append To First Table results in data corruption.

dtA = new table("A");

dtB = new table("B");

dtA << Add Rows(3);

dtB << Add Rows(1);

colA = Column(dtA, 1);

colB = Column(dtB, 1);

colA[1]="A";

colA[2]="A";

colA[3]="C";

colB[1]="B";

dtA << Compress Selected Columns(dtA << Get Column Names());

// This data has no List Check, but is not corrupted

(dtA << Concatenate(dtB)) << Set Name("Not Corrupted");

// This data is corrupted

dtA << Concatenate(dtB, Append To First Table(1)) << Set Name("Corrupted");

[#2]

Hi Robert,

I tried your scrit and do not see data corruption. I just see there is a forth row in dtA with no entry because of the list check. If you use the menu "data table" -> "concatenate" and then have table A and B and the check box Append to First Table checked AND check the "Data Source Column" Option you will see that there is the data from table B, but not entered because o the list check, which gets introduced becuase of the compression.

If that's not what you mean, as I have no corrupted data set now (and I would understand corrupted not readable anymore), please help me understand your issue. If you really get a corrupted data table you cannot open anymore please send your question to our technical Support so it get tracked for our development as well they hopefully can help you find the reason (Support@jmp.com).

Btw. I've used JMP 12.1.

Regards,

Martin

[#3]

Martin,

Thanks for looking at this.  That missing "B" IS the data corruption.  JMP has introduced a value (empty string) that was not present prior to concatenation, and as dropped a value ("B") that was present prior to concatenation.  So the table is not a valid concatenation of the two original tables.  If this were a critical business application, JMP would be injecting error, right?

I understand how it happens, but JMP should re-evaluate the necessary items in the list check to ensure the concatenated dataset does not drop/change any data.

Thanks,

- Bob

[#4]

Hi Bob,

sorry for the late response, was on the road at customer visits.

I understand your perspective. At the same time I think there is just few what could be done to prevent this. I will try to explain why. With the compression you have a list check initiated with the values which are within the columns at the time of compression. Let's assume there was no compression and you create a list check on your own. The purpose of a list check is that you only allow those values to be in that column. Other said you do not want to have other values than those.

No matter if you concatenate other tables to the original with the list check or just typing new observations in that table it will only allow those values as it believes you have set those rules by purpose, and I would not like to have Software work against my set rules.

That said, it is in Charge of the rules maker to Change the rules if other values have to be alloewed to be entered.

That said, and I#m sure you have understood the above already before, it is intended behaviour, though not exactly what you like.

I'd suggest you write to our technical support (support@jmp.com) to add an enhancement request to add a Question Dialog in case one concatenate a list checked (or compressed) data table with a new table using the "append to original table" flag. The dialog could state sth. like this: "There is an active list check in the original table. Your to be concatenated table has values not listed there, those values will convert to missing values after concatenation.Do you still want t concatenate? (If not you have to Change the list check properties first)" - or something shorter

To work around you have two choices: First to create a new table which you compress afterwards (may you want to have the data table "invisible" until compression has been done). Or second to change or remove the list check property in advance.

Hope that helps,

Martin

[#5]

Ok, thanks Martin for the advice how to get this on the dev team's radar...

- Bob

[#6]

As Martin said, the data table has not been "corrupted".

I assume you mean that the resultant data table is not what you expected. You expected "B" to be the data for the 4th row.

If you look at the list check property generated, there are 2 values, "A" and "C".

By definition, only "A" and "C" are valid data for the column.

Since "B" is not a valid value, it was mapped to missing.

Compress generate the list check table based on the information it knows at the moment.

Maybe Concat should be smarter, and expand the list check table.

Alternatively,  remove the list check property entirely if not all of the columns to be concatenated have the same list check property.

[#7]

I really don't understand the point of arguing over the semantics of the word "Corrupt".  Is the resulting data table a valid concatenation of the two original data tables?  It is clearly not.  Records have been changed, despite proper usage of JMP features.  The fact that Concatenate WITHOUT Append to First Table gives a different (correct) result than Concatenate WITH Append to First Table should concern you.  The fact that a customer is telling you there is a bug, should also concern you...

Thanks,

- Bob

[#8]

Sorry, I did not mean to argue over the word "corrupt", neither was I trying to dismiss what you think is a bug.

I was trying to explain what you got. If you remove the list check property from the column, then the append works as you expected.

It was the list check property that seems to cause the confusion.

By definition, list check property says that only the values in the list check table are valid.

If you add or change a value to the column to a value that is not in the list check table, then it is set to missing.