Accepted Solutions
Continuing with my earlier idea by separating text file into sections and then parsing those sections (this recording partially covers this topic ).
Names Default To Here(1);
// Test sections seem to always start with a line starting with "SetupTitle"
// and end to same UNLESS it is last section when it ends with
// line of ",,,,,,,,,,,,,,,,,,,,,,,"
TEST_SECTION_START = "SetupTitle, ";
EMPTY_LINE = ",,,,,,,,,,,,,,,,,,,,,,,";
// Section patterns for regex
PROJECT_PATTERN = "^(.+)_(\d+\.\d*)(W\d{1,2})_(T\d+)_(.+): (\d+ \d+)\,+?";
SUBSITE_PATTERN = "<Subsite>(\d+)</Subsite>";
DATASTART_PATTERN = "^DataName";
find_test_section_lines = function({lines}, {Default Local},
start_idx = 0;
end_idx = 0;
TEST_SECTION_START = TEST_SECTION_START; // JMP "feature", 00075244
TEST_SECTION_END = TEST_SECTION_END; // JMP "feature", 00075244
For Each({line, idx}, lines,
If(Starts With(line, TEST_SECTION_START),
If(start_idx == 0,
start_idx = idx;
,
end_idx = idx;
break();
);
);
);
If(start_idx != 0 & end_idx == 0,
end_idx = N Items(lines);
);
return(Eval List({start_idx, end_idx}));
);
parse_test_section = function({lines}, {Default Local},
data_start = 0;
PROJECT_PATTERN = PROJECT_PATTERN; //JMP "feature", 00075244
SUBSITE_PATTERN = SUBSITE_PATTERN; //JMP "feature", 00075244
DATASTART_PATTERN = DATASTART_PATTERN; //JMP "feature", 00075244
For Each({line, idx}, lines,
If(!IsMissing(Regex(line, PROJECT_PATTERN)),
matches = Regex Match(line, PROJECT_PATTERN);
projname = matches[2];
lotid = matches[3];
waferid = matches[4];
time = matches[5];
notes = matches[6];
reticles = Words(Trim Whitespace(matches[7]), " ");
, !IsMissing(Regex(line, SUBSITE_PATTERN)),
subsite = Regex(line, SUBSITE_PATTERN, "\1");
, !IsMissing(Regex(line, DATASTART_PATTERN)),
data_start = idx;
break();
);
);
data_str = Concat Items(lines[data_start::N Items(lines)], "\!N");
dt = Open(Char To Blob(data_str), "text", invisible);
For Each({colname}, Reverse(dt << Get Column Names("String")), // drop empty columns
If(Col Number(Column(dt, colname)) == 0,
dt << Delete Column(colname);
,
break(); // break on first "ok" column
);
);
// define order and names here
dt << New Column("project", Character, Nominal, Set Each Value(projname));
dt << New Column("lot", Character, Nominal, Set Each Value(lotid));
dt << New Column("waferid", Character, Nominal, Set Each Value(waferid));
dt << New Column("time", Character, Nominal, Set Each Value(time));
dt << New Column("notes", Character, Nominal, Set Each Value(notes));
dt << New Column("x", Character, Nominal, Set Each Value(reticles[1]));
dt << New Column("y", Character, Nominal, Set Each Value(reticles[2]));
dt << New Column("subsite", Character, Nominal, Set Each Value(subsite));
dt << Move Selected Columns({:project, :lot, :waferid, :time, :notes, :x, :y, :subsite}, To First);
dt << Delete Columns("DataName");
return(dt);
);
// Start parsing
filepath = "$DOWNLOADS/Raw data examples_3DP.csv";
txt = Load Text File(filepath);
lines = Words(txt, "\!N");
lines = Filter Each({line}, lines, line != EMPTY_LINE); // drop empty lines
{start, end} = find_test_section_lines(lines);
dt_result = Empty();
While(All(start, end),
cur_testset = Remove From(lines, start, end - 1);
dt = parse_test_section(cur_testset);
If(Is Empty(dt_result),
dt_result = dt;
,
dt_result << Concatenate(
dt,
"Append to first table"
);
Close(dt, no save);
);
{start, end} = find_test_section_lines(lines);
);
dt_result << Show Window(1);
Like others have said this can be highly dependant on your real data format. Below is example using the data copy pasted from excel to a text file
Names Default To Here(1);
filepath = "$DOWNLOADS/744027.txt";
txt = Load Text File(filepath);
lines = Words(txt, "\!N");
project_pattern = "^(.+)_(\d+\.\d*)(W\d{1,2})_(T\d+)_(.+):(.+)?";
subsite_pattern = "<Subsite>(\d+)</Subsite>";
datastart_pattern = "^DataName";
data_start = 0;
For Each({line, idx}, lines,
If(!IsMissing(Regex(line, project_pattern)),
matches = Regex Match(line, project_pattern);
projname = matches[2];
lotid = matches[3];
waferid = matches[4];
time = matches[5];
notes = matches[6];
reticles = Words(Trim Whitespace(matches[7]), " ");
, !IsMissing(Regex(line, subsite_pattern)),
subsite = Regex(line, subsite_pattern, "\1");
, !IsMissing(Regex(line, datastart_pattern)),
data_start = idx;
break();
);
);
data_str = Concat Items(lines[data_start::N Items(lines)], "\!N");
dt = Open(Char To Blob(data_str), "text");
dt << New Column("lot", Character, Nominal, Set Each Value(lotid));
dt << New Column("waferid", Character, Nominal, Set Each Value(waferid));
dt << New Column("x", Character, Nominal, Set Each Value(reticles[1]));
dt << New Column("y", Character, Nominal, Set Each Value(reticles[2]));
dt << New Column("subsite", Character, Nominal, Set Each Value(subsite));
dt << Move Selected Columns({:lot, :waferid, :x, :y, :subsite}, To First);
dt << Delete Columns("DataName");It uses regex to find specific lines and then parses them. For data it finds the line index where it starts and then parses it by letting JMP just open it (depending on the real data, this might require additional step(s)).
Hi txnelson and jthi, thank you so much for helping out!
To txnelson:
1. uninteresting information row interference.
//Agree the script is highly content-dependent. Will check when new format data is used.
2. JMP recognize the row that contains the project name
//very valid. The job ID is typed by myself. Will make sure format style unique and consistent.
3. Actual data file
//Attached "Raw data examples_3DP.csv". Note it contains 7 columns instead of simplified 3 columns (I/V/R) in the early files.
To jthi,
Wow, this is the first time I see the use of "patterns"! This thread is already worth it because of that! Your script very well makes sense. It did run into error when I challenged it by adding 2nd DP into the attached txt ("744027_modified.txt"). 2nd DP created in the following way to make it different from the 1st one:
1. Duplicated the first DP block and retained those junk lines.
2. Changed Job ID as following and subsite from "8" to "9"
- 1st DP: ProjTiger_753367.1W1_T0_1strun: 3 5
- 2nd DP: ProjTiger_753367.2W11_T168_2ndrun: 3 6
3. Multiplied the "I" column by 10.
The result pasted below. Note, in future, this script will see different column patterns (say, more than three in the early example), and varying data length (say, by using different measurement step size).
Please feel free to use either new raw data CVS or the modified txt, which is simpler. Either way, appreciate the helps!
Hi txnelson, it works! I literally will need this script for my next week report. Thank you!!!~~~
Meanwhile, could you make some improvements if it is not too complicated?
1. The subsite info (column) is missing.
//To help future viewers, one wafer has multiple reticles (aka stepper field or die); each reticle has its own unique (index X, index Y); each reticle has numerous devices called "subsite" (aka subdie). The measurement is done per subsite-reticle-wafer.
2. Output all DPs into a single JMP table
//Without thinking too much, I applied your script to my full raw file (hundreds of DPs...), you can imagine what have happened : fought with numerous forever-popup JMP windows, and eventually surrendered by turning to task manager...
3. Change the data table portion column properties from character to numeric.
4. Is it possible to compress the coding into a more concise format?
//Current one is very intuitive, great for me as a beginner. But, down the road, a shorter code could be more user-friendly. Please forget it if it is complicated or no longer first-time-viewer-friendly.