topic Re: Import character value issue in Discussions

Import character value issue

NRW — Sat, 10 Jun 2023 23:30:54 GMT

Folks,

I'm importing values from cvs files. One of them is a string. However, JMP is unable to read the value. The first image is the cvs file where all values are imported except "FlexSEM1000." So no issues there. However, second image shows the JMP table with the F????? value. How do I tackle this issue? Relevant code attached.

Neil

Join_files = Expr(
		
	For( i = 1, i <= N Items( path ), i+=2, 
			/*area, %area, and pore pixel count*/
			jointable1 = Open( path[i] );
			//jointable1 << Get Name;
			samplename = Substr( jointable1 << Get Name, 1, 18 );			//sample replicate name //
			samplename2 = Word(1,samplename, "_");							// Only sample name, no replicate suffix (i.e. a1, a2...)			
			
			
			
			jointable1 << New Column( "Sample",
				Character,
				Nominal,	
				Set Each Value( samplename )								// setting each row in column 1 to samp rep name //
				);
			
			/*diameter and pore pixel count*/
			jointable2 = Open( path[i + 1] );

			dt = Data Table( jointable1 ) << Join(
				With( Data Table( jointable2 ) ),
				Select( :Sample, :Area, :Name( "%Area" ),:scale, :thick, :pixelrad, :mag, :system),
				SelectWith( :pixD1, :count1), 
				Cartesian Join	
				);
			
			Close( jointable1, no save );
			Close( jointable2, no save );
			
			Columns();
			Graphs();
			Graphs2();
			Graphs3();
			dt << Set Name(samplename);
			po = dt << get name;
			dt << Save(SD1 || po);
			
		);
	
	
);


Columns = Eval Expr(
	//dt << New Column( "Condition", Character, Nominal, Set Each Value( "1:1" ) );
	dt:Name( "%Area" ) << Set Name( "Image A pore area (%)", Numeric, Continuous );
	dt:Area << Set Name( "Total Image pixel count (pore_part)", Numeric, Continuous );
	dt:scale << Set Name("pix/mm scale", Numeric, Continuous);
	dt:pixD1 << Set Name( "Pore Diameter (pixel)", Numeric, Continuous );
	dt:count1 << Set Name( "Pore Pixel Count", Numeric, Continuous );
	dt:thick << Set Name("d - thickness (m)", Numeric, Continuous);
	dt:pixelrad << Set Name("LT Radius", Numeric, Continuous);
	dt:mag << Set Name("Mag", Numeric, Continuous);
	dt:system << Set Name("Image Mode", Character, Nominal);
	dt:Sample << Set Display Width( 150 );

Re: Import character value issue

Craige_Hales — Fri, 28 May 2021 15:33:21 GMT

Away from computer, can't give exact words...
Try using an alternate charset, either through preferences.. text import, or through the text import wizard. The diamond question mark means the character set jmp guessed is not correct.

Re: Import character value issue

NRW — Fri, 28 May 2021 15:43:44 GMT

Thanks Craige,

Per the cvs value "FlexSEM1000", when I change this to "FlexSEM", JMP is able to read without issues.

I'll poke around using your suggestions.

Neil

Re: Import character value issue

Craige_Hales — Fri, 28 May 2021 16:02:11 GMT

Depending on the editor, retyping it might change it back to the preferred unicode charset. Ideally the program that produces the data would use ascii or unicode. Or, if that can't work, tell jmp which charset to use via the charset option.

Re: Import character value issue

NRW — Fri, 28 May 2021 16:26:58 GMT

Craige,

I'll keep plugging away. I tried attaching the csv doc in the case someone could get it to work, but the site was not allowing the upload.

In the meantime I'll go to program that generated the file to see how it is formatting. Originally the system ID data was being read from a system text file (accompanied with images from a SEM) which was UTF16. However a script was written to change to UTF8 and save to csv file.

I've had JMP look at both types, but no success.

Neil

Re: Import character value issue

Craige_Hales — Fri, 28 May 2021 16:39:57 GMT

You can see the hex representation of the file using jmp. Approximately like this...
Write(hex(loadtextfile(filename,blob)))

Re: Import character value issue

NRW — Fri, 28 May 2021 18:26:19 GMT

Craige,

Quick update.

I found out if I manually open the csv file and save as a "csv UTF-8", JMP will recognize "FlexSEM1000". However that's a manual step I would like to eliminate.

Neil

Re: Import character value issue

NRW — Tue, 01 Jun 2021 14:44:56 GMT

Craige,

I was able to go back to the source code that generated the csv, and found the issue. All the items in the the table were UTF-8, EXCEPT the last column item of interest. I fixed the code and now everything works. I guess JMP was seeing both UTF-8 and UTF-16 and and since it was already seeing UTF-8, didn't recognize the last column.

Many thanks for all your valuable time.

Neil

Re: Import character value issue

Craige_Hales — Tue, 01 Jun 2021 14:57:24 GMT

Yes, if there is a unicode Byte Order Mark at the beginning, Jmp believes it. Without a BOM, I'm not sure exactly how the guess proceeds. Glad you found it!