cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Try the Materials Informatics Toolkit, which is designed to easily handle SMILES data. This and other helpful add-ins are available in the JMP® Marketplace
Choose Language Hide Translation Bar
vince_faller
Super User (Alumni)

downloading large columns from database via odbc

I'm have a sql column that is varbinary(MAX).  It seems that the column width for JMP is 65536, so I can't download the whole column. It doesn't seem to be an ODBC only issue.  

Currently I'm doing these one at a time by just grabbing substrings until they're all empty and concatting into an array. 

 

This takes FOREVER.  

 

Does anyone have any suggestions for ways to optimize this?  I'm trying to avoid using python/sqlcmd/etc. to grab the blobs directly (avoiding JMP).  

 

 

Vince Faller - Predictum
3 ACCEPTED SOLUTIONS

Accepted Solutions
Jeff_Perkinson
Community Manager Community Manager

Re: downloading large columns database from database via odbc


@vince_faller wrote:

It seems that the column width for JMP is 65536, so I can't download the whole column. It doesn't seem to be an ODBC only issue.  

I don't have any immediate thoughts on your actual inquiry, but I do want to clarify the statement above. The max width for a character column in JMP is MUCH larger than 65536. 

 

New Table( "untitled 15",
	Add Rows( 1 ),
	New Column( "repeat",
		Character,
		"Nominal",
		Formula( Repeat( "a", 128000000 ) ),
	),
	New Column( "length",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Formula( Length( :repeat ) ),
	)
)

I'm sure this detail isn't relevant to solving your speed/query problem and I may have misunderstood what you meant by column width.

 

 

-Jeff

View solution in original post

Re: downloading large columns from database via odbc

Per bryan_boone the behavior is due to these drivers not retuning the column length. Subsequently, JMP sets a default lenght of 65536 when this data is not provided.

 

I've added a new feature request to have this value increase since JMP can handle this data.

 

In the interim, using SQL Server (native) seems to work.

 

 

 

View solution in original post

hardner
Level VI

Re: downloading large columns from database via odbc

 Just a note that this limitation does seem to have been removed in JMP16.1.  Thanks!

View solution in original post

15 REPLIES 15
Jeff_Perkinson
Community Manager Community Manager

Re: downloading large columns database from database via odbc


@vince_faller wrote:

It seems that the column width for JMP is 65536, so I can't download the whole column. It doesn't seem to be an ODBC only issue.  

I don't have any immediate thoughts on your actual inquiry, but I do want to clarify the statement above. The max width for a character column in JMP is MUCH larger than 65536. 

 

New Table( "untitled 15",
	Add Rows( 1 ),
	New Column( "repeat",
		Character,
		"Nominal",
		Formula( Repeat( "a", 128000000 ) ),
	),
	New Column( "length",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Formula( Length( :repeat ) ),
	)
)

I'm sure this detail isn't relevant to solving your speed/query problem and I may have misunderstood what you meant by column width.

 

 

-Jeff
vince_faller
Super User (Alumni)

Re: downloading large columns database from database via odbc

VERY relevant.  My bad.  Something weird is going on then. 

 

Thanks @Jeff_Perkinson , 

Vince Faller - Predictum
vince_faller
Super User (Alumni)

Re: downloading large columns database from database via odbc

@Jeff_Perkinson 

 

Looking more into this I can't explain this behavior.  I'm able to get values from the exact same table and connection string through python.  For whatever reason JMP blanks out the column as soon as it hits 65537.  

 

Names default to here(1);
driver = "ODBC Driver 17 for SQL Server"; // using SQL Server
server = "localhost";
database = "Test";
username = "sa";
password = "REDACTED";

cxn_string = "DRIVER="||driver||";SERVER="||server||";UID="||username||";PWD="||
	password||";Database="||database;

cxn = create database connection(cxn_string);
executesql(cxn, 
	"
	CREATE TABLE [dbo].[LargeChar](
		[LargeCharID] [bigint] IDENTITY(1,1) NOT NULL,
		[SomeBigChar] [varchar](MAX) NOT NULL,
	 CONSTRAINT [PK_Blob] PRIMARY KEY CLUSTERED 
	(
		[LargeCharID] ASC
	)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
	) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
	"
);

executeSQL(cxn, 
	"INSERT INTO [LargeChar]
	VALUES('"||REPEAT("A", 100000)||"')"
);

dt_download = executeSQL(cxn, 
	"SELECT SomeBigChar
	, SubString(Convert(VARCHAR(MAX), SomeBigChar), 1, 65536) AS Char65536 -- this works
	, SubString(Convert(VARCHAR(MAX), SomeBigChar), 1, 65537) AS Char65537 -- this doesn't
	FROM [LargeChar]", 
	"This should work?"
);
show(length(dt_download:SomeBigChar[1]), // full text doesn't show
	length(dt_download:Char65536[1]), // text that is only 65536 shows
	length(dt_download:Char65537[1]) // text that is one higher? nope
);

//execute SQL(cxn, "DROP TABLE [LargeChar]");
close database connection(cxn);

//Python script returns
/*
0    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA...
Name: SomeBigChar, dtype: object
0    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA...
Name: Char65536, dtype: object
0    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA...
Name: Char65537, dtype: object
*/

 

Is there something that I'm missing?  Or maybe underneath python is adding some functionality.

 

*EDIT* sorry for so many edits.  Hope you don't get an email for each one. 

Vince Faller - Predictum
Jeff_Perkinson
Community Manager Community Manager

Re: downloading large columns database from database via odbc

This gets beyond my ability to test to understand what's happening.

 

@briancorcoran?

-Jeff

Re: downloading large columns database from database via odbc

Can you try using:

Preferences( ODBC Disable Block Fetch( 1 ) );

Sometimes the ODBC Driver doesn't report the length of the column.

SQL Server is one of the drivers that supports block fetch (which is much more efficient than a row at a time), but if the length of the column isn't reported by the driver, the default value of 65536 is used.

vince_faller
Super User (Alumni)

Re: downloading large columns database from database via odbc

Ah, okay. so since I'm using VARCHAR(MAX), maybe there is no column length?  

 

Unfortunately, even with the block fetch off it doesn't work.  

Vince Faller - Predictum
stan_koprowski
Community Manager Community Manager

Re: downloading large columns database from database via odbc

@vince_faller,

Are you using the same ODBC driver for Python and JMP?

Stan

vince_faller
Super User (Alumni)

Re: downloading large columns database from database via odbc

Yep. Using the exact same connection string.
Vince Faller - Predictum

Re: downloading large columns from database via odbc

Per bryan_boone the behavior is due to these drivers not retuning the column length. Subsequently, JMP sets a default lenght of 65536 when this data is not provided.

 

I've added a new feature request to have this value increase since JMP can handle this data.

 

In the interim, using SQL Server (native) seems to work.