Choose Language Hide Translation Bar
Highlighted
ekallou3
Level II

Automate opening an hdf5 file

Hello,

 

I am trying to create a .jsl script that will import the hdf5 files from Tensorflow, identify how many files are there (kernels + bias) and save them as datatables to then translate it to a form readable and usable as a surrogate by JMP. The way I am opening the datatables so far is very manual; I use 

Open( "filename.h5", {"list_of", "dataset_names"});

to import all the dataset names. However, in order to be able to work with the datatables, I am then using this manual way (this is for a Keras 2-layer ANN, the last kernel and bias layer concatenates the others together) : 

layer_0_bias = Data Table( "-model_weights-dense-dense-bias-0" ) << Get As Matrix();
layer_0_kernel = Data Table( "-model_weights-dense-dense-kernel-0" ) << Get As Matrix();
layer_1_bias = Data Table( "-model_weights-dense_1-dense_1-bias-0" ) << Get As Matrix();
layer_1_kernel = Data Table( "-model_weights-dense_1-dense_1-kernel-0" ) << Get As Matrix();
layer_2_bias = Data Table( "-model_weights-dense_2-dense_2-bias-0" ) << Get As Matrix();
layer_2_kernel = Data Table( "-model_weights-dense_2-dense_2-kernel-0" ) << Get As Matrix();

 

What I want to do is automate the process of importing, saving the datatables, and renaming them and understanding how many layers are in the Keras model (number of datatables / 2). Is there a way to understand how many datatables are there and also renaming each one separately without having to do the manual process shown above?

 

Thanks! 

 

 

9 REPLIES 9
Highlighted
klk
klk
Level II

Re: Automate opening an hdf5 file

Hi,

I am also looking to do something similar with hdf5 files - get the number and names of 'datasets' and do something intelligent with it. As I'm sure you've discovered the documentation is quite sparse in this area. I emailed jmp support to ask about doing this, if I hear anything back I'll post here. If you end up figuring this out yourself could you please share your solution(s)?

Thanks!
Highlighted
ekallou3
Level II

Re: Automate opening an hdf5 file

If I find the solution to this for sure I will share it. So far no luck however.

Highlighted
ekallou3
Level II

Re: Automate opening an hdf5 file

Unofrtunately I still ahve not found a solution to this. It is not the main feature of my application so I will not bother anymore. But if I bump into something I will inform for sure!

Highlighted

Re: Automate opening an hdf5 file

JMP doesn't really provide a rich JSL feature set for HDF5 at this time.  We have open feature requests to add JSL support to query table names and to merge like datasets.  We are considering these additions for the future.  If there is support beyond this that you feel would be helpful, please convey that to Tech Support and it will help us prioritize future work.

 

Thanks, and I'm sorry we can't be of more help right now.

 

Brian Corcoran

JMP Development

 

 

Highlighted
klk
klk
Level II

Re: Automate opening an hdf5 file

Thanks for the reply. At least for me, if I could query the structure of the HDF5 file I'd be happy to script the rest myself.
Highlighted
klk
klk
Level II

Re: Automate opening an hdf5 file

Actually another question here - it seems that JMP assumes an HDF5 file will use the extension 'h5'. If my files have a different extension, say 'hdf5', can I force JMP to use that file type with the Open() command? The File Type() argument doesn't seem to work for HDF5.
Highlighted

Re: Automate opening an hdf5 file

You are correct, in that JMP assumes .h5 is the only extension for a HDF5 file.  This does seem to be the norm, but .hdf5 and .hdf are accepted alternatives.  We will see about getting the file dialog and JSL open commands to support the .hdf5 and .hdf extensions in JMP 15.2.  Certainly it will be in 16.0.

 

Brian Corcoran

JMP Development

Highlighted
robert_j_moser
Level III

Re: Automate opening an hdf5 file

Hi Brian,

I've only found one way to generate an HDF5 file from Python that JMP can read directly into table format...meaning the GUI wizard doesn't ask me which subset(s) of the HDF structure to load.  I don't want to paste all the code, but the basic path is to convert the Pandas dataframe to a Numpy recarray, and write this to HDF5.

 

BUT...it's absurdly slow to open this .H5 file in JMP.  For a moderate-sized file (131k rows x 500 cols), it's about 10x slower than the same data stored as CSV.

 

Can you provide example Python code that stores table-like data in an HDF5 file that JMP can interpret and open directly, with reasonable performance for the volume of data?  Or a Python module that writes a .JMP file would be great.

 

The Python interface appears to save the Pandas dataframe as a .csv to open in JMP...hopefully that's not because .csv is the fastest intermediary.

 

Thanks,

- Bob

 

 

Highlighted

Re: Automate opening an hdf5 file

I'm sorry Bob, but I don't have any such sample code or really the time to develop it right now.

 

Brian

Article Labels

    There are no labels assigned to this post.