- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Automate opening an hdf5 file
Hello,
I am trying to create a .jsl script that will import the hdf5 files from Tensorflow, identify how many files are there (kernels + bias) and save them as datatables to then translate it to a form readable and usable as a surrogate by JMP. The way I am opening the datatables so far is very manual; I use
Open( "filename.h5", {"list_of", "dataset_names"});
to import all the dataset names. However, in order to be able to work with the datatables, I am then using this manual way (this is for a Keras 2-layer ANN, the last kernel and bias layer concatenates the others together) :
layer_0_bias = Data Table( "-model_weights-dense-dense-bias-0" ) << Get As Matrix();
layer_0_kernel = Data Table( "-model_weights-dense-dense-kernel-0" ) << Get As Matrix();
layer_1_bias = Data Table( "-model_weights-dense_1-dense_1-bias-0" ) << Get As Matrix();
layer_1_kernel = Data Table( "-model_weights-dense_1-dense_1-kernel-0" ) << Get As Matrix();
layer_2_bias = Data Table( "-model_weights-dense_2-dense_2-bias-0" ) << Get As Matrix();
layer_2_kernel = Data Table( "-model_weights-dense_2-dense_2-kernel-0" ) << Get As Matrix();
What I want to do is automate the process of importing, saving the datatables, and renaming them and understanding how many layers are in the Keras model (number of datatables / 2). Is there a way to understand how many datatables are there and also renaming each one separately without having to do the manual process shown above?
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
I am also looking to do something similar with hdf5 files - get the number and names of 'datasets' and do something intelligent with it. As I'm sure you've discovered the documentation is quite sparse in this area. I emailed jmp support to ask about doing this, if I hear anything back I'll post here. If you end up figuring this out yourself could you please share your solution(s)?
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
If I find the solution to this for sure I will share it. So far no luck however.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
Unofrtunately I still ahve not found a solution to this. It is not the main feature of my application so I will not bother anymore. But if I bump into something I will inform for sure!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
JMP doesn't really provide a rich JSL feature set for HDF5 at this time. We have open feature requests to add JSL support to query table names and to merge like datasets. We are considering these additions for the future. If there is support beyond this that you feel would be helpful, please convey that to Tech Support and it will help us prioritize future work.
Thanks, and I'm sorry we can't be of more help right now.
Brian Corcoran
JMP Development
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
You are correct, in that JMP assumes .h5 is the only extension for a HDF5 file. This does seem to be the norm, but .hdf5 and .hdf are accepted alternatives. We will see about getting the file dialog and JSL open commands to support the .hdf5 and .hdf extensions in JMP 15.2. Certainly it will be in 16.0.
Brian Corcoran
JMP Development
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
Hi Brian,
I've only found one way to generate an HDF5 file from Python that JMP can read directly into table format...meaning the GUI wizard doesn't ask me which subset(s) of the HDF structure to load. I don't want to paste all the code, but the basic path is to convert the Pandas dataframe to a Numpy recarray, and write this to HDF5.
BUT...it's absurdly slow to open this .H5 file in JMP. For a moderate-sized file (131k rows x 500 cols), it's about 10x slower than the same data stored as CSV.
Can you provide example Python code that stores table-like data in an HDF5 file that JMP can interpret and open directly, with reasonable performance for the volume of data? Or a Python module that writes a .JMP file would be great.
The Python interface appears to save the Pandas dataframe as a .csv to open in JMP...hopefully that's not because .csv is the fastest intermediary.
Thanks,
- Bob
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Report Inappropriate Content
Re: Automate opening an hdf5 file
I'm sorry Bob, but I don't have any such sample code or really the time to develop it right now.
Brian