Converting Torch Addin files to a model

[#1]

Putting this here in case anyone is able to understand the docs more than me, but I'm trying to just load any model output from the torch addin into python.  Anyone have any luck doing this?  I'm hoping that I can just load the state dict of the given model I selected but that doesn't seem to be working.  

# %%
from pathlib import Path

import torch
from torchvision.models import efficientnet_b2
# %%
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_path = Path(__file__).parent.parent/"models/VI Models/E353 EffNetB2 MLP DOR20"
model_paths = list(model_path.glob("*v1f1*.pt")) # just pick one model for now

# %%
model = efficientnet_b2(dropout=0.2).to(device)
# %%
model_paths[0].relative_to(model_path) # WindowsPath('tdl_jmp_v1f1_jnet.pt')
# %%
model_part = torch.jit.load(model_paths[0], map_location=device)
model_part
# %%
model.load_state_dict(model_part.state_dict())
# this fails with 
# RuntimeError: Error(s) in loading state_dict for EfficientNet:
#     Missing key(s) in state_dict: "features.0.0.weight", "features.0.1.weight", "features.0.1.bias", "features.0.1.running_mean", .... very long list
#     Unexpected key(s) in state_dict: "0.0.0.weight", "0.0.1.weight", "0.0.1.bias", "0.0.1.running_mean", .... very long list
# %%

[#2]

@vince_faller   Thanks for the post.   After 

model_part = torch.jit.load(model_paths[0], map_location=device)

the model is ready to go for predicting images.  There is no need to instantiate efficientnet_b2 from torchvision and load a state_dict.   The just-in-time compiled model file contains both the model architecture and trained weights.

Just run model_part(image) after setting up an image tensor.   After this there are three more steps:  pooling, embedder, and final linear predictor.

I'm attaching an end-to-end example notebook and example image, including steps to show intermediate outputs.   Please change the suffix on deploy_image_model.py so the file is named deploy_image_model.ipynb 

[#3]

that's the same one that's in the documentation and doesn't really help.  How do we know from a screenshot of the Addin Dialog what any of these are.  like how do we know the normalization method or the imagenet normalization values to use?  How do we know which pooler to use (the architecture is supposed to be in JIT?)?  I'm confused why you're doing tnet and tnet1?  Just showing that they match?  Sometimes y'all use the 1 version of the layer sometimes you don't.  

Maybe my problem is that all I get are these files and this screenshot.  

Sorry for my ignorance.  And thanks for your patience.  

[#4]

@vince_faller It is admittedly confusing and we'll be working to streamline it.   There are basically four pieces in the following order:

1. Image Model:   architecture and weights contained in the *_jnet.pt model file. 

2. Pooling:   Matches Pooling Layers specification
3. Tabular Model:  matches all other parameters, and final output has dimension embed_size, which is the last value of Layer Sizes.
4. Final Linear:  must be nn.Linear(embed_size, y_size)   where y_size is the dimension of Y that is being predicted. 

After each step you should be able to take input from the previous one and create new output and build up the model step-by-step.

If something is not aligning, it can be very helpful to look at the state dicts and dimensions of output tensors to see where mismatches are and adjust accordingly--that's why we put those extra printout cells in the example notebook.

If you have a new example you would like to see worked out, please send it.  

[#5]

Thanks to the efforts of experts, deep learning is possible at JMP.

I would also like to ask a question about JMP image recognition, although I am still unable to obtain JMP Pro 18 software:

If the images I use to identify them are all generated from known data tables.When using the Torch plug-in for identification, can you import the original table data directly?

But I think it would be better to see it through an image, you know?Does that make sense?

Thanks Experts!

[#6]

Hi @lala   Thanks for your question.   I am not quite following what you mean by "import the original table data directly".   If you mean importing images from .png or .jpg files on disk, that is fairly easy with File > Import Multiple Files.   If this is not what you are asking, if you could please explain further or perhaps provide a small example I can comment further.

In the mean time, even without JMP Pro 18, it might be informative to download Torch_Storybook.jmp from https://community.jmp.com/t5/JMP-Add-Ins/Torch-Deep-Learning-Add-In-for-JMP-Pro/ta-p/733478 (see link near upper right hand corner), open it in an earlier version of JMP, and click on various links of interest see how image data is being handled in various examples.  There are also links on the preceding page to a few video tutorials, including one on image classification.   These may help answer your question.   

[#7]

Thanks for the expert's reply.

I came up with a more intuitive example:

Just like stock price charts from day to day, they are generated by changes in "Open","High","Low","Close","Volume" from day to day.

It is possible to predict the future price change directly with the change rule of the original data, and of course, it is also possible to use the different forms of the stock price chart to predict through graph recognition.

Just like that, JMP 18's Torch plug-in can directly import raw stock price data for training, omits the image recognition process.

Is there a difference between the two types of training?

Thank you very much for the expert's help.

[#8]

Hi @lala    Well I'm sure advanced traders have tried nearly every conceivable way of predicting future stock prices.

Images are a certain way of encoding data using pixel locations and colors.   While deep neural training on images is certainly different from training on the tabular time-series features, in the end similar input data in both forms will tend to produce similar results.  

We're seeing a similar phenomenon when training on spectral data using either functional or image inputs in JMP.   See the Torch Storybook for a few examples.