topic Re: How to update efficiently a table that contains a prediction model based on DTM (Document Term Matrix)? in Discussions

How to update efficiently a table that contains a prediction model based on DTM (Document Term Matrix)?

Florent_M — Sat, 10 Jun 2023 20:52:14 GMT

Hello!

I work on textual data classification. So I made a model that predicts the class by taking as input the DTM. Now, I would like to update, in a regular and efficient way, the data table with new rows. My problem is to be able to easily compute the DTM using a formula so that the update is automatic.

Here is a file based on the example of potato chip reviews:
- first, we use the text explorer to save the DTM
- 2nd, we use a neural network to predict the column "Buy again?"

From there, I'd like to add rows to the table, and have the model predictions computed automatically, without having to relunch the text explorer.

Do you have any suggestions?

Thank you,
Florent

Re: How to update efficiently a table that contains a prediction model based on DTM (Document Term Matrix)?

Mark_Bailey — Fri, 26 Aug 2022 13:46:01 GMT

The DTM is saved as values. Did you try saving it as a formula? Does it update when a new row is added to the table?

Re: How to update efficiently a table that contains a prediction model based on DTM (Document Term Matrix)?

Florent_M — Tue, 30 Aug 2022 09:01:18 GMT

I tried to save the DTM as Formula. Unfortunatly, the formula return an array, which cannot be used in a predictive model.
Do you have an idea to use directly the array in a model or to convert an array into columns?

Thanks,

Re: How to update efficiently a table that contains a prediction model based on DTM (Document Term Matrix)?

Mark_Bailey — Tue, 30 Aug 2022 14:09:43 GMT

Yes, it can be used as a set of predictors. Did you try?

Table with saved formula:

Logistic regression with array as only model term, but expands to all terms in the DTM: