topic Re: How to calculate TF-IDF in Discussions

How to calculate TF-IDF

markw — Fri, 09 Jun 2023 00:25:42 GMT

Dear all,

I am new to using this software and am struggling to figure out how to use it to calculate the TF-IDF values for documents in my data set.

I am sure it is included in the functionality of JMP but it must be under a different name. Would anyone be able to point me in the right direction?

Thanks

Mark

Re: How to calculate TF-IDF

Georg — Thu, 03 Dec 2020 16:43:59 GMT

you may have a look at text explorer (main menu analyse),

please see the following example as script, or in scripting index, or in statistics index (main menu help).

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Consumer Preferences.jmp" );
obj = dt << Text Explorer( TextColumns( :Reasons Not to Floss ) );

Re: How to calculate TF-IDF

markw — Thu, 03 Dec 2020 16:50:59 GMT

Thank you for your reply Georg. Please note I am a complete beginner so in need of a very basic explanation...

In practice I have a set of x documents, for 2 of which I would like to calculate the TF-IDF.

When I go to text explorer, can I get JMP to calculate the TF-IDF through using menu options and how do I specify for which documents I would like the TF-IDF to be calculated?

Re: How to calculate TF-IDF

mlo1 — Thu, 03 Dec 2020 17:12:44 GMT

Hello

did you check out the "Local data Filter" to chose the documents you want.

Re: How to calculate TF-IDF

Georg — Thu, 03 Dec 2020 17:42:29 GMT

My way would be to get the text first imported into a table (JMP can Import even full Folders via multiple file Import …),

and then using text Explorer on that data table.

So this depends on type and structure of your files.

Re: How to calculate TF-IDF

mlo1 — Thu, 03 Dec 2020 20:36:50 GMT

May be it is worth to check out the following AddIn as well

Re: How to calculate TF-IDF

Mark_Bailey — Thu, 03 Dec 2020 19:22:11 GMT

The TD-IDF is one of the built-in transforms that may be applied to the document-term matrix when you save it back to the data table from the Text Explorer platform.

Re: How to calculate TF-IDF

P_Bartell — Thu, 03 Dec 2020 20:51:53 GMT

Have you looked at the JMP documentation, Text Explorer section? Here's a link: Text Explorer

Much will depend on how your curate your documents...so pay attention to that first before any analytics.

Re: How to calculate TF-IDF

Martin1407 — Fri, 04 Dec 2020 10:03:08 GMT

Hi Mark,

I've got the same issue. Need to solve this as part as an Assignment for my Degree. Did you find a solution?

Best,

Martin

Re: How to calculate TF-IDF

markw — Fri, 04 Dec 2020 11:25:30 GMT

Thanks everyone for your help, with your collective help I have managed to find the solution:

Under text explorer, go on save document term matrix and the option for TF-IDF is found under weighting.

Re: How to calculate TF-IDF

MNS — Wed, 08 Dec 2021 12:05:18 GMT

Hey Martin . Can you explain this a little further? When you are in text explore you save ? I don’t see ‘term’?
Thanks
Any help appreciated for a new user

Re: How to calculate TF-IDF

Georg — Wed, 08 Dec 2021 15:20:38 GMT

red triangle in Text Explorer, see screenshot.