topic Re: Entropy in Discussions

Entropy

Lu — Fri, 09 Jun 2023 00:31:53 GMT

Hello,

I want to perform some basic descriptive statististics on time series data such as mean, median, kurtosis, etc.. but also the degree of variability over time which is described in the literature as "Entropy". I can not find this in JMPpro. Anybody a suggestion how to calculate this?

Re: Entropy

P_Bartell — Sun, 11 Apr 2021 11:38:01 GMT

Can you provide a literature citation or two for 'entropy'? Truth be told, I understand the definition of entropy from a physics point of view...but it's a term unfamiliar to me wrt to statistics, data visualization, or analysis. Maybe with a citation we could suggest a path forward in JMP?

Re: Entropy

Thierry_S — Mon, 12 Apr 2021 06:08:25 GMT

Hi Lu,

Here is the simples description I could find on how to calculate Shannon Entropy (see file attached and source information):

Calculate the frequency of occurrence of each symbol / case = px (Distribution > Frequencies > Make Into Table)
In the resulting table ==> New column "B"= px * Log2 (px)
In the resulting table ==> New column "H" = - Col Sum (:B) or if you prefer to express in terms of bits : -ROUND (Col Sum (:B),0)

Best,

Re: Entropy

Thierry_S — Mon, 12 Apr 2021 06:19:00 GMT

Hi Lu,
I forgot to mention that this is only applicable of discrete symbols / cases because the entropy of a sequence of continuous data (i.e. real numbers) is essentially infinite.
Best,
TS

Re: Entropy

Craige_Hales — Mon, 12 Apr 2021 08:49:40 GMT

Cool! .zip files do this. If you've ever compressed a .zip a second time, you might have noticed it does not help; the zipped data already has nearly maximum entropy (looks random) and can't be compressed much more and may get a little bigger to hold the header information.

Re: Entropy

Lu — Wed, 14 Apr 2021 14:08:31 GMT

Hi,

I attached a recent publication in which entropy statistics was used on time series data; See feature engineering (page 4).

As I understand, entropy describe the change in variability over time.

Regards

Ludo

Re: Entropy

Lu — Wed, 14 Apr 2021 14:10:56 GMT

Hi,

Thanks for the support, but this is not what I needed. My data are indeed continuous.

Regards

Re: Entropy

Thierry_S — Wed, 14 Apr 2021 14:36:20 GMT

Hi Lu,
Sorry to hear that you were thinking of something different. For my education, could you point me to a reference or resource that defines or uses Entropy for continuous variables?
Best
TS

Re: Entropy

Lu — Wed, 14 Apr 2021 14:43:56 GMT

Hi Thierry

See my reply to P Bartell and the attached file above.

regards

Re: Entropy

Thierry_S — Wed, 14 Apr 2021 14:47:31 GMT

Thanks.
I'll continue searching for possible reference on how Entropy is calculated for continuous variables.
Best,
TS