JMP Blog articles

We’re doing things a little differently with Discovery Summit Europe

AllisonHines — Thu, 18 Nov 2021 16:27:34 GMT

Join us throughout March for Discovery Summit!

Discovery Summit brings statistically minded people together to tackle the challenge of improving how data is explored, analyzed and shared. These events facilitate statistical conversations between scientists and engineers, professors and students, statisticians and other data explorers alike.

We’re excited to announce that your next chance to {virtually} mix and mingle with like-minded folks is just around the corner.

But we're doing things a little differently in 2022. Instead of a conference over the course of several consecutive days, we’re spreading it out over the course of a couple weeks in March.

Don't fret – you'll still get the same data discovery and statistical inspiration as before.

Registration is free and will open Dec. 13.

The event is virtual so that everyone can experience the power of statistical discovery. All the things you know and love about Discovery Summit will remain online: conversation-starting keynotes, simu-live papers, poster presentations, Ask the Experts sessions and networking opportunities.

Although it is too early to register for the event, it’s not too early to block your calendars and add this: “I’m busy exploring data, challenging assumptions and learning analytic best practices with some of the world’s most sophisticated statistical thinkers. If you think you can keep up, join me. It all starts at jmp.com/discovery-europe.”

And don’t forget to add your name to our mailing list at jmp.com/discoverysummit.

Impulse sus esfuerzos de mejora continua con JMP

JMP_is_fun — Thu, 18 Nov 2021 13:45:26 GMT

La mejora continua se trata de un esfuerzo continuo para mejorar productos, servicios o procesos. El proceso de evaluar que tan bien sus métricas cumplen con los requisitos de sus clientes puede ser difícil, engorroso o lento, dependiendo de la herramienta estadística usada para la evaluación.

Imagínese que usted tiene docenas o cienes de mediciones que se deben de evaluar con regularidad. Sin embargo, también tiene otras responsabilidades y no tiene horas o días para evaluar cada medición individualmente. La plataforma de cribado de proceso en JMP fue programado con usted en mente, y ahorrara tiempo y dinero, permitiéndole evaluar todas sus mediciones rápidamente de forma visual.

Abajo, puede ver la plataforma de cribado del proceso en JMP, que ofrece una interfaz sencilla donde puede ingresar sus mediciones con solo un clic. En este ejemplo, se evalúan 128 métricas, cada una contra sus límites de especificación correspondientes.

Los resultados que se presentan por defecto para cada variable están ordenados por el índice de estabilidad, que es una medida de la estabilidad del proceso, ya que, para evaluar los índices de capacidad, los datos deben de provenir de un proceso estable. También tiene la opción de ordenar los resultados por los índices de capacidad como el Cpk o el Ppk.

Uno de los grandes beneficios que JMP le ofrece, es que una vez que ha corrido su análisis, puede fácilmente agregar otros elementos a los resultados, brindando la flexibilidad que desea para ver la información que necesita. Por ejemplo, se puede agregar el índice de capacidad Cp a la tabla mostrada arriba. O también se pueden agregar otras gráficas para eficientemente evaluar sus mediciones de forma visual, como las dos gráficas mostradas abajo.

El gráfico de portería es útil para rápidamente ver como las variables se ajustan a los limites de especificación. El triangulo en la parte inferior representa a su meta para el Ppk; el deslizador hacia el lado derecho le permite cambiar su objetivo Ppk de manera interactiva. Las zonas verdes, amarillas y rojas representan que tan bien sus variables se conforman con el Ppk seleccionado. También puede obtener más detalles sobre cada variable en la gráfica de forma interactiva, y puede anclar a los detalles como mostrado en la gráfica de control para la variable IVP2. Para más información sobre la gráfica de portería, por favor vea esta página.

El gráfico de rendimiento del proceso es una gráfica de cuatro zonas que permite visualizar a la capacidad de cada proceso contra la estabilidad. La leyenda hacia el lado derecho le ofrece una interpretación fácil y eficaz. Y como todas las gráficas en JMP, esta también le ofrece la interactividad que permite obtener mas detalles sobre cada variable. Mas detalles sobre el gráfico de rendimiento del proceso están disponibles en esta página.

Si aún no ha usado a JMP, los invitamos a descargar su prueba gratuita de 30 días. Pruébelo con sus datos y descubra lo fácil y divertido que puede ser su análisis. Si prefiere agendar una presentación personalizada con nuestro equipo para aprender más, comuníquese con nosotros. ¡Hablamos español!

JMP para optimización, fabricación y calidad de fármacos: Parte 3

JMP_is_fun — Wed, 17 Nov 2021 14:41:07 GMT

Este video es parte de una serie de tres presentaciones para la industria farmacéutica.

En el primer video (JMP para optimización, fabricación y calidad de fármacos: Parte 1 - JMP User Community), mostramos el diseño de experimentos, el ajuste de un modelo estadístico, y finalmente optimizamos el proceso a partir del modelo. En la segunda sesión (JMP para optimización, fabricación y calidad de fármacos: Parte 2 - JMP User Community), nos enfocamos en la manufactura, la visualización de variables de producción y usamos el aprendizaje automático para análisis de causa raíz, lo que junto a una simulación Monte Carlo, nos ayudó a encontrar una solución inmediata para un problema de baja disolución de tabletas de dosificación oral.

En esta tercera y última sesión, evaluamos la capacidad y control de una multitud de variables de fabricación a la vez para identificar los procesos que más necesitan atención y así enfocar los esfuerzos de mejora, y realizamos un análisis de estabilidad para establecer la fecha de vencimiento de un medicamento según las pautas del ICH.

Data visualization: Facilitating the analytic workflow for scientists and engineers

anne_milley — Wed, 17 Nov 2021 14:02:58 GMT

If you missed the Statistically Speaking episode on Data Visualization for Scientists and Engineers, the on-demand version is now available. We heard some really inspiring perspectives on the importance of data visualization throughout the whole analytic workflow!

Amoolya Singh, Head of Discovery Technologies at Calico Labs gave an amazing plenary talk, “Learning to see patterns in data: How data visualization facilitates the analytic workflow.” Who knew that weaving, computing and visualization were so interconnected? Fascinating!

Amoolya also joined the panel discussion with Kathleen Schneider, Senior Research Associate at Lundbeck; and Scott Wise, Principal Analytic Training Consultant at SAS. Since we didn’t have time to answer all the questions from our audience, our panelists have kindly agreed to answer some of them here.

Many visuals show data separated by color. What accommodations would you suggest for colorblind people?

Scott: Xan Gregg, Director of Visualization Development at JMP, has some good recommendations on graphing in JMP for colorblind people:

For categorical colors, the "universal" color theme supports all color deficiencies; it is derived from "Barrier-free presentation that is friendly to colorblind people."
For continuous colors, the sequential themes are good because they vary in brightness along the scale.
There are some good tips found in JMP Community Discussions.

While the node graphs and the tSNE plots are both data-rich, I often find them information-poor. I feel as though I have not developed the graphical literacy to interpret them. I always wish for graphs and displays that have a pre-attentive quality to them, such that I can see clearly which bits are informative. Any suggestions?

Scott: Both node graphs (network diagrams, etc.) and tSNE plots (PCA results, etc.) really need to include the context and story in the graph (either verbally explained or featured as text on/around the graph) for you to really understand “how” to view the patterns in the graph. Don’t be afraid to incorporate the telling of the story in conjunction with these useful graphs.

What impact will Web 2.0 have on data science and analytics?

Note: Web 2.0 is the second stage of development of the World Wide Web, characterized especially by the change from static web pages to dynamic or user-generated content and the growth of social media.

Scott: Web 2.0 is already here (as seen in the rise of social media). If you aren’t gathering this dynamic data source to analyze and visualize trends (like using text mining on social media about your company, products, patents, competitors, etc.) then you already behind the curve. For a good introduction, check out this “Thriving in a Web 2.0 World” blog from SAS.

3D graphs were brought up, but I've found that they are hard to share in a fixed format. How would you recommend sharing 3D graph findings?

Scott: The first rule is to stay away from 3D graphs if they do not add value to the viewer, which is why you don’t see 3D bar charts in JMP since they are often confusing and don’t add any value.

The second rule is to ensure that when 3D graphs are valuable, make sure to use them interactively (like with filters, interactive HTML5 formats, or even with videos) instead of leaving them in a fixed, static format. Doing so allows the user to properly interact with and interpret them. For example, the JMP 3D scatterplot (as featured in our panel discussion on checking three factor experimental settings) is considerably less effective outside of JMP if you aren’t able to move the 3D cube around to change the viewpoint orientation of the points within the image. So, for people who don’t use JMP, try simulating the interactivity of exploring a 3D scatterplot by setting up local data filters that change the viewpoint orientation over a period of time and save this out as an interactive HTML5. Another option would be to create a short video recording yourself exploring the 3D graph that the user can play back.

How have you seen an animated bubble plot used at different companies and industries? Can they be annotated as the animation occurs?

Scott: I have seen animated bubble plots and maps used in nearly all industries (high tech, healthcare, government, etc.) and in many different organizational areas. In manufacturing/supply chain, animated bubble plots are used successfully to show improvements or trends in products over time (like my example in the panel discussion). In research, animated bubble plots are often used to bring life to growth or changes in products/components throughout their R&D cycles. In sales, animated bubble maps excel in showing sales performance over time across geographic regions. You can create annotation directly on bubble plots, but they won’t move over time. Instead, work with the bubble labelling to show desired annotation information over time.

I teach AP statistics so I may be the only one here who is not in the private sector. As we try to grow interest in this field, I am always looking for videos, TED talks and guest speakers who can walk through a collection of real-world, present-day data and create a visualization and demonstrate the process step by step, so my students can see how particular careers might apply to what we are learning. Do you have any suggested videos?

Scott: For teaching graphicacy, the formula that always seems to work is engaging your students early with compelling graphics that tell a story and make the statistics come alive. I can think of no better example than showing Hans Rosling’s 2007 TED talk “The best stats you’ve never seen.” If they don’t get excited after watching Hans interact with the chart, they probably don’t have a pulse!

Also, the visualization themed JMP Blogs in the JMP Community are another great place to get intriguing stories of visualizations enhancing statistics over many interesting topics. Make sure to check out the contributions from me, Ryan DeWitt and Byron Wingerd. Also, the graphics in the JMP Public site has an array of interesting and often interactive graphics that tell a story!

Please define graphicacy.

Scott: I like the definition in Wikipedia: “Graphicacy is defined as the ability to understand and present information in the form of sketches, photographs, diagrams, maps, plans, charts, graphs and other non-textual formats.” This definition emphasizes that graphicacy encompasses both understanding and presentation. It also shows that it can happen via many different creative forms.

How JMP can help the user to evaluate not only statistical significance, but also practical significance?

Scott: Practical significance means showing the magnitude of the difference or effect size. Therefore, I think the best way to combine practical and statistical significance is to always include the graph (which demonstrates the practice significance) next to the statistics (which addresses the statistical significance). Begin with the graph and then show the statistics, as this seems to work best with how our minds process differences in information.

Kathleen: I work in the biopharmaceutical industry designing downstream (purification/separation) processes so that they can be scaled-up and transferred to manufacturing facilities. Frequently, the processes that I help design are complex and have many interactions. Typically, when I am using JMP, I am trying to solve practical problems including increasing process efficiency (greater yield or shorter process times) or mitigating risk (reducing impurities and risk of process failures). Often, increasing the yield or process efficiency will come at the cost of increasing the impurities or increasing the risk of process failure.

The process steps that we design have multiple substeps. Sometimes early process conditions can lead to increased process impurities or other negative outcomes that are not observed until later process steps. Due to the complexities of the interactions, using JMP for design of experiments and multivariate analysis – including evaluating the statistical significance of main factors, squared factors (whether the effect is linear or has a curve) and two-factor interactions – is a powerful way to evaluate which main factors and two-factor interactions are the most important factors to track and to modify, and which factors are not critical for the process. Using JMP for visualizing this data is also very powerful. The data is used to improve the process or reduce process- and product-related impurities or mitigate process failure. The data can also provide a guide for optimizing a process to produce the highest process efficiency with the lowest levels of impurities and lowest risk of process failure.

As an example, I was working on a process step as part of a team trying to reduce Impurity U. There was much discussion in the team about whether Factor A or Factor B was more important to the level of Response U. We discovered that we were all wrong. The critical factor turned out to be Factor C! Fortunately, we had been tracking Factor C and had data available to track the factor in the process. From this and other experiences, I recommend tracking as many factors as you can, including factors that you think may not be important. You might be surprised.

We appreciate these experts taking the time to provide their perspectives on some of the many questions we received. The rich experiences they have are evident in these answers, as well as in the on-demand version of this episode of Statistically Speaking.

Cringeworthy statistics statement #2 explained: Statistical significance vs practical importance

JerryFish — Tue, 16 Nov 2021 19:58:48 GMT

We've all heard someone say something that isn't "right." Sometimes you just let it go, either because it isn't important enough to challenge, or you don't want to cause conflict with the speaker.

This is the second "cringeworthy statistics" blog series installment. In general, here's how it works: First, I post some kind of statistics-related question in a JMP Community Discussions thread. Community users are encouraged to reply. After about a week, I give my take on what is wrong with the statement in a blog post like this.

Here is a table of discussion topics and blog posts to date. Many thanks to those who reply to each discussion post (and/or add comments to the blog!)

Discussion Thread Topic	Blog Post	People who Responded to Discussion (or commented on Blog)
#1: Misinterpretation of p-value	Dangers of misinterpreting p-values	@dale_lehman, @P_Bartell, @statman , @ih, @Georg, @brady_brady
#2: Too much emphasis on statistical significance	This blog post	@P_Bartell, , Craige_Hales, @ih,
#3: Reporting Measurements	Coming soon!	Please post your comments in the Discussions space. Also feel free to contribute cringeworthy statistics statements of your own!

So let's take a look at the second cringeworthy statement I posted...

My take on Cringeworthy #2

Recall in our scenario that a study had been done to try to determine why yield had fallen in a production process. A t-test was performed to study differences between two suppliers' subassemblies. Indeed, the t-test showed a very low p-value, giving high confidence of a difference in the means of the parts from the two suppliers.

The team manager wanted to jump at the fact that the p-value was small. He wanted to immediately start a study on why the two population means were different. (Perhaps he was driven by past experience that a difference between these vendors had once caused a problem.) But what the manager didn't mention was whether the difference was "practically important."

In my experience, people often confuse "statistical significance" with "practical importance."

Statistical significance is simply a mathematical statement that there we can detect some amount of difference in the means of the two populations at some level of confidence. This is nice, but it says nothing about whether that difference is important to us.
Practical importance is based on your judgement, experience and wisdom. Maybe the statistical test detects a difference of 0.1 units between the two treatments. Is a difference of 0.1 important to your application? If not, then perhaps devoting resources to discovering the source of this difference would be better spent looking for other sources that might lead to poor yield.

Maybe it is unfortunate that the term "significance" can be interpreted as "importance" in the English language. I don't know of a better word to use than significance, but it can be misleading to some people.

Moral of this story: Make sure to understand the difference between statistical significance and practical importance when interpreting statistical analytic results!

Now, please come on over to Cringeworthy Discussion #3!

Mark your calendar for The Effective Statistician’s 200th episode

anne_milley — Mon, 15 Nov 2021 15:27:40 GMT

Are you a statistician or analyst interested in learning how to lead and influence others? Then join the livestream of the 200^th episode of The Effective Statistician podcast on Nov. 30 at 2:00 CET. JMP’s own Sam Gardner will co-moderate the podcast along with its founder, Alexander Schacht, Executive VP of Launch and Commercialisation Data Sciences at Veramed; and Benjamin Piske, Senior Director of Biostatistics at Cytel.

The lineup is impressive and includes one of our favorite data visualization experts, Alberto Cairo, author of How Charts Lie: Getting Smarter about Visual Information. In addition, presenters from the 20 most frequently downloaded podcasts return to share important lessons learned. It'll be live, and you're invited to interact with the presenters by asking your questions and sharing comments. See how Alex describes it in this video.

Don’t miss this wisdom-packed episode. It will be live-streamed. All you need to do to join is register and be ready to learn from experts as they discuss their amazing work.

JSL自動化分析，4步驟讓你的報告自動「跑」起來！

Michelle_Wu — Mon, 15 Nov 2021 14:34:05 GMT

近來有不少使用JMP程式設計語言（JMP Script Language，JSL）的朋友向JMP諮詢：

「我的日常報告非常多，是否可以自動地運行編譯好的JSL分析程式？

這樣就可以每次需要運行JSL程式的時候不必長時間耗在電腦面前，打開JMP程式，點擊自訂的菜單就行了。更重要的是，有時分析程式的運算量比較大，耗時長，為了避免需要使用報告時不必等待太久，就不得不提前執行分析程式。」-- 來自頭疼的Peter

以上的情境是不是很似曾相識呢？為了減輕日常工作繁重的工程師們，本篇文章就要來分享，JMP軟體內的強大功能 – JSL 自動化，幫助你自動化產出分析報表。先來看一下在半導體產業工作的工程師Peter，最近遇到了哪些問題：

Peter手上有個需要通過JSL執行分析的品質報告，需要每天8:30上班時就向管理層彙報，但JSL運行所需時間的約為1小時，為了安全起見，他在每天早上7:00便需要開始啟動分析，不過，這麼早就要到公司，對Peter來說實在太痛苦啦（對小編來說也是一樣的痛苦）。有什麼方法可以幫助Peter解決這項難題呢？

使用JSL 任務計畫程式

要實現這一目標，可以使用windows系統自帶「任務計畫程式」啟動JSL分析程式。以下我們剖析步驟讓你更加了解。

步驟1：啟動任務計畫程式

首先，我們需要在Windows的「控制台」➡「管理工具」中啟動「任務計畫程式」。

圖一：打開JSL任務計畫程序

步驟2：創建任務並命名

開啟之後，點擊「創建任務」，在常規選項卡的「名稱（M）」中給任務取個名字，在這裡我們就命名為「自動分析報告」。

圖二：設定任務並命名

步驟3：配置任務啟動時間

配置任務啟動時間。在「觸發器」的選項卡中點擊「新建(N)」。

圖三：設定任務觸發器

設定任務啟動的時間為每天7:00AM，點擊確定就可以完成時間設置。

如果只需工作日執行任務，可在「每週(W)」的選項中選擇執行的日期。

圖四：設定觸發器的觸發時間

步驟4：瀏覽並點擊完成

連接啟動任務。在「操作」選項卡中點擊「新建(N)」。

圖五：選擇事先編譯好的JSL檔

通過「瀏覽(R)」控制項選擇事先編譯好的JSL檔，點擊「確定」。

此時，我們在任務列表中看到「自動分析報告」就代表大功告成啦！

圖六：設定完成示意圖

不過JSL自動化有幾個值得注意的地方，稍不留神往往會使人摸不著頭腦，怎麼也達不到預計的結果。首先，是使用筆記型電腦的朋友們要特別留意的：

問題1：在每個任務大的「條件」設置中有一個對「電源」的管理限制，預設情況下是要求「只有在電腦使用交流電源時才啟動此任務」嗎？

解答：

很多初次使用任務計畫的朋友都會有這個問題，因為沒有接電源而導致無法執行自動任務。為了避免困擾可以將取消勾選該選項，其他的條件選項也根據實際的情況酌情設定。這樣就無後顧之憂啦！(如圖七)

圖七：在設定中選擇電源選項

其次，是JMP中對於運行JSL的設置。

問題2：任務按時啟動，卻發現每次只是打開了JSL的程式檔，並未執行。為什麼會這樣呢？

解答：

出現這種情況是因為JMP的對於JSL打開方式的設定不當導致的，也即是每次打開JSL文檔是自動執行程式，而非開啟文檔編輯視窗。

解決這個問題，需要勾選JMP首選項 ➡ Windows特定中的「從最近使用的檔列表或檔流覽器選定JSL腳本後應該只運行而不打開」的選項，如圖八所示：

圖八：JMP細節設置

如此這般，JSL定期自動分析的問題就輕鬆愉快地解決了。

Peter再也不用早起做報告啦！透過今天的介紹，你可以看到JSL強大的自動化功能，若想知道更多JMP深入應用，務必要加入JMP LINE官方帳號，定期獲得實用的JMP操作技巧與線上課程資訊！

推薦閱讀：

Retirement income: What you can expect yearly and monthly

Bill_Worley — Fri, 12 Nov 2021 20:38:18 GMT

So you have decided it’s time to retire, and you know how much you have in your retirement nest egg. You been good, and you have saved what you could. But there are two big questions that always come up: Did I save enough? How long will my nest egg last?

This is not the end-all and be-all guide to retirement, but coupled with a post on what to expect from Social Security that I co-wrote with Scott Wise (), you can get a reasonable idea of what to expect as far as your monthly retirement income is concerned.

There are several options and lots of suggestions on how much you should pull from your retirement accounts to support yourself. The most prevalent suggestion is to pull 4% per year, but is that enough? Or could it be too much depending on your living situation? One other question that you need to answer is what your tax rate will be on money withdrawn from a traditional 401K. If you have a Roth IRA, you will not have as big a tax burden, but there will still be taxes to pay; so please make certain you are up to date on the rules of engagement. These change all the time and could trip you up if you are unaware.

I keep saying "you ” – and although the reality is that retirement is still several years away for us – we wanted to know what we can expect based on what we have right now. We also wanted to know if what we have is enough to keep us comfortable and maintain a standard of living we are used to at this point in our lives.

What I have done is bracket several potential amounts folks may have saved up for retirement. So, between $100,000 and $2,000,000 and eight levels in between. I am mindful that some folks have less than $100,000, and some have way more than $2,000,000. Again, this is just a guidepost. Everyone should take the financial pundits’ suggestion to heart and save for retirement early and often. Having too much money saved for retirement seems impossible to me. The more you have, the better you will live out your “golden” years.

Yearly Income

So, let’s look at some of the numbers. In the JMP graph below, you will see a side-by-side comparison of yearly income based on the size of your retirement nest egg and withdrawing at a rate of between 1% and 5%. Remember, the pundits like to say that 4% is the optimal withdrawal rate.

This graph was very enlightening for me at least. If you are lucky enough to have $1,000,000 saved and draw down at 4% annual rate your yearly income will be $40,000. Will that be enough to live on? There are a couple of things to consider here that will add some comfort. You can expect some sort of Social Security income until 2034 (that could change), and you can always take on a side gig to earn extra income. For 2022, you can earn up to $19,560 without impacting your Social Security benefits, and that is only if you haven’t reached full retirement age (FRA). Once you reach FRA, you do not have to worry about income from a job impacting your benefits.

Monthly Income

The graph below shows monthly income based on the given 1-5% withdrawal rates. This graph provided a different perspective, and it was another eye-opener to see how much monthly income can be expected compared to what you earned before retirement. You can view both graphs, side by side, at JMP Public.

In the end, it all comes down to what you have and what you need to live a comfortable lifestyle. Please use this blog post as it is intended, which is to educate yourselves on what you have now and what you hope to have in retirement. The formulas used in calculating are very simple and are found under the column headers. So if the numbers I have chosen to show don’t match your current numbers, grab the data table, dive in and make the changes. Happy retirement planning!

雙十一數據解密：電商銷售額與哪些因素有關？數據分析告訴你

Michelle_Wu — Fri, 12 Nov 2021 14:56:15 GMT

購物者的年度春晚「雙十一」購物狂歡節，你下了多少單？

雖然今年小編買的不算太多，但也算是為「雙十一」貢獻了一份力量吧！

雙十一做為電商零售一年一度的促銷大季，各家電商和品牌方都在摩拳擦掌，火力全開，品牌方想要在電商大戰中拔得頭籌，對於消費者洞察就必不可少，品牌方對消費者的購買行為瞭解地越透徹，就越能有效地幫助企業更有針對性地開展市場行銷活動。

本篇文章透過數據分析的方式，針對電商的幾個關鍵因子作分析，解釋哪些可能因子影響銷售額，幫助電商零售找到提升銷售額的洞察。

今天，我們就借助文獻Sakar, C.O., Polat, S.O., Katircioglu, M. et al. Neural Comput & Applic (2018) 中的一份公開樣本資料，結合JMP軟體來對網購使用者購買行為做一些有趣的探索性資料分析。

Step.1 資料獲取

原始資料來源

原始資料的獲取，可以通過網址下載csv格式原始資料，並通過JMP打開；也可以直接利用JMP的網頁讀取功能，直接獲取網頁端資料。

（JMP操作：文件-> 從internet打開 ->網頁）

資料背景介紹

該資料集包含12,330 名網購用戶一年內在該網站的購買行為，以及對應的17個使用者行為記錄和最終交易結果。其中17個行為記錄，包括10個數值型特徵，7個分類型特徵。

基本資訊

管理類網頁，管理類停留時間，資訊類網頁，資訊類停留時間，產品類網頁，產品類停留時間，表示用戶在不同類型網頁上的打開數量及停留時間總和。

跳出率

跳出率表示從某個特定路徑進入網站頁面，有多少百分比的用戶什麼都沒有做，就直接離開了網站，它既可作為衡量整個網站的度量，也可作為衡量頁面的度量。

退出率

退出率表示對某一個特定頁面而言，從這個頁面離開網站占所有訪問到這個頁面的百分比，一般作為衡量頁面的度量；

頁面價值

頁面價值表示使用者在完成交易之前訪問過的網頁的平均值；

特殊日

特殊日，表示網站存取時間與特定特殊日子的間隔；

其他

此外還包括使用者使用的作業系統、流覽器、區域、流量類型、訪客類型，是否為週末以及一年中的月份資訊。

Step.2 視覺化探索性分析

跳出率高低，關係到網路行銷的成功與否

客戶僅僅查看單個頁面後退出，讓品牌方很難有機會說服消費者購買產品，畢竟他們只流覽了一頁。讓我們來查看下面跳出率的情況吧。

（JMP操作：分析-> 分佈）

從圖上可以看出，90%客戶的跳出率低於6%，所有用戶的平均跳出率只有2%，是不是很完美？請先不要著急高興。它可能是不準確的，或許是網站的分析跟蹤代碼如何集成到網站出現了技術問題。

因為根據以往經驗，「正常」跳出率在 40%-60% 之間，低於 40% 是非常罕見的，高於 70% 是令人擔憂且是需要趕緊採取行動的。

當前跳出率超出預期範圍並且看起來「好得令人難以置信」，應該是網站中的某個地方重複的分析代碼所造成。

使用者數值型特徵的多元探索

在做購物行為分析的時候，使用者的數值型特徵可能維度很多，借助JMP的多元分析方法，可以快速發現各個維度之間的關係，並有可能實現降維操作，為後續的特徵監控減少不必要的資源浪費。

（JMP操作：分析 -> 多元方法 ->多元）

基於當前資料，使用者在各個不同類型網頁上的打開數量和停留時間成正相關，這個很好理解。跳出率和退出率因為計算公式相似也成明顯正相關，此外，沒有發現明顯的數值特徵相關。

使用者上網方式對銷售的影響

通過下圖卡方檢驗的統計結果，我們可以捕捉到完成交易與否與客戶的作業系統、流覽器類型和流量類型之間的關係。

（JMP操作：分析->以X擬合Y）

就作業系統而言，不同作業系統，使用者完成交易的比例是不一樣的。

通過圖形也能看出，作業系統是「2」的時候略高，而「1」和「3」則偏低，這可能意味著網站頁面對這些作業系統的支援不夠友好，如果要提升這部分的收益轉化，則需要做出相應的改進。同理，對流覽器類型和流量類型，我們也看到了他們對使用者完成交易比例的統計學影響，說明網站在這方面也有改進空間。

新老客戶和工作日/週末對銷售的影響

借助2.3部分卡方檢驗的方法，我們也能快速發現一些新老客戶和工作日/週末對銷售的規律，但這裡嘗試另一種資料表匯總的方法。

（JMP操作：分析-> 消費者研究 -> 分類）

結合上面的圖形和資料，能清楚地看到：老客戶是網站訪問的主力，說明網站在客戶維繫上做得很好；但是我們也看到，不管是在平日（13.2% vs 26.1%）還是在週末（16.5% vs 21.9%），新客戶的完成交易的比例都要高於老客戶，這說明網站可以在老客戶的轉換率上做出些改進。

例如老客戶在購買商品的時候可以通過介紹新客戶的方式來享受更大的折扣，這樣既調動了老客戶的購買熱情，也為網站增加了更多的新客戶。

多樣分析結果的集中展示

如果想把各種分析圖表以報表的形式集中展示，可以通過JMP的腳本功能，就可以一鍵實現報表連結資料的即時更新，節省大量的重複性手動操作。

（JMP操作：檔 -> 新建 -> 應用程式）

以上是一些探索性資料分析的結果。下面，我們來進一步嘗試用資料採擷的方法對上面提到的使用者主要特徵與交易結果建立量化的統計模型。

Step.3 資料預測建模

通過決策樹，篩選影響銷售的關鍵特徵

決策樹是一種有監督學習方法，能夠從一系列有特徵和標籤的資料中總結出決策規則，並用樹狀圖的結構來呈現這些規則，結果解釋方便，在各個行業和領域都有著廣泛的應用。

（JMP操作：分析 -> 預測建模 -> 分割）

從圖中我們可以觀察決策樹的各個階段，從上到下顯示影響交易結果的最重要的特徵。

其中最重要的是網頁價值，網頁價值低於0.067和高於0.067的成交比率分別為3.85%和56.4%，差距明顯。後面還有些比較重要的特徵參數也都一併列出，比如跳出率，月份和產品相關頁面等，這些資訊都是驅動交易結果的重要因素，現在可以快速被挑選展示出來，從而讓品牌方有了一個更清晰的改進優化重點。

優中選優，更多資料採擷方法的嘗試

除了決策樹，JMP還提供了諸如神經網路、隨機森林、提升樹和支援向量機等多種資料採擷的方法，並且可以輕鬆完成模型演算法之間的比較，實現優中選優。

通過JMP Pro 16 全新的模型篩選來對多種資料採擷方法一次性完成比較，在這之前，為了防止構建的模型過擬合，可以先按照訓練集，測試集，驗證集 6：2：2的比例對原始資料進行拆分，生成驗證列。

（JMP操作：分析->預測建模 ->生成驗證列）

如下圖所示，一次輸入特徵參數和交易結果，平臺會同時構建多個演算法模型，並自動篩選出當前的最佳建模方法為隨機森林，模型在測試集上的表現，也就是對將來新資料的預測能力R方達到了0.6，預測準確性達到了90.7%。

精確的預測模型可以説明品牌方儘早瞭解每一個使用者可能的交易結果，尤其是預測交易失敗的情況，提早做出應對和補救，比如打折，比如在客戶退出頁面前彈出挽留介面等。

（JMP操作：分析 -> 預測建模 ->模型篩選）

怎麼樣？看了今天的分析，是不是讓你在「電商大戰」中更有信心了呢？

對品牌方而言，提供個性化產品、服務與商品推薦，並針對個別需求做出一對一行銷，是網路行銷相對于傳統行銷的一個巨大優勢。

結合JMP資料分析軟體，您能夠透過對網購用戶消費行為的深入分析，可以幫助企業設計出更能滿足目標顧客群需求的商品集合頁與促銷活動，並及時針對發現的潛在問題，做出相應的改進，從而為企業帶來更大的收益。

>> 立即下載JMP 16，即享30天免費試用 <<

JMP para optimización, fabricación y calidad de fármacos: Parte 2

JMP_is_fun — Wed, 10 Nov 2021 16:55:28 GMT

Este video es parte de una serie de tres presentaciones para la industria farmacéutica.

En la primera sesión, presentamos el diseño de experimentos, ajuste de un modelo estadístico, y optimización del proceso a partir del modelo.

En esta segunda sesión para manufactura de fármacos, exploramos y visualizamos los datos de fabricación, y luego usamos la minería de datos para el análisis de causa raíz.

Suponga que somos parte de una compañía de fármacos que esta sustentando una alta tasa de rechazo de lotes debido a problemas con la disolución de tabletas. El límite de especificación inferior es 70%, o sea que la tasa de disolución de las tabletas debe ser por lo menos 70, y lotes con una disolución menor a 70 se rechazan.

En vez de optimizar el proceso usando un diseño de experimentos, deseamos identificar las principales influencias en la disolución e identificar medidas correctivas a corto plazo para implementar inmediatamente para reducir la tasa de lotes rechazados.

Finalmente, vemos cómo usar una simulación Monte Carlo, para tomar en cuenta la variación en las variables de fabricación, incorporar el conocimiento del proceso, y obtener una idea más realista de como los cambios que implementamos afectarán al proceso de fabricación a largo plazo.

Tomorrow, and tomorrow, and tomorrow … finding significance and meaning in repeated measures data

HadleyMyers — Mon, 15 Nov 2021 14:33:41 GMT

Repeat anything enough times and it becomes meaningless. The repeated use of the word “tomorrow” in the first line foreshadows the conclusion of the famous soliloquy whereby Macbeth, upon learning of the death of his wife, searches the entirety of the temporal universe (from “all our yesterdays” right through to “the last syllable of recorded time”) for life’s significance and finds nothing. Years later, Andy Warhol echoed this theme by demonstrating that any image, however familiar or iconic, can too be made meaningless through endless repetition.

Many people struggle with the same phenomenon in their data. The effect of a treatment can be tested easily by taking a single set of measurements and analyzing the results using a one-way ANOVA to evaluate the significance of the effect between groups. The problem occurs when these measurements are repeated, complicating the analysis (think drug trials, where subjects are tested daily throughout the length of the trial, or the performance over time of products produced on different machines).

There is a great post on this topic in the JMP support notes titled "Analyzing Repeated Measures Data in JMP Software," in which three approaches are discussed. The purpose of this post is to demystify repeated measures using the univariate split-plot approach.

The example below (Figure 1) consists of three parts, each measured using one of two machines by four operators who recording the result in Y. In this case, there is no time variable at all; the repeated measures component comes from each part being measured a total of four times. Here, we’re trying to answer the question of whether Operator or Machine, or both, have a significant effect on the value of Y.

Figure 1: Table of repeated measures data; three parts made on two machines (6 parts total), each measured repeatedly by four operators.

We can use Standard Least Squares to answer these questions. The way to do that is to construct a model that includes Machine, Operator, and the interaction of Machine and Operator. We can also include Part in the model to ensure any variation due to differences in the parts are accounted for and not included in the other effects. Setting it as a random variable ensures the variation due to part is not treated as specific to these parts in particular but to all parts in general (see Figure 2). In this way, part defines the batches of a split-plot design, with Machine taking the role of the hard-to-change variable.

Figure 2: Fit Model dialog window.

However, this misses a crucial element of the story: The three parts measured using Machine A are not the same parts measured using Machine B. They are simply the ordering of the parts measured on each machine (the first three parts on A, and the first three parts on B). In order to make this clear, it is necessary to nest Part in Machine, as in Figure 3. Setting up the effect in this way ensures the software knows that “parts are different when machine is different.”

Figure 3: Fit Model dialog window, with the random variable Part nested in Machine.

The results from the Effect Summary (Figure 4) indicate that the measurements are significantly different for the different operators, as well as the machines. However, it doesn’t appear to matter which operator uses which machine, as the interaction effect of the two is above the criteria for significance (although we might have them test a few more parts to be sure).

Figure 4: The effects of Operator and Machine on Y are significant (PValue below 0.05), but the interaction between the two is not.

By analyzing your repeated measures data in JMP, you avoid the phenomenon described above and ensure your data is not meaningless.

A cringeworthy statistics statement: First in a series

JerryFish — Mon, 08 Nov 2021 22:02:54 GMT

We've all heard someone say something that isn't "right." Sometimes you just let it go, either because it isn't important enough to challenge, or you don't want to cause conflict with the speaker.

This is the first of a series of statistical statements that I consider "cringeworthy." I am initially posting these statements in the Community Discussion forum to get feedback from others. Happily, several of you (shoutouts to community members , , , , , and ) gave very thoughtful and insightful responses. Please continue the discussion! Find the second Cringeworthy Statistics discussion, which I posted today!

And read on for my take on why this first statement is cringeworthy.

Introduction

One of my "pet peeves" in the world of statistics has to do with the most basic of statements relating to a hypothesis test. Often, if a t-test (or ANOVA) ends up with a relatively high p-factor, the statistician/analyst will say, "We have p>0.05, so we can't prove the means of the two populations are different. Therefore, WE CONCLUDE THAT THE POPULATION MEANS ARE THE SAME."

The first half of that statement is correct. If p>0.05 (and we are looking for 95% confidence, and the samples come from normally distributed populations, and...), then we cannot conclude that the means of the two populations are different.

But that DOES NOT mean that the means of the two populations are THE SAME! I hear this statement all the time, even from people who should probably know better. It's just so so easy to slide right into this false conclusion. And it can be damaging, if the listener isn't familiar with statistics!

A simple example: Average height of men vs. women

As a simple example, let's say that we want to determine whether the heights of adult males is different from the heights of adult females. We have limited resources, so we can only measure the heights of three males and three females in this test. We randomly stop three males and three females on the street and measure their heights (shown below):

We use JMP's Fit Y by X platform to perform a t-test to compare the means of the two samples. The resulting p-value is 0.7096. Assuming we want 95% confidence in our conclusion, and since p>0.05, we conclude that we can't be 95% certain that the samples come from populations with different means. In other words, based on this sample we cannot say that men's average height is different from women's.

But this DOES NOT MEAN THAT MEN AND WOMEN HAVE THE SAME AVERAGE HEIGHT. (You can look it up... According to Medical News Today, in 2017 the average height of men was 67". And according to the Cleveland Clinic, in 2018 the average height of women was 64".)

Why did this happen? Why did the t-test not prove that men are (on average) taller than women? It is because of sampling. We can't measure all men and all women, so we chose to only sample three of each for this test. We happened to choose three men of about average height, but the three randomly chosen women happened to be a little taller than average. Combining that with the range of the measurements resulted in quite a bit of uncertainty about where the true average height of men and women might be, hence the high p-value.

Is this a "big deal?"

Yes, I believe this can indeed be a big deal! Let's take a more practical, industrial example. Let's say you have a machine that is filling vials with a powder. Your company wants to expand capacity, so you buy a second machine. You are asked to make sure that Machine B is filling to the same weight as Machine A.

You run a quick test, sampling several vials from Machine A and several from Machine B, and run a t-test. It comes back with a p-value of 0.35. Should you go to the boss and say that the machines are producing the same fill level? NO! If you do that, you risk making a mistake. If all vials coming from Machine B actually have a higher average fill weight than Machine A, then you are costing the company money by giving away "free" material. If B actually has a lower average fill weight than A, then you risk making your customers unhappy.

What can we do about it?

If you want more certainty in the averages, you would increase the sample sizes. This will also give a more accurate assessment of the comparison of men's and women's heights. But you are still faced with only being able to declare that the two populations are either proved different, or not proved different. You are not concluding that they are "the same."

You could also run an Equivalence Test (involving multiple t-tests). If interested, I would direct you here for more information. I'll leave it at that, as equivalence testing this is beyond the scope of this post.

Conclusion

So the takeaway is that while you can use a t-test (or ANOVA) to prove that something is DIFFERENT, you can't use it (by itself at least) to prove something is THE SAME. Please be careful with your terms and conclusions to avoid making mistakes in your applications!

Don't feel bad if you have made these statements in the past. In my experience, it is an extremely easy and common mistake to make. (Full disclosure: I have been guilty of making it myself!) So even though this has been said many times by many people before me, I think it bears repeating.

画期的なコロナ飲み薬、重症化を89%減少させるってどのような根拠から？

naohiro_masu — Mon, 08 Nov 2021 13:49:00 GMT

週末に、今のコロナ禍から脱却できそうな希望のニュースがありましたね。ファイザー社が開発中の新型コロナウィルスを治療する飲み薬の臨床試験（第2/3相試験の中間解析）で、重症化するリスクを89%減少させるという結果が報じられました。

この記事では、この飲み薬、パクスロビド(PAXLOVID)が、重症化リスクを89%減少させるとは、どのような結果を根拠に算出されたのかについてお伝えします。併せて、リスクの89％減少というのはあくまで限られた患者から得られた推定値なので、推定に対する信頼区間を算出して考察していきます。

ファイザー社のホームページに記載されているニュースによると、重症化リスクを持つ、入院していない大人の患者が対象で、ランダムにパクスロビドを投与したグループと、プラセボ(偽薬)を投与したグループに割付した結果、登録後28日以内に入院した患者の割合を比較しています。

パクスロビド投与：　389人中3人が入院（死亡者なし）

プラセボ投与： 385人中27人が入院（その後7名が死亡）

■入院リスクの比較

標題にあるリスク減少の数字を出すためには、相対リスクを算出することになります。そのために、JMPでは次のようにデータを作成しておきます。列「Group」には薬剤の投与グループを、列「Hopitalized」は、結果として入院したかどうか(Yes, No) を、列「N」には、「Group」と「Hospitalized」の各値に該当する人数を入力します。

この後、通常は「二変量の関係」(Fit Y by X)プラットフォームを使って求めますが、ここでは、同じことを「カテゴリカル」(Categorical)プラットフォームで行ってみます。

メニューバーから [分析] > [消費者調査] > [カテゴリカル] を選択し、次のように列を指定します。「N」を [度数] に指定するのがポイントです。

指定後、オプションである「相対リスク」(Relative Risk) を選択します。すると、対象となる水準と対象となる標本水準を聞いてきますので、それぞれ"Yes", "PAXLOVID" を選択します。つまり、パクスロビド投与グループの入院割合をプラセボ投与グループの入院割合に対して比較していることを指定しています。

レポートには、クロス集計表と各グループの割合を示すグラフ（シェアチャート）が表示されます。クロス表にあるように、各投与グループの入院割合を計算すると、パクスロビド投与グループは0.77% ( = 3 / 389) 、プラセボ投与グループは7.01% ( = 27 / 385) です。

レポート下側には、相対リスクのレポートが表示されます。今回の例において、”パクスロビド投与グループ” の ”プラセボ投与グループ”　に対する入院に関する相対リスクは次のように計算されます。

相対リスク = パクスロビド投与グループの入院割合 ÷ プラセボ投与グループの入院割合

　　　　　 = (3 / 389) ÷ ( 27 / 385 )

　　= 0.11

つまり、パクスロビドを投与すると、プラセボを投与することに対する相対的なリスクは0.11(11%)に減少することを示しています。これより入院のリスクがどれぐらい減少されるかで考えると 89% （100% － 11%）というニュースで報道された数字となるのです。

■相対リスクの信頼区間

前節で計算された相対リスクの値( = 0.11)は、あくまで試験で対象となった限られた患者での推定値です。そのため推定値だけでなく、その信頼区間も考えた方が良いでしょう。

再度、相対リスクのレポートを参照してみましょう。95%信頼限界が表示されています。

これより、相対リスク (= 0.11)に対する95%信頼区間は、およそ0.03～0.36 と案外広い区間であることがわかります。そのため、ラフな言い方をすると、入院のリスクの減少は、64%～97%と幅をもって考えることになります。64%と低めに考えてもかなり効果があるなあという実感ですが。

当然のことながら、サンプル数が多いほど推定に対する信ぴょう性は高くなるので、信頼区間は狭まります。仮に、今回の臨床試験の投与者数、入院者数ともに10倍多かったとします。すなわち、パクスロビド投与グループは3,890人中30人入院、プラセボ投与グループは3,850人中27人が入院となりますが、このときの相対リスクは、先ほどと同様に0.11ですが、95%信頼区間は0.08 ～ 0.16です。実際の例と比べて、信頼区間がかなり狭まっていますね。

今回の中間解析の結果は、十分有効性と安全性が示されたということで、今後承認申請が行われるようですが、問題なく承認され、実際のコロナ患者に利用できるようになれば、患者側としても医療機関側としても大きなメリットを享受できそうですね。その頃には、コロナ前のように、公の場でマスクをしなくて済む生活が待っているのかもしれません。

運用JMP淨化離群值，以半導體分析WAT資料為例

Michelle_Wu — Mon, 08 Nov 2021 13:48:40 GMT

當您分析資料時，第一個步驟是什麼？

是使用能力強大的互動性 JMP Graph builder (圖形生成器) 做視覺化分析？

或是利用 JMP的Distribution (分佈) 平台做敘述統計量分析？

又或是利用 JMP 多樣的預測建模工具，找到最適當的預測模型？

雖然這些都是為人所樂道的 JMP 功能，但是做為使用 JMP 分析的第一步，除了上述的平台外，您不妨考慮利用 Explore Outliers (探索離群值)及 Explore Missing Values (探索遺失值) 平台，清潔整理您的資料。

也許您認為探索遺失值還能理解，畢竟空值要先剃除對後續分析比較合理。但是，為什麼需要將離群值也排除？假如這樣的離群值正是代表資料母體的狀況，是否對我們的分析會有影響？而且，離群值很難抓取出來，不能先忽略這個影響嗎？

要釐清這些問題，我們需要先知道甚麼是離群值。

什麼是離群值？

我們先討論一維的狀況，隨機生成1000個標準常態分配Nor(0, 1)的值，如圖1，其中標記紅色的點，為超出[-4,4]範圍的離群點，這樣的發生機率為0.0063%，是非常小的機率，於是這這樣小機率區域上發生的點我們就視為離群值。

圖1 隨機生成的常態分配點圖

接著我們來看二變數的狀況，利用 Graph builder 分析，不論資料點是兩個變數之間具有相同均值及變異數(X2 vs. X1)或是具有不同均值及變異數(X3 vs. X1)，如下方圖二，我們都能用肉眼發現這些離群點(紅點)似乎跟其他的點有不一樣的趨勢，且與資料中心點的距離較遠。

圖2 (左) 2變數具有相同均值及變異數散點圖 (右) 2變數具有不同均值及變異數散點圖

而當兩變數間有強相關性，也能發現有些離群點距離其他的樣本點之間的距離較大 (圖3 紅點)。另外，利用Fit Y by X平台的histogram boarder功能觀察兩變數的分配 (圖3)，我們發現如果我們只關注單一變數，有些離群值我們會偵測不到，如下圖所示，這也是一個很好例子說明，當我們通盤考慮多變量的離群值，會抓到一些潛在的離群點，避免只觀察管控單一變量的誤判。幸運的，JMP提供了這樣的多變量離群點觀測平台。

圖3 兩變數間有強相關性的散點圖

到此，我們能夠了解離群值的概念，就是找出相較於其他的樣本點，距離樣本中心較遠，或是點的座落位置與樣本整體「趨勢」不符合的點。有趣的是，離群點不代表有問題的點，而是代表與其他的樣本有著差距而無相同的「趨勢」。所以離群值，有可能代表良善社會的問題人物，也可能代表萬惡城市中的一股清流，就看你的樣本來自哪裡。

善用JMP分析WAT資料，解決半導體製程問題

說到這裡，我們用一個半導體製造的電性測試資料作為說明。

半導體晶圓製造的元件參數都有其對應電子特性，因此透過電性測試這些參數的結果便可反推對應製程的品質。晶圓代工廠一般稱電性參數測試稱為WAT (Wafer Acceptance Test)，其客戶需要檢視WAT測試數據來確認製造品質。而新製程研發階段更需要蒐集大量WAT資料來解決製程問題或尋找最佳化的因子配置。

我們利用一組簡單的WAT資料作為示範，其中包含10個觀察參數資料，當我們直接分析產線三的資料，利用Analyze>Distribution 功能我們可以做出相應的直方圖及Outlier box plot 觀察是否有離群值，如圖四(左)，Outlier Box 上的紅色線段標記出最集中涵蓋50%的區域，從此處我們可以發現資料集中在中間的區段。同樣的你也可以利用Graph Builder 的直方圖功能完成直方圖的繪製，如圖四(右)。

圖四(左) Distribution 平台的直方圖；Outlier box plot (右) Graph builder的Box plot

根據Quantile Range 離群值檢驗我們發現參數一存在一個離群值(紅點)，利用JMP的Analyze>Screening>Explore Outliers 平台中的Quantile Range Outliers 功能可以發現其離群值(圖五)，並可以針對資料做標記、排除、視為遺失值等動作。

同時你也可以調整判定為離群值的規則，可以調整尾端百分位值(Tail Quantile)和其倍數(Q)，以常見的IQR離群值檢驗，上限為Q3+1.5*IQR，上限為Q1-1.5*IQR為例， Tail Quantile 即為0.25, Q則為1.5。而當您的資料較不符合常態分配時，可以考慮使用Robust Fit Outliers作為離群值篩選平台。

圖五 Explore Outliers 平台中的Quantile Range Outliers

但當我們更全面考慮，納入所有產線的資料，你會發現產線三擁有較高的參數一測試結果，如圖六，原本產線三上的離群值8.01反而較符合大多數資料的測試結果，而實際的離群值則轉為產線三最高的三個值 (圖六紅點)，這樣的結果也呼應前述樣本出處的重要性，廣泛獲取能夠代表母體資料對於篩選正確的離群值有舉足輕重的腳色。

圖六全部產線的Box plot

當考慮全部參數的影響，找出對於所有參數影響下的離群值時，JMP也提供Robust PCA Outliers 或是K Nearest Neighbor Outliers 功能供您使用。您可以在Analyze> Screening> Explore Outliers 下找到這兩個功能平台，這邊利用K Nearest Neighbor Outliers 平台找尋離群值，利用複合選取距離較高的樣本點，標記為離群值(圖七)。

圖七 K Nearest Neighbor Outliers 平台下的距離圖

藉由Graph builder 確認這些被選為離群值的資料點在每一個參數上的分佈位置可以發現不是所有的點都屬於單一參數的離群值(如資料點290, 295, 296, 297, 298, 299)，這樣的結果也和前述提及藉由JMP的多變量離群值篩選平台，可以抓出許多在觀測單變量下所忽略的潛在問題點(圖八)。

圖八用Boxplot 比較離群值分別在變數上的分佈

移除離群值，建構更準確的預測模型

我們知道了JMP有很好的平台可以幫助我們把離群值篩選出來，但是，沒有把離群值篩選掉會造成甚麼影響? 化繁為簡，我們先看一下兩參數間的離群值影響。

我們觀測WAT資料參數一級參數二之間的2次效應迴歸式估計的R-square值，在還沒移除篩選出的離群值前，R-square值為0.602，如圖九(a)，而移除離群值後，R-square值上升為0.740，如圖九(b)，這說明移除離群值能夠建構出更準確的模型，擁有差距更小的估計值。

當我們利用這些參數建構出預測結果，例如良率預估或是CP/FT 測試預估值等，我們能更有效偵測不良，降低成本，增加效益。

圖九(a) 沒移除離群值的二次迴歸估計 (b)移除離群值的二次迴歸估計

接著放大考量全部參數，並用這些參數預估後續Bin值結果，找到關鍵因子。並在最小化Bin值的設定下找到最佳因子設定值。相同的，我們先考量不移除離群值，找出最適合的迴歸模型，利用Fit model平台，我們得到迴歸模型R-square 大約為0.971，包含許多因子效應項，這其中可以看到許多的交互作用項效應比主效應強，似乎比較混亂，且觀察Residual by Row Plot 可以發現篩選為離群值的那幾個樣本差距是比較大的(圖十)。

圖十不移除離群值迴歸式R-square、Residual by Row Plot及影響的因子效應項

下一步把離群值移除，再做一次迴歸估計，我們發現，不僅R-square 上升到0.999，關鍵因子也縮減到四個因子效應項，觀察Residual by Row Plot，也沒有發現差異比較大的樣本(圖十一)。根據此迴歸公式預測的結果也會更準確，避免做出錯誤的判斷。

圖十一不移除離群值迴歸式R-square、Residual by Row Plot及影響的因子效應項

蒐集多一點資料做後續測試迴歸式的準確度，我們可以在Analyze>Fit Y by X的平台下，比較迴歸式的預估值及實際的Bin 值，新資料依舊能夠有很好的水準。R-square有0.999的準確度，而且沒有特殊的點有過大的差異。

圖十二比較迴歸式的預估值及實際的Bin 值

藉由JMP的Explore Outliers平台，我們能夠輕易快速的排除離群值，避免受到離群值影響誤判因子之間相關性或是做出錯誤的模型預測，導致錯誤的預估及決策。作為分析的前哨站，JMP的Explore Outliers 平台能夠針對不同情景，不論單變數或是多變數，資料是否為常態，都能提出適當的平台幫助使用者找到離群值，快速達到資料清理的目的。

衍伸閱讀：

如何在30秒內在JMP中製作晶圓圖(英)

使用功能數據分析進行圖像和晶圓分析

報名11/26半導體研討會

>> 透過 JMP如何讓使用者快速有效的得到最佳化的參數組合，進而提昇產線良率及增加產出？報名11/26半導體研討會，瞭解如何善用 JMP 進行WAT分析，改善製程良率及挖掘關鍵因子。

>> 立即報名

JMP para optimización, fabricación y calidad de fármacos: Parte 1

JMP_is_fun — Fri, 05 Nov 2021 17:25:19 GMT

Este video es parte de una serie de tres presentaciones para la industria farmacéutica. El software estadístico JMP es extremamente útil en casi todos aspectos de la industria de fármacos, desde el desarrollo, ensayos clínicos y la producción.

El video a continuación introduce el tema y le ofrece una vista de alto nivel sobre como JMP se utiliza en la industria farmacéutica.

En esta primera sesión, presentamos diseño de experimentos, modelación de un proceso, tan como la optimización del proceso a partir de un modelo estadístico.

Imagínense que somos parte de una empresa de biotecnología que desea maximizar el rendimiento de un proceso de extracción. El objetivo es obtener un rendimiento de 45 mg/cl.

Para investigar y optimizar el proceso de extracción, usamos un diseño de experimentos personalizado con seis factores:

Cuatro disolventes: metanol, etanol, propanol, butanol
Dos variables de proceso: pH, tiempo en disolvente

Este diseño también tiene una restricción lineal: La suma de los cuatro disolventes debe ser menor a 10 ml/cl.

El siguiente video muestra los pasos detallados para optimizar el proceso de extracción usando JMP.

En el próximo video, veremos cómo JMP se usa en la manufactura de fármacos - la exploración visual de datos de fabricación, y como usar la minería de datos y machine learning para el análisis de causa raíz.

Celebrating Dyslexia Awareness Month — and neurodiversity in general

anne_milley — Fri, 29 Oct 2021 15:32:29 GMT

In observance of Dyslexia Awareness Month, I feel compelled to call some attention to one of the most common learning differences that interferes with fluent reading. I care a great deal about this topic because my daughter was tested for and found to be dyslexic when she was 15. Even though we know dyslexia is one of the most common neurobehavioral learning differences in children, the data on its prevalence worldwide is poor.

This is surprising, especially in light of the fact that a tremendous amount of research has taken place over the last several decades in many different disciplines. Why is it so difficult for us to identify the estimated 15-20 percent of people who have dyslexia when we live in a society that values literacy? It’s complicated on many levels.

Dyslexic people can become fluent readers, but their brains will always work differently. Many organizations have stopped categorizing dyslexia as a “disability,” preferring to refer to it as a learning “difference.” This distinction is important as the stigma of being labeled “disabled” often adds to the toll dyslexia can take on self-esteem. (Of course, to get academic support, the way much of educational policy is implemented requires use of the word “disability,” but it doesn’t have to be discussed that way.)

Previously, I wrote a blog post on diversity as the state of variation within a population and the importance of understanding sources of variation so we can better understand our world and infer useful things from the data we collect. This previous post showed some natural sources of variation in gender/sex that many of us have never seen. In a similar vein, I would love to better understand the apparently natural biological variation in neurodiversity, which includes, but is not limited to, the following:

Developmental Coordination Disorder (DCD or dyspraxia)
Dyslexia
Attention Deficit Hyperactivity Disorder (ADHD)
Dyscalculia
Autistic Spectrum (ASD)
Tourette Syndrome (TS)

Unfortunately, the data collection for most of these conditions is largely unavailable since some is understandably kept private and research data is often not readily available. As a result, the data can vary widely in how the conditions are measured. Looking at multiple sources, the graph below illustrates estimates of some of these conditions:

To dig into some of the research findings about dyslexia, language-based differences appear to matter. There is growing evidence that overcoming dyslexia is more challenging in languages of orthographic depth or irregular orthography like English. Examples of orthographically shallow languages are those where the spelling-sound correspondence is direct — by knowing consistent rules of pronunciation, pronouncing the words is straightforward. Orthographically shallow languages include Spanish, Hindi, Latin and others. Orthographically deep languages are those where the rules of pronunciation are inconsistent, for example, "tough" and "through" with the “gh” representing both “f” and silent; "knot" and "not" with both “kn” and “n” pronounced as “n.”

Here is an oft-cited chart on orthographic depth:

From: Orthographic Depth and developmental dyslexia: a meta-analytic study

Much has been written about successful people who have not only overcome challenges that their dyslexia posed, but also capitalized on it, which they see as a gift (seeing the world differently is a gift for many people with neurodivergent conditions). In general, these individuals have had emotional support, advocates, luck, or some combination of these to achieve success despite many obstacles.

Most people know that Einstein was thought to be dyslexic; Richard Branson, Charles Schwaab, Keira Knightley and many others are also success stories. People like American actor and author Max Brooks (who made a compelling presentation as part of The Science of Dyslexia Full Committee US Congressional Hearing in 2014), amazing microprocessor engineer Jim Keller, award-winning Australian artist, Vincent Fantauzzo, astrophysicist and visual learning advocate Matthew Schneps, and many others have also made amazing contributions to society. Many of these folks attribute their successes to their dyslexia.

Another statistic I find striking: While as many as 15 to 20 percent of the population are thought to have dyslexia, estimates of prison populations with dyslexia are thought to be much higher. A 2000 study of Texas prisoners found that 48 percent were dyslexic, and two-thirds struggled with reading comprehension.

If we better understood the gifts that neurodiverse conditions bring, wouldn’t we do a better job of trying to measure these conditions so we could remove obstacles and help those with these gifts make the most of them? We all stand to benefit. Nature gives us diversity in many forms, even if we can’t readily see them — I marvel at them and hope we can see and appreciate more of them.

Piecewise Nonlinear Solutions Part 3: Using JMP's Formula Editor to solve for unknown parameters

JerryFish — Thu, 28 Oct 2021 16:31:39 GMT

This is the third of a series of posts on how to fit piecewise continuous functions to data sets. I'll begin each post with links to all of the other posts in the series. They are:

Part 1: Description of the problem, introduction of example data
Part 2: Choosing functions that we want to fit, and setting boundary conditions between piecewise segments
Part 3: Using formula parameters
Part 4: Choosing convergence criteria and algorithms, and running the Nonlinear platform to get parameters to converge
Part 5: Using "canned" routines in the Curve Fit platform

Onto part 3...

Introduction

If you have followed along with my previous posts in this series, you know that we have now developed equations containing 10 unknown parameters that need to be determined in order to best fit the pieces of the example problem. How do we do this in JMP?

First, we need to use JMP's Formula Editor to input our equations and unknown parameters. I will assume that the reader is generally familiar with the Formula Editor. (If not, here is a short video on Creating Formulas and a longer video with much more detail on the Formula Editor.)

Setting up Parameters in JMP's Formula Editor

What you may not be aware of is the ability in the Formula Editor to set up Parameters. Parameters are variables that are local to the formula that you are creating. They are accessed through the dropdown menus (default is Table Variables, switch to Parameters as shown in Figure 1.)

Figure 1: Accessing Parameters in the Formula Editor

This brings up an option to add a "New Parameter" to the formula. Let's do a simple example and add a parameter called "a0" to the parameter list. Click on New Parameter, then change the parameter name to "a0", and give it a Value of 1:

Figure 2: Creating a new parameter and giving it an initial value

"Name" is arbitrary. You can choose any valid JSL variable name.
"Value" is the starting value for this parameter. You must give a starting value. Sometimes (depending on the formula that you have chosen), the starting value can be important, and should be selected to be close to the real value as you can guess. (This helps to prevent convergence problems later on.) In this case, I have no idea what the value should be, so I'll just arbitrarily choose a starting value of 1.

Now we can use this new "a0" parameter in a formula, just as if it were a column variable. Let's simply add it to the values in the column labelled "y". Double-click on a0 in the Parameter list (it should appear in the Formula panel,) then type "+", and then double-click on "y" in the column list. This should give the following formula:

Figure 3: Using a parameter in a formula

Now when we click OK and look at the result in the data table, we have

Figure 4: Results of using a parameter in a formula

So the formula now recognizes the current value of the a0 parameter and applies it accordingly. This important concept will be used in the next blog post when we use the JMP Nonlinear Platform to solve for the unknown parameters.

Entering Parameters for the Piecewise Functions

In Part 2 of this blog series, we developed the five simple polynomials to apply between breakpoints in the example data series. These polynomials had six unknown parameters: a0, b1, b2, c1, d1, and d2. We also had four cutpoints defining the regions where the polynomials were active: X1, X2, X3, and X4. We need to add each of these parameters to our Formula parameter list, along with reasonable starting values. When finished, the parameter list looks like this:

Figure 5: All ten parameters entered and initialized for our piecewise example

Note that I assumed starting values of "1" for each of the coefficients in the polynomials. For the breakpoints X1 through X4, I chose to estimate values based on "eyeballing" where I thought they might occur. This simply gives JMP a better place to start looking for the breakpoints.

Entering the Piecewise Formulas - including a Boolean Trick!

Now you know how to set up parameters in the Formula Editor, and you know how to enter any of the five Piecewise equations shown in Part 2 into the Formula Editor as well. But how do we enter ALL FIVE Piecewise equations into the same formula?

One way is to use a large IF statement, e.g. "If x<X1 then (Piecewise Equation 1), else If x<X2 then (Piecewise Equation 2), etc.) This is relatively straightforward. But here I need to give a shoutout to my friend , who posted a helpful trick in a previous JMP Community Discussion Thread.

JMP recognizes Boolean operators (And, Or, Nor, etc.). If a Boolean operator is used in a formula, the Boolean logic returns a 1 if true, and a 0 if false. So we can set up an operator such as (X1<:x & :x<=X2). This operator returns a 1 if the value in the "x" column (referred to as ":x" in the expression) is between the parameters X1 and X2, and 0 otherwise. So we could simply multiply the piecewise equation for the X1-X2 segment by the above Boolean expression. If the value of x is in the proper range, then the segment equation is activated. If not, we return 0. We would then do the same thing for all other ranges, and sum all equations together.

If we do all of that, we end up with a formula that looks like this:

Figure 6: Complete piecewise formula (shown in Formula Editor mode)

And below is the same information in JSL input format, should you prefer:

Figure 7: Complete piecewise formula (shown in JSL mode)

So to summarize, in Figures 6 and 7, we essentially have built a formula in a new column (called ypred) that:

Defines 10 unknown Parameters (and given each parameter an appropriate starting value) which will be used in the formula.
Describes four piecewise equations, each using their appropriate parameters.
Activates each piecewise equation by multiplying by an appropriate Boolean inequality that describes the ranges for each equation.
Sums the four equations to give the final formula.

Pretty cool...

Next in the series

Next time, we'll finally use the equations with the Nonlinear platform to solve for the best fit parameters. We will also discuss choices for convergence criteria and algorithms.

Exploring 19th century data with a 21st century lens: The Broad Street pump

O_Lippincott — Fri, 05 Nov 2021 15:43:51 GMT

In 1854, a public health crisis was occurring in Soho, London, with a cholera outbreak in the neighborhood around Broad Street. By talking to residents, English physician John Snow suspected that the contamination source was the now infamous Broad Street water pump, so he had the pump handle removed. Later, he quantified the outbreak using a map that visualized the source of the contamination, which confirmed that the Broad Street water pump was the problem. By solving that public health mystery, Snow is now celebrated as one of the founders of modern day epidemiology. Given the same data points today, how would the exploration of the data look using a 21^st century lens?

John Snow's Original Map, drawn and lithographed by Charles Cheffins.

Source:https://en.wikipedia.org/wiki/John_Snow

Present day location on Broadwick Street.

Source: https://en.wikipedia.org/wiki/John_Snow

The data points from Snow’s original map include the location and number of cholera cases at each residence, as well as the location of the nearby water pumps. Broad Street is now named Broadwick Street in present-day London. Plotting the longitude and latitude of the points in JMP’s Graph Builder, we get a visualization of the residences reporting cases, as well as the water pump locations.

In different iterations by Snow of the original map, he denoted the density of the cholera cases, both as bars and as dots on the map. At first glance, by having the size of the marker denote the number of cases (as shown in the map below), we start to see locations of interest, but it is difficult to determine relative size of the circle markers to denote density of the cases.

The data set we are working with contains the specific addresses of the affected households, making it easy to extract the street name. Upon closer examination, we see that there are four streets that have greater than 5% of the total cases.

Using the dynamic linking in JMP by selecting the four bars in the chart, the selected streets are highlighted back in our map. Notice the location of the water pump on Broadwick Street, formerly Broad Street. A clearer picture emerges.

Today, we have additional tools that can show the density of cases more effectively than points alone could. To show the patterns in density, nonparametric density can be added to the map using JMP’s Fit Y by X platform. The quantile contour lines are added at 5% intervals. The density contours clearly show the hot spot of the cholera cases, with the Broad Street water pump right in the center.

Modern visualization techniques that reveal density allow this 19^th century data to be explored in ways that were not available to Snow. However, the overarching goal for graphing this data set is the same today as it was in Snow’s day: concisely communicate that the Broad Street water pump marked the spot of the outbreak’s epicenter.

References

Images: https://en.wikipedia.org/wiki/John_Snow

Data Source: http://blog.rtwilson.com/updated-snow-gis-data/

Statistically Speaking: Answers to audience questions on design of experiments

HadleyMyers — Mon, 25 Oct 2021 15:31:01 GMT

JMP held a Statistically Speaking seminar on design of experiments (DOE) earlier this month. A very special thanks goes to Moderna’s Julia O’Neill, Roche’s Andrea Geistanger, and CPI’s Rachel Findlay for engaging in a compelling discussion with moderator Malcolm Moore. Attendees also heard two keynote presentations.The first was by Dame Sally Davies, Executive Chair of The Trinity Challenge, who described the challenges faced by scientists in the current data landscape and what can be achieved when analytics is embraced. Dr. O’Neill delivered the second keynote, in which she explained how Moderna utilized DOE and analytics to successfully solve a real-world problem, namely, how to develop a vaccine against the novel coronavirus, SARS-CoV-2.

The event attracted a lot of interest, judging by the number of questions the panelists received. While they did a great job addressing many of them, it was not possible to answer all in the time allowed. We’ve, therefore, decided to answer them in this post.

Q: Would you please relate sample size with possibilities of having a reliable innovation?

A: We can do better than that. The new Sample Size Explorers, introduced in JMP 16 and JMP Pro 16, were created to encourage exploration of the relationship between power, sample size, and other key metrics involved in determining how best to ensure your experiments are a success, as explained in this blog post by Caleb King (). We recommend anyone interested in allocating resources or budgets between development projects, planning timelines, or simply understanding sample size calculations to check out this new feature.

Q: Do you have recommendations on resources to tackle the most challenging part of DOE: choosing factors and defining ranges?

A: This is a topic of much interest and for good reason: Choosing factors that affect your response and ranges large enough to detect an effect are the critical first steps in planning and executing a successful DOE. At this point, it would also be helpful to define success. A DOE is successful if it results in a model that encompasses enough understanding of the process to satisfy a business need: “How can we set our factors to achieve this level of quality in our output?” for example.

A DOE is also successful if it identifies significant factors, quantifies the variation not captured by the model, and points the way towards a follow-up experiment (an augmented DOE). An experiment that fails to identify a single significant factor would constitute a failure, although the learning would be that the wrong factors were chosen, or the ranges weren’t set wide enough. Any good resource on DOE includes some discussion on how to choose factors and ranges, although it is often not much more than “choose factors that are likely to affect your response, and ranges that are wide enough to see an effect if one exists, but not so wide as to cause the runs to fail.” This is because the choice of factors and ranges is entirely dependent on the specifics of the situation in question. Therefore, the best people to determine these are the process experts, rather than statisticians with great understanding of DOE but limited understanding of the application. Having said all this, take a look at this discussion on the JMP User Community, which includes much useful insight on the matter.

Q: How do you secure the reliability of "historical" data or data that are coming from "external sources"? Someone once said "never analyze data you haven't collected yourself." How do you overcome this in order to be sure your historical data are reliable, meaningful, the measuring system used is capable to assess the process variations to a minimum, etc.?

A: I don’t know who said that about data you haven’t collected yourself, but I don’t agree with it. In her keynote, Dame Sally Davies discussed the multitude to ways data is being generated and collected, as well as the risks for analysts “working in silos” using different data sets that are “not informing each other.” In her words, there is currently “greater engagement between scientists and data than ever before,” and limiting our analysis to data we collect ourselves would severely impede progress.

Of course, we should always be skeptical of data, just as we are skeptical of the models generated from the data. But treating something with skepticism is not equivalent to dismissing it outright. Nothing is perfect, and this certainly true for data and models, as George Box succinctly expressed in his famous aphorism. It does not follow that because something isn’t perfect or can’t be trusted that nothing can be learned from it. We are right to ask questions about the source or accuracy of the data. Any conclusions drawn from its analysis should be considered alongside our own process knowledge and subject matter expertise, and never accepted blindly.

Q: Would you be willing to elaborate on the common mistakes that new DOE users should be aware of?

A: One of the mistakes people make is over-focusing on the design rather than why they’re doing a DOE in the first place. It happens far too often that someone will begin by first choosing a design, and then compromising their experimental needs to fit that design. The Custom Design platform in JMP lets you create optimal designs tailored to any situation. Additionally, CPI’s Rachel Findlay gave some great advice during the event: “Share your learning and your mistakes” with colleagues. The more we promote DOE within our organization, the more the organization will benefit.

Q: What is the most important factor to keep in mind when selecting a DOE?

A: There are many important factors in selecting the right DOE, beginning with the business question we are trying to answer. What exactly are we aiming to accomplish by executing a DOE in the first place? Do we want to minimize defects or increase yields? Maybe we want to incorporate material from a new supplier into our process, figure out how to speed production or decrease costs. DOE is often an iterative process of incremental learning, and understanding our target objective is critical when determining whether to experiment further. We also need to ask how many variables might be affecting our process, as well as our timeline and budget. Once all of this is clear, we can begin the process of choosing a design.

For example, if we already know that a small number of variables are significant and we’re looking for the best combination of settings to maximize output, we may choose to opt for a Response Surface Experiment. Alternatively, a large number of factors and a tight budget may point the way towards a Definitive Screening Design. Well-defined answers about our situation, our constraints, limitations, and what we are trying to solve will point the way towards a design, or series of designs, that ensure our requirements are satisfied. It should also be mentioned that the Evaluate Design and Compare Design platforms are useful tools to help make the best possible decision.

Q: DOE can be a hugely powerful technique… when used correctly…. but I have equally seen too many examples of heavily overfit models which are not predictive, and this often isn’t obvious to novice statisticians/biologists. How do we better help novices to understanding the tools they are using and what the outputs actually mean?

A: There are plenty of resources to help new experimenters get up to speed with design of experiments very quickly. They also cover the various ways of removing insignificant terms from a model and using validation to avoid overfitting. Ultimately, the best proof of a model is verification with independent data: Is the proposed prediction as the solution determined from our model supported by verification runs? An excellent place to start is the Statistical Thinking of Industrial Problem Solving online course, which includes a module on DOE. There also several textbooks available here. Those new to both JMP and DOE can go through the New Users Welcome Kit and Design of Experiments Intro Kit. The Mastering JMP webinars include DOE as well. Finally, the JMP User Community is a powerful resource where anyone with a specific question can post it as a discussion topic; questions are usually answered by one of the 30k community members within a few hours.

Q: How can artificial intelligence complement DOE?

A: The topic of artificial intelligence (AI) was discussed in a previous Statistically Speaking event. To say that it is a very promising field does not quite do it justice. It has already delivered on many of its promises. However, the greatest risks arising from AI relate to our over-reliance on it to solve all problems, or to blindly accept the solutions it purports to present. The adage that “correlation does not mean causation” highlights AI’s shortcomings. To separate the two and get to the heart of understanding of any process, DOE is needed. Of course, DOE is not able to drive a working process autonomously. I would say there are opportunities for both AI and DOE to complement each other.

To learn more about how DOE can speed innovation, please sign up for these free events in November:

秩和檢定與兩兩比較的思考脈絡與分析方法

Michelle_Wu — Fri, 15 Oct 2021 13:19:13 GMT

在之前的文章中，我們介紹了兩組和多組常態分佈資料的組間比較方法。在本文中，我們將要講解的則是用於檢定非常態分佈資料差異的統計方法，也就是「秩和檢定」。本文將重點介紹以下：獨立樣本秩和檢定、單樣本秩和檢定和配對樣本秩和檢定。

秩和檢定的基本概念

秩和檢定，屬於非參數檢定，非參數檢定不考慮總體的參數和總體的分佈類型，而是對樣本所代表的總體的分佈和分佈位置進行假設檢定。

秩和檢定的基本概念，首先進行編秩，然後用秩次代替原始資料來進行檢定。基於秩次檢定各組的平均秩是否相等，如果經檢定得各組的平均秩不相等，則可以推論資料的分佈不同。

本文主要用到兩個資料進行講解，一個是BMI的資料（圖1），另一個是IgA的資料（圖2）。

圖1 BMI數據

圖2 IgA數據

獨立樣本秩和檢定

獨立樣本的秩和檢定主要有兩種方法：

用於兩組比較的Wilcoxon秩和檢定
用於多組比較的Kruskal-Wallis秩和檢定

詳見 <一個神奇的JMP功能表，實現資料的所有組間比較>。二者的關係有點類似於t檢定和變異數分析的關係。

這裡要注意的是，兩組比較可以用Wilcoxon秩和檢定，也可以用Kruskal-Wallis秩和檢定；多組比較只能用Kruskal-Wallis秩和檢定。兩組和多組獨立樣本秩和檢定的操作一致，為了避免重複介紹，本文僅以多組樣本的秩和檢定為例進行介紹。

在 (圖1) 資料中，假設我們要比較不同心功能分級患者的BMI是否存在差異，因此江心功能分級分成四級，分別用1-4表示，BMI為非常態分佈，因此考慮Kruskal-Wallis秩和檢定，分析過程如下：

首先通過點選JMP「分析」→「以X擬合Y」（如下圖3），進入組間差異比較的介面。

圖3 獨立樣本秩和檢定操作 -- 功能表選擇

本例中BMI為結果，心功能分級為分組，因此在對話方塊中將BMI放入「Y，回應」，將「心功能分級」放入「X，因數」（圖4）。

圖4 獨立樣本秩和檢定操作——變數選擇

進入分析結果的介面後，點擊「心功能分級-BMI」單因數分析上方的紅色三角形按鈕，在下拉式功能表中選擇「非參數→Wilcoxon檢定」（圖5）。

圖5 獨立樣本秩和檢定操作 -- 方法選擇

輸出結果見圖6. 結果主要有兩部分：

第一部分是資料描述，給出每組的例數、秩和、平均秩等信息。從得分均值的結果可知，心功能分級為4的患者BMI最高。
第二部分為統計檢定結果，給出了Kruskal-Wallis秩和檢定的結果。表明不同心功能分級患者的BMI差異有統計學意義（卡方=10.4306，P=0.0152）。

貼心提醒：這裡給出的統計量是卡方，是因為Kruskal-Wallis秩和檢定的結果服從卡方分佈，並非做了卡方檢定，大家不要搞混了。

圖6 獨立樣本秩和檢定操作——檢定結果

注意事項：如果進行兩組樣本的秩和檢定，會同時給出Wilcoxon秩和核對總和Kruskal-Wallis秩和檢定兩個結果，兩種檢定的結果一致。Kruskal-Wallis秩和檢定結果顯示有統計學差異，通常我們還對具體哪兩組有差異感興趣，這就需要進行組間兩兩比較。

接著，我們點擊「心功能分級-BMI」單因數分析旁邊的紅色三角形按鈕，在下拉式功能表中選擇「非參數」→「非參數多重比較」→「對所有對執行Steel-Dwass檢定」，操作見圖7。

圖7 獨立樣本秩和檢定操作——兩兩比較操作

輸出結果見圖8，兩兩比較結果表明，心功能分級中的水準2和4（Z=3.089，P=0.0108）；水準3和4（Z=3.057，P=0.0120）的差異有統計學意義。

圖8 獨立樣本秩和檢定操作——兩兩比較檢定結果

本例分析結果表明，不同心功能分級患者的BMI的差異有統計學意義。兩兩比較結果顯示，心功能分級水準2和4；3和4的差異有統計學意義。

單樣本秩和檢定

基於圖1資料，探索患者的BMI與健康人群的正常值是否存在差異。假定BMI的正常值為19，BMI為非常態分佈，則進行單樣本秩和檢定。

在過往文章中，我們已經為大家講解了常態分佈資料如何進行單樣本t檢定，不過，在現實情況下資料不一定皆呈現常態分佈，當我們拿到非常態分佈的資料，應如何進行分析呢？

單樣本秩和檢定操作的前四步與單樣本t檢定相同：

①選擇JMP功能表「分析」→「分佈」；

②在彈出的對話方塊中，將BMI放入「Y，列」；

③在結果介面中點擊BMI左側的紅色三角形按鈕，在下拉式功能表中選擇「檢定均值」；

④在彈出的對話方塊中的指定假設均值中填寫19。

上述步驟詳見《一文學會t檢定的3種常用方法》。

完成以上步驟後，點擊確定則輸出單樣本t檢定結果。

若想輸出單樣本秩和檢定結果，則繼續在對話方塊中勾選「Wilcoxon符號秩」（圖9）。

圖9 單樣本秩和檢定操作——方法選擇

結果如圖10所示，結果不僅包含t檢定的結果，還包含秩和檢定的結果（「符號秩」部分的結果）。結果顯示差異有統計學意義（檢定統計量=3114.500，P<0.0001）。

圖10 單樣本秩和檢定操作——檢定結果

配對樣本的秩和檢定

本例採用圖2資料，分析患者化療前後IgA是否有差異。在既往的文章中我們已經為大家講解了配對t檢定的操作步驟。

配對秩和檢定的前兩步操作與配對t檢定相同，操作流程為：

①選擇JMP功能表「分析」→「專業建模」→「配對」；

②在彈出的對話方塊中的「Y，配對回應」中先放化療前IgA，再放化療後IgA。

上述步驟詳見《一文學會t檢定的3種常用方法》。

完成上述步驟後，在結果介面中點擊「配對」旁邊的紅色三角形按鈕，在下拉式功能表中選擇「Wilcoxon符號秩」，即可獲得配對秩和檢定的結果（圖11）。

圖11 配對秩和檢定操作——方法選擇

結果如圖12所示，患者化療前後IgA差異有統計學意義（S=-2523.0，p<0.0001）。

圖12 配對秩和檢定操作——檢定結果

結論

通常情況下，組間比較的資料呈明顯偏態，我們會考慮採用秩和檢定，而不是t檢定或變異數分析(ANOVA)。很多人可能對秩和檢定存在一定誤解，總覺得秩和檢定像是「備胎」，實際上秩和檢定的效率並不低。

如果資料為常態分佈，秩和檢定的檢定效率比t檢定、方差分析等差不了多少；如果資料呈現偏態分佈，秩和檢定的效率則遠高於t核對總和方差分析。所以建議大家在實際中一定要考慮資料的分佈情況，進而選擇合理有效的分析方法。

原文：干货 | 秩和检验及其两两比较的思路与解析

推薦閱讀：

變異數分析 (ANOVA) 與兩兩比較的思考脈絡與分析方法

在JMP中進行常態檢定與變異數同質性檢定

實驗設計 (DOE)入門：經典篩選設計與全因子設計

評估SMT供應商品質有困難？使用JMP讓分析變簡單

Michelle_Wu — Fri, 15 Oct 2021 13:18:46 GMT

現代化產業無論是OEM專業代工，或是ODM原廠委託設計，甚至是委外代工都會有一個共通點，那這個共通點是什麼？首先工廠生產產品，最重視的是產品的良率，也就是落在規格內的百分比，但是如果要評估產品的好壞，良率並不是唯一的指標，最常評估的就是製程能力優劣，如CPK>1.33等，但是熟知如Google、Intel、Apple等大公司，為什麼可以讓產品品質穩定，這就牽涉到他們評估使用的工具及指標，以下將介紹一個簡單的SMT案例，並使用 JMP 進行分析。

在 SMT (Surface–mount technology 表面黏著技術) 製程產線中，其中一站點膠製程，則需要評估兩間不同的廠商設備品質，選擇合適的機台。在分析前這裡先讓大家簡單了解點膠的製程。

點膠製程，晶片與基板熱膨脹係數不同，溫度變化造成熱應力集中於兩者間的凸塊造成破裂形成斷路 (如圖1)，當基板受外力產生變形時或環境氧化也會造成凸塊變脆變質，所以就必須點膠增加晶片與基板間的接合強度及隔絕空氣並使應力重新分布(圖2)，當然點膠製程的因子很多，這次只單純評估機台點膠後的重量一致性以及x和y方向的偏移程度。

左圖一；右圖二

機台通常提供的技術規格如下圖3所示(非此例的標準)

圖3

01：敘述性統計分析比較

分析時，我們先從點膠重量著手，資料收集有公司(M，W)以及天數資料(下圖4)，一開始建議先從敘述性統計下手，先了解其分布，並且將規格輸入，以便比較。

圖4

從(下圖5)可以看到兩家公司都不是非常對稱的分配，使用 JMP 中的 Distribution 和 Graph Builder 平台可以輕鬆比較。

圖5

接著搭配常態檢定(如下圖6)，確定為非常態分配，所以接著可以針對製程能力去做比較，

圖6

M公司製程能力是高於W公司，但是都有大於1.33的水準，(圖7)

圖7

02：假設檢定

比較完敘述性統計後，接著可以使用假設檢定去了解兩家公司是否有顯著差異，由檢定結果得知兩者間的變異不相等 (因為兩者都非常態分配使用Levene方法去做判斷)，接著由 Wetch Test 了解兩者均值間有顯著差異，所以在重量的部分就可以很明顯看出 M 公司是佔有優勢的，如圖8。

圖8

03：ANOVA分析

接下來就是針對天數的部份去了解不同天是否會造成兩間公司的重量會有顯著差異，根據收集的資料進行不同天的ANOVA分析，如下圖9、圖10，其中M公司兩天在變異部分沒有顯著差異，而均值部分有顯著差異，假設偏差不影響品質，一般公司會設定另外一個Practical difference當作水準，視實際是否會造成品質影響。而W公司很明顯兩天的水準有顯著差異，即使均值沒顯著差異，一般來說，調整變異會比調整偏差來的更加困難，

圖9 -- M公司的ANOVA分析

圖10 -- W公司的ANOVA分析

04：管制圖

除了統計檢定外，也可以搭配管制圖的功能(圖11)，搭配Local Data Filter篩選及比對資料，圖形中可以看出兩間設備商第二天的管制界線較窄，但是W公司很明顯有接近規格的點出現，如果真的因為價格因素考量，就必須調查原因為何。

圖11

接著就xy的偏移程度作確認，一樣照著上面的流程去確認，從製程能力(圖12)確認還是M公司較佳，而兩公司進行ANOVA分析(圖13)無論是變異或是均值兩家設備商有顯著差異。

圖12

圖13

接著細分成兩天去比較(圖14)，這裡僅列出有差異的部分，W公司無論是x或y方向，天數均會造成差異，而M公司均無影響，

圖14

最後搭配管制圖確認(圖15)，確認W公司的水準，M公司無顯著差異就不放上圖，所以無論是點交的重量或是xy的偏移確認都是M公司較佳。

圖15

結論

最後從客戶端的角度，光是一個製程，就有許多的品質參數要確認，此外，各個參數也都有不同的方法需要分析，以上的功能雖然散落在不同的平台，但是可以搭配JSL客製化視窗或是Dashboard將所有的分析結果集中在一起並標準化，呼應一開始提到的共通點就是「品質以及分析的時效性」，現實生活中還有很多不同產業的應用，歡迎報名JMP線上研討會，你將會聽到更多實用的分析場景，幫助你解決日常工作中的分析難題。

實驗設計 (DOE)入門：經典篩選設計與全因子設計

Michelle_Wu — Fri, 08 Oct 2021 17:20:37 GMT

實驗設計 (DOE) 常應用在醫療、生技製藥及高科技製造等產業，做為降低實驗次數及成本的一種方法，在進行實驗設計時有哪些重要原則與步驟？傳統實驗方法中的經典篩選設計與全因子設計的概念與應用有哪些？本篇文章將介紹DOE基本原則、經典篩選設計與全因子設計方法的概念與應用案例。

什麼是實驗設計(DOE)

實驗設計 (DOE) 是透過篩選實驗條件來設計實驗，並且降低外在因子以減少對實驗的影響，特色在於可以透過較少的實驗次數、較少的實驗成本與時間，以有效率的方式收集資料或是找到因子之間的相關性與問題等，其應用範圍廣泛。

實驗設計的三大推手

1920 年代，RonaldFisher 將實驗設計的觀念引進農業，現在不僅是農業上會用到實驗設計 DOE，舉凡醫療、生技製藥、高科技製造等產業皆會應用到實驗設計方法。隨後 1950 年代田口玄一發表田口法、引進直交表 (Orthogonal arrays)，並於 1970 年代早期發表質量損失函數的概念；1951 年英國統計學家George EP. Box 和 K. B. Wilson 引進反應曲面法 (Response Surface methodology)，1960 年 George EP. Box 接續與 Hunter, J. Stuart 發表兩水準部分因子設計，讓實驗與研究人員能快速找到影響因子，並帶入了解析度的觀念。

傳統實驗方法：試誤法、一次一因子和傳統設計

試誤法：以經驗法則為基礎，僅限於簡單或變數較少的製程
一次一因子：其他因子固定，一次實驗測試一個因子
傳統設計
- Stage1 部分因子設計
  - 目的是篩選&識別重要因子，通常存在因子數過多或是對製程一無所知的情況下使用
- Stage2 全因子設計
  - 目的是考慮是否有主效應其他的效應存在，影響估計準確性；估計主效應及交互作用效應；理解系統特性
- Stage3 反應曲面
  - 優化(最佳化)：針對顯著的引子找出最佳設定值
  - 估計主效應及交互作用效應及非線性效應
  - 著重在預測的準度與精度上

實驗設計指導原則

定義問題以及目標
辨別反應變數
*重要! 確認量測系統 (MSA) – 過多誤差會對因子效果不顯著
辨別因子數及其水準數
辨別是否有限制式或其他設限條件
建立設計
實驗前準備
執行實驗
分析資料
做出結論並決定下一步

進行實驗設計目的

篩選 --> 探索 --> 最佳化 --> 驗證 --> 穩健

篩選：找出影響反應變數最顯著的因子，通常存在因子數過多或是對製程一無所知的情況下
探索：找出是否有新的因子或是新的水準
最佳化：針對顯著的因子找出最佳設定值對應到期望的反應變數
驗證：確認系統或是製程是否與預期的行為相符合，如使用不同批原料去驗證結果獲相同機型、但是不同機台去驗證參數設定是否可以直接套用
穩健：最小化反應變數變異，包含控制條件以及因子設置

使用 JMP 進行實驗六步驟

Step1. Describe 定義反應變數(responses)、因子和水準數 (Factors and levels)
Step2. Specify 決定模型包含哪些效應項
Step3. Design 決定實驗次數是否加入中心點或重複次數，並提供相關的實驗評估項目
Step4. Collect 根據實驗條件及順序收集數據
Step5. Fit Model 擬合模型，找出關鍵因子
Step6. Predict 找出適合的模型，並根據目標進行最佳化設定

經典篩選設計VS全因子設計

(1) 經典篩選設計 Screening Design

偵測主效應是否顯著、或稱線性效應，中心點雖然可以偵測是否有曲面效應，但無法確切得知是哪個因子
在部分因子設計中，需要處理交絡 (混淆, confounding) 作用，通常解析度 (resolution) 預設要四以上
可能因為交絡作用無法辨別是別名項或是效應項顯著，需要額外擴增實驗去區分

經典篩選設計與主效應設計的使用時機

案例分享：部分因子與主效應設計比較

(2) 全因子設計 Full Factorial Design

考量各因子的水準，生成每種組合
水準越多或是因子數越多，效率越低及成本越高
優點：可以估計所有交互作用項，並且比一次一因子及試誤法考慮更全面
缺點：太多實驗次數，成本過高

案例分享：電池壽命實驗

上千位工程師共同參與【DOE入門課】

以上是 DOE 實驗設計中的經典篩選設計與全因子設計的簡單介紹，

若想完整瞭解經典篩選設計的案例應用，歡迎收看 JMP 專業顧問為您準備的 DOE 入門課程。

點擊以下連結，即可免費觀看。

Piecewise Nonlinear Solutions Part 2: Choosing functions that we want to fit, and setting boundary conditions between piecewise segments

JerryFish — Mon, 01 Nov 2021 19:32:53 GMT

This is the second of a series of posts on how to fit piecewise continuous functions to data sets. I'll begin each post with links to all of the other posts in the series. They are:

Part 1: Description of the problem, introduction of example data
Part 2: Choosing functions that we want to fit, and setting boundary conditions between piecewise segments
Part 3: Using formula parameters
Part 4: Choosing convergence criteria and algorithms, and running the Nonlinear platform to get parameters to converge
Part 5: Using "canned" routines in the Curve Fit platform

This post, by necessity, is a little "math-y."

Introduction

In Part 1 of this blog series, I introduced an example problem to demonstrate something called "fitting piecewise functions" to a dataset. These data were broken into regions where we would like to fit (as yet TBD) functions. Even the X locations that define the boundaries of the regions are as yet unknown. Figure 1 shows the data and the suggested regions.

Figure 1

Choosing Functions

For each region, we wish to fit different functions that serve to model the data. In JMP's Nonlinear platform, you can use any expression that you like (e.g. exponentials, logs, trigonometrics, etc.) In general, selection of the functions will be entirely up to the wisdom, insight, and subject matter expertise of the modeler. The following discussion uses simple polynomials, which offer the benefit of simplifying math when it comes to establishing boundary conditions. These have worked well for me over the years.

By simple polynomial, I mean an equation of the form:

Equation 1

where:

y = the measured response

n = the segment of interest

Am = the "mth" unknown coefficient for the nth segment

x = the independent predictor

The polynomial can be truncated to any desired length, depending on the complexity of the function to be fit.

Let's look at each segment of Figure 1 individually.

Segment 1: Pre-start-of-test data

As described earlier, this region might consist of data that are collected before the test actually starts. In this case, it might ideally have a y value of zero, though there could be some instrumentation offset. We expect this to be a constant value, so we would set the polynomial for the first segment as:

Equation 2

Segment 2: Fixture Tolerance Take-up

Recall that the example data set might represent data from a tensile test machine, where a specimen is placed in the tester using a mounting figure. The second segment/region shown in Figure 1 (between X1 and X2) might indicate a region where fixture tolerances are being taken up. We might be interested primarily in how much take-up there is (X2-X1), or in how much force is required to take out all of the tolerance.

Since there is curvature in this region, I chose to use a quadratic function for the fit. I also chose to change the parameters to b's instead of a's, just to help distinguish the coefficients in this segment from those in Segment 1:

Equation 3

Segment 3: Linear/Elastic Region

Segment 3 might be the most important region for the test, since our goal is to understand the elastic region of the curve. Here the relationship between x and y is linear, perhaps representing the elastic region for our example test specimen in the tensile fixture. We would be most interested in the slope of this region, so I chose a simple slope-intercept form for this equation:

Equation 4

Segment 4: Yield Initiates

Segment 4 might cover the area in our example test data where our tensile specimen begins to yield. Visual inspection of the data shows that there is curvature again (similar to Segment 2), so again I chose a quadratic fit:

Equation 5

Segment 5: Data Following End of Test

Segment 5 might occur after the actual tensile test has finished. The test specimen is still under tension. The amount of stretch (y) is unchanging, so the equation would be similar to Segment 1:

Equation 6

Setting Boundary Conditions

We now have five equations with unknown coefficients (a0, b0, b1, b2, c0, c1, d0, d1, d2, and e0) along with the four breakpoints X1, X2, X3, and X4. Each of these needs to be determined. But there is more information that needs to be provided. Specifically, each of the curves needs to "connect" at the transition boundaries.

Segment 1 to Segment 2 transition

For the Segment 1/2 transition, we want y1=y2 at x=X1. This means:

Equation 7

Solving for b0, we get:

Equation 8

This "connects" the function of Segment 1 with the function of Segment 2 at x=X1. This gives us a way to eliminate b0 from any upcoming equations.

Remaining segment boundaries

Without spelling out all of the algebra, we can do the same thing with each of the other boundaries (e.g. y2(X2) = y3(X2), etc.). Solving for c0, d0, and e0, we arrive at:

Equation 9

Equation 10

Equation 11

After making multiple substitutions into the segment equations, we arrive at:

Equation 12

Equation 13

Equation 14

Equation 15

Equation 16

So now we have whittled down the unknowns to a0, b1, b2, c1, d1, d2, X1, X2, X3, and X4. How do we get JMP to solve for these unknowns? Join me for the next blog post!

使用JMP進行表格視覺化與個性化報表 -- Tabulate 的進階設定

Michelle_Wu — Thu, 07 Oct 2021 15:06:21 GMT

透過〈使用JMP快速生成統計報表 - Tabulate的基本應用〉這篇文章，你可能已經大致清楚如何生成符合預期格式的表格，但在實際應用上，表格還有一些細節問題需要調整，例如如何儲存表格、設定表格的小數位等。本篇文章將繼續介紹 Tabulate 模組中的更多功能，包含如何根據研究需求優化表格、進行表格的設置；如何將表格圖形化、讓資料變得很簡單好懂；以及如何存儲表格。

在本篇文章範例中，我們依然使用以下的範例資料 (圖1)，和生成的結果表格 (圖2) 進行講解。

圖1 本文所用資料

圖2 資料生成的表格形式

功能一：在Tabulate 設定小數點格式

從 (圖2) 中可以看到，生成的表格中的結果保留多位元小數，但我們在發表文章時，通常只需要保留 2 - 3 位就夠了，也就是說，我們需要統一設置小數點。小數點的設定，可以透過左下角的更改格式介面修改(圖3)。

圖3 更改表格格式選項

點擊 (圖3) 的更改格式後，會出現 (圖4) 所示的介面。

圖4 更改小數位數

在 (圖4) 的介面中勾選使用相同的小數格式，出現 (圖5) 的介面。

圖5 資料展示格式選項

這裡有 3 個選項，除了「固定小數位數」外，還有「最佳」和「百分比」兩個選項，「固定小數位數」顧名思義就是，我們自己指定小數位數；「最佳」則是由軟體根據指標實際情況顯示最佳的顯示位元數；「百分比」則是以百分數的形式來展示。

本篇文章裡，我們想讓這幾個指標都顯示 2 位元小數，因此選擇固定小數點，小數點填寫 2，這樣可以將所有統計量均修改為保留 2 位小數，結果見 (圖6)。

圖6 保留2位元小數的表格

功能二：改變預設統計量選項

(圖3) 中除了「更改格式」外，大家可以看到，在它的上面是一個「預設統計量」，該選項很實用。可能大家還記得上一篇文章提到，連續變數 (如BMI) 拖到表格中時候，預設的都是求總和。不過，在多數的實際應用中，你可能想要的是平均值和標準差。在這種情況下，就可以通過「默認統計量」進行修改。點擊該按鈕後，會出現 (圖7) 所示畫面：

圖7 預設統計量介面

在尚未調整前，默認的是 N 和總和，分別對應分類變數和連續變數的初始統計量，現在我們勾選成平均值和標準差 (圖7)，調整後再將連續變數拖入表格，便能直接顯示均值和標準差，是不是更方便了呢？

功能三：顯示缺失比例與合計

在很多研究中，可能收集的變數會有缺失。

預設情況下，JMP 表格是不顯示缺失的。但有時你在發表文章時，可能審稿人會讓你同時把缺失的比例也列到表格中，這在 JMP 中其實很簡單，只要勾選一下 (圖3) 中的「包含分組列的缺失值」即可，這時候就會顯示缺失的例數。

此外，還有一種情況是：對於分組變數，除了顯示每一組的統計量（均值、標準差等）外，有時還想同時顯示不分組的統計量做為合計，如 (圖6) 中顯示了吸煙和不吸煙人群的年齡等 3 個變數，如果還想顯示所有人的年齡等變數的均值，這就可以通過勾選 (圖3) 中的「添加聚合統計量」來實現。當你勾選了這一選項，就會發現分組變數後面多了一列，顯示的是不分組的所有人的統計量。

功能四：將表格圖形化

JMP 的基本特色，在於能以圖形直觀展示統計或研究結果，在 Tabulate 這項功能中也不例外。

在 Tabulate 功能表列底下，點選紅色三角形勾選「顯示圖表」，讓你的表格透過圖形的方式呈現，更直觀地展示數值與統計量大小 (圖8) 。

圖8 顯示圖表選項

結果如 (圖9) 所示。

圖9 以圖形式展示表格統計量

大家可以看到，以橫條圖展示了各自數值的大小。通過圖形呈現，我們可以更直觀感受到數值間的差異。

功能五：匯出你的統計表格

功能一至四主要是講解如何調整 Tabulate 細節，那麼做好表格後，該如何儲存表格呢？

(1) 打開「文件」→ 選擇「另存為」→ 選擇「rtf」格式

可以直接用 word 打開，而且打開後直接給出的就是表格格式，非常方便快捷，如 (圖10)。

圖10

(2) 點擊 Tabulate 左側的「紅色三角形」→ 點擊「製成資料表」

便可以將統計結果直接生成 JMP 資料表，並保存成 Excel、Word 等格式，如 (圖11)。

圖11

以上就是本期關於 Tabulate 功能的進一步探索，只要善於探索和挖掘，你就會發現使用 JMP 就像搭積木一樣簡單而又充滿樂趣。下載 JMP16 進一步動手操作看看。

原文：手把手教你在JMP中快速实现报表的个性化与可视化

推薦閱讀：

在JMP中進行常態檢定與變異數同質性檢定

變異數分析 (ANOVA) 與兩兩比較的思考脈絡與分析方法

一文學會3種常用的t檢定 - 獨立樣本t檢定、單樣本t檢定和配對t檢定

Some things are better online

Diana_Levey — Fri, 01 Oct 2021 15:13:16 GMT

Join Discovery Summit from wherever you are.We miss being with scientists who use our software to make the world safer, engineers who ask the tough questions and keep us on our toes, people who love JMP as much as we do.

We miss Discovery Summit in-person interactions as much as you do.

That’s why we’ve created an online event that includes the things you know and love about Discovery Summit: asking questions about contributed papers and posters, meeting with JMP experts to get the inside scoop on your favorite features, listening in on the technical conversations because that’s how you learn brand new things.

Plus, there’s nothing to stop you from attending Discovery Summit Americas online. Papers, posters, keynotes, discussions, networking and more will take place Oct. 4-7. There's no fee to attend.

Fitting piecewise functions with JMP's Nonlinear platform

JerryFish — Mon, 01 Nov 2021 19:33:12 GMT

This is the first of a series of blog posts on how to fit piecewise continuous functions to data sets. I'll begin each post with links to all of the other accompanying posts. These series covers:

Part 1: Description of the problem and introduction of example data
Part 2: Choosing functions that we want to fit and setting boundary conditions between piecewise segments
Part 3: Using formula parameters
Part 4: Choosing convergence criteria and algorithms and running the Nonlinear platform to get parameters to converge
Part 5: Using "canned" routines in the Curve Fit platform

So let's get started!

Introduction

Recently I was asked if JMP could do segmented analyses, i.e., where several different types of curves were parametrically fit to different regions of a data set. This involves not only finding the proper parameters for each of the fitted segments but also adjusting the break point locations along the X axis. I can envision ways in to do this in JSL, but I'd like to use the Nonlinear point-and-click feature in JMP rather than writing and debugging code. How can we do that?

A test problem

The figure below shows some sample data.

Figure 1: Sample Data Set

This type of data might result from any of several types of data. Your data might look like this, or it might look entirely different. The point is that we might desire to build a model out of several different polynomials that are each valid for a given region of the data.

For the purposes of this blog series, let's assume that these data come from a tensile testing machine. In that case, X might be a tensile force applied to a test specimen, and Y is a strain (or deflection, or stretch) of the test specimen. We might be interested in several features of this data, such as:

The initial measured Y (i.e. deflection) that the instrument reads. This might represent a measurement offset that needs to be corrected.
The slope of the linear portion of the curve (representing the specimen's Elastic Modulus, for example.)
The amount of force required to begin yielding the test specimen (i.e., where the straight line begins to bend, signifying the yield strength of the part.)
The distance that the specimen stretches over its elastic range.

For this example, we might decide to divide the data into five zones, shown in Figure 2 below:

Figure 2

How would you go about doing this? Join me next week for an approach using JMP’s Nonlinear solver platform!

變異數分析 (ANOVA) 與兩兩比較的思考脈絡與分析方法

Michelle_Wu — Tue, 28 Sep 2021 13:51:18 GMT

在之前的文章中，我們介紹了組間比較的基本操作，並在上期文章中詳細介紹了t 檢定在JMP中的實現。 t 檢定是用於檢驗兩組均值差異的統計方法，本篇文章將帶您詳細瞭解什麼是變異數分析(ANOVA)、使用變異數分析前須考量哪些條件、以及如何使用JMP進行變異數分析(ANOVA)與兩兩比較。

在本文中，我們以圖1的資料為例進行講解。

圖1 範例資料

什麼是變異數分析(ANOVA)?

變異數分析，是把全部觀察值的總變異分解成組間變異和誤差變異，然後將組間變異與隨機誤差進行比較，從而判斷總體均數間的差別是否具有統計學意義。

使用變異數分析須滿足3大條件

變異數分析是 t 檢定更一般性的推廣，t 檢定可以看做是變異數分析的特例。

因此使用變異數分析的前提條件與 t 檢定一致：
①各個樣本是相互獨立的；
②各組資料均為常態分佈；
③各組間的變異數相等。

意思是，進行變異數分析前，我們需要進行常態檢定和變異數同質性檢定 (Homogeneity of variance test)，由於變異數分析只能得出「組間是否有差異」的結論，具體哪幾組之間有差異，仍需要進一步統計分析，這時就需要用到兩兩比較方法，常見的兩兩比較方法有 Bonferroni 法、Tukey’ HSD 法和 Dunnett 法。

Bonferroni法

Bonferroni 法，是在進行兩兩比較時調整檢驗水準。通常組間比較以 0.05 作為檢驗水準，但在兩兩比較時，每次比較就會有 5% 的概率發生 I 類錯誤。

使用 Bonferroni 法的思路，是通過將 0.05 除以要比較的次數，降低檢驗水準，從而減少假陽性錯誤。如 4 組兩兩比較共需比較 6 次，則兩兩比較的檢驗水準需調整為 0.05/6=0.0083，即認為 p<0.0083 才算有統計學差異。但是該方法在比較次數較多時不太適合使用，因為校正後的檢驗水準會過小。

Tukey'HSD法

Tukey 法，是常見的兩兩比較方法，該方法曾經只能用於各組例數相等的情形，後來提出了改進的 Tukey 法，可用於各組例數不等的情形。 JMP 提供的就是改進的 Tukey 法，該方法可作為兩兩比較的首選方法。

Dunnett法

Dunnett t 檢定，專門用於比較 1 個對照組和多個試驗組間的差異，試驗組之間不做比較。

使用JMP進行變異數分析(ANOVA)

在 (圖1) 資料中，若比較不同心功能分級患者的軀體健康評分是否存在差異，心功能分級分為一到四級，因此這是一個四組之間的比較，不能直接用 t 檢定，而應考慮多組比較的方法。首先通過點選JMP菜單「分析→以X擬合Y」（如圖2），進入組間差異比較的界面。

圖2 變異數分析操作——菜單選擇

本例中軀體健康評分為結果，心功能分級為分組，因此在對話框中將軀體健康評分放入「Y，響應」，將心功能分級放入「X，因子」（圖3）。

圖3 變異數分析操作——變量選擇

進入結果畫面後，我們需要結合常態檢定和變異數同質性檢定 (Homogeneity of variance test) 的結果，選擇合適的統計方法，分析結果如下：

> 常態檢定結果顯示各組資料均為常態分佈。

> 變異數同質性檢定 (Homogeneity of variance test)結果見 (圖4)。

多組資料的變異數同質性檢定 (Homogeneity of variance test)多用 Bartlett 檢定和 Levene檢定。至於兩種檢定的使用情況略有不同，Bartlett 檢定主要用於常態分佈的資料，Levene檢定多用於資料不滿足常態分佈的情形。

閱讀文章〈在JMP中進行常態檢定與變異數同質性檢定〉瞭解如何進行該檢驗。

在這個範例，資料為常態分佈，因此我們採用Bartlett 檢定，分析結果顯示為 P=0.0224，為方差不齊。

圖4 變異數同質性檢定 (Homogeneity of variance test)結果

當資料為常態分佈但不滿足變異數同質性檢定 (Homogeneity of variance test)時，採用Welch變異數分析，方法選擇可參考文章〈實現資料的所有組間比較的強大功能 -- 以X擬合Y〉。

Welch 變異數分析結果見變異數同質性檢定 (Homogeneity of variance test) 結果的最後一部分（圖5）。結果顯示四組間軀體健康評分的差異有統計學意義（F=40.2951，P<0.0001）。

圖5 Welch變異數分析輸出結果

如果資料滿足常態且方差齊，則可直接採用變異數分析，儘管從條件來看，本例資料應該用 Welch 檢定，但做為範例，我們同時也介紹一下變異數分析的結果如何輸出。點擊「心功能分級-軀體健康評分」單因子分析旁邊的紅色三角形按鈕，在下拉菜單中選擇「均值/變異數分析」，如圖6。

圖6 變異數分析操作——方法選擇

輸出結果見 (圖7)，變異數分析結果表明四組的軀體健康評分差異有統計學意義 (F=16.0080，P<0.0001)。

圖7 變異數分析輸出結果
從上述分析結果可以看出，變異數分析的 F 值與 Welch 檢定結果有一定的差異。因此對於連續變量的組間比較一定要綜合考慮其常態性與變異數同值性。

JMP中的兩兩比較

如果總的變異數分析結果，顯示無統計學差異，提示各組間均無統計學差異，就不要再做兩兩比較；不過如果總的組間比較結果顯示四組的差異有統計學意義，那麼通常還需要進行組間兩兩比較，以明確具體是哪兩組之間有差異。

JMP中常態資料的兩兩比較比較操作在「比較均值」的選項中完成操作，由於我們要比較任意兩組之間的差異，可選擇 Tukey 法，操作見圖8。

圖8 兩兩比較操作

點擊「心功能分級-軀體健康評分」單因子分析旁邊的紅色三角形按鈕，在下拉菜單中選擇「比較均值→所有對，Tukey HSD」。

輸出結果見 (圖9)，結果顯示除了心功能分級3和4間無差異，其它組之間都有統計學差異。本例分析結果表明，不同心功能分級人群的軀體健康評分差異有統計學意義 (F=16.0080，P<0.0001)，除了心功能分級3和4間無差異，其它心功能分級之間的差異都有統計學意義。

圖9 兩兩比較輸出結果圖

以上就是本期我們為大家帶來的實用分享，立即下载 JMP 試試以上的操作吧！

原文：干货 | 方差分析及两两比较的思路与实现

推薦閱讀：

一文學會3種常用的t檢定 - 獨立樣本t檢定、單樣本t檢定和配對t檢定

使用JMP快速生成統計報表 - Tabulate的基本應用

Michelle_Wu — Tue, 28 Sep 2021 13:50:59 GMT

很多朋友都曾面臨這樣的問題：統計軟體做出一堆結果，要逐一複製貼上到文章中，或是一個個手動抄寫下來，這樣複製黏貼或抄寫的過程中，不僅容易出錯，還有可能讓人質疑資料的合理性。那有沒有既節省時間、又不用擔心弄錯的好方法，能在 JMP 直接輸出分析表格呢？當然有！今天這篇文章我們將通過實例說明，透過簡單幾個步驟，便能在 JMP 中快速生成交叉分析的統計報表。

Tabulate的基本應用

假設我們有下面的資料 (圖1，僅顯示部分)：

圖 1 - 示範資料

JMP的製表功能位於「分析Analyze」菜單中，選擇「制表Tabulate」，如下圖 2：

圖 2 - JMP 的製表菜單

進入 Tabulate 的界面後，你會發現 JMP 的一貫特色 -- 互動性，也就是說，不是在菜單中選定你想要輸出的統計描述指標，然後結果中輸出固定的表格；而是根據你的需求拖拉你想要顯示的指標，這樣的設計可以方便使用者更加自由快速地調整輸出的內容。

圖 3 - JMP tabulate 操作介面

從圖 3 可以看出，製表功能支持包括數目、均值、標準差、最小值、最大值等多種常用統計量的輸出，基本上可以滿足研究論文的所有需求。

01 - 顯示分類變量的例數

JMP中表格製作的方式，延續了JMP一貫的拖移功能。例如，我們想看一下吸菸和不吸菸人群各自的例數，只需簡單將變量「吸煙」拖入右上角「欄 (column)」的拖放區即可 (也可拖入左下角「列 (Rows) 的托放區」，取決於是豎列顯示還是橫行顯示)。

02 - 顯示連續變量的均值、標準差

對於吸菸這種分類指標，只要拖入，默認顯示兩組人數。如果想顯示吸菸和不吸菸兩組人群的年齡情況，只需將「年齡」拖入行的拖放區即可，如圖 5。

圖 5 - 拖入第二個變量示意圖

把「年齡」拖入相應位置後，結果如圖 6 所示，JMP 會默認給出「總和」的結果 (連續變量都是默認顯示其總和)，即吸菸和不吸菸兩組人年齡的總和。

圖 6

如果想進一步顯示其它統計量，如平均值和標準差。直接在左側的列表中選擇「標準差」，將其拖入右側表中相應位置即可。這裡最關鍵的是注意拖拉時的游標位置，如果「標準差」拖到「總和」下方 (圖7所示位置)，此時顯示一條藍色短線，意思是在「總和」的下方增加一行「標準差」指標。

圖7 - 增加不同統計指標示意圖

如果想要將「總和」替換為

均值”，可將“均值”拖到「總和」的位置 (圖8所示位置)。此時該位置顯示為一個藍色方框，意思是替換，用「平均值」取代「總和」。

圖 8 - 替換統計指標示意圖

這樣我們就獲得了吸菸與年齡之間的平均值和標準差資料，如圖9。

圖 9 - 統計指標顯示結果

03 - 顯示多個變量的均值、標準差

除了年齡，假設我們還想顯示身體健康評分的均值和標準差，此時無需重複一遍對”變量的操作，只需將「軀體健康評分」拖至「年齡」下方即可 (圖10)，此時在年齡變量下方位置出現一條藍色粗線 (注意與圖7比較，圖7是在統計指標下方，這裡是在變量下方)，根據「年齡」已有的統計指標 (平均值和標準差)，「軀體健康評分」顯示相同的內容。

圖 10 - 增加新變量示意圖

結果如下圖 11 所示，這樣我們就獲得了以是否吸菸為分組的，年齡和軀體健康評分的均值和標準差。

圖 11 - 增加新變量的結果

04 - 不同變量顯示不同的統計描述指標

從前面圖 10 可以看出，只要設置好 1 個變量的統計指標，其它變量如果想顯示同樣內容，只要把它們拖到已有的變量下方即可。然而實際中，這樣的拖拉方式如圖 12 所示，拖到整個表格下方，相當於另起爐灶，重新加一個新表格。

圖 12 - 不同變量顯示不同指標的拖拽示意圖

由於是新加的表格，而且 BMI 是連續變量，因此仍然默認顯示「總和」(圖13)。如果要換成「中位數」，此時操作與前面介紹相同，將中位數覆蓋「總和」即可。拖拉方式見圖 8。

圖 13 - 新加入不同變量不同指標的結果

使用JMP快速找出因子之間的關係

透過前面的介紹，可以知道其實這種操作方式非常簡單，唯一需要注意的就是拖拉時藍色線條的位置及其形狀。為了讓大家更好地熟悉這幾種方式，圖 14 進行動態展示。

圖 14

可能有些人會說，該有的指標是有了，但是看起來並不是最終想要的，小數點太多了，此還該如何進行數值上的調整呢？下篇文章我們就介紹一下如何針對表格格式進行調整，以及如何保存成你直接能用的格式，立即下載 JMP 試用，跟著文章練起來吧！

原文：如何在JMP中快速生成統計報表？

【推薦閱讀】資料分析必學10技巧：

We're name dropping so you don't miss out on Discovery Summit Americas

Diana_Levey — Thu, 16 Sep 2021 14:51:24 GMT

At the risk of being pretentious, we’re going to drop some names. It’s for a good cause – so you don’t miss out on the statistical discovery event of the year, Discovery Summit Americas. Because once you know who will be there, we think you’ll want to be there, too. (The conference is Oct. 4-7, and if you're already ready to register, here is where to go.)

Let’s start with several members of the event Steering Committee – the folks who help us pick the best papers and posters.

At Lundbeck, Patricia McNeill leads a team in designing and evaluating fermentation and cell culture experiments for the production of therapeutic monoclonal antibodies.
Cameron Willden supports engineers and scientists across many different product lines at W.L. Gore, focusing on manufacturing and new product development.
Supriya Satwah uses her expertise with advanced analytics to deliver critical results for business decision making for the Modeling & Analytics team in Digital R&D at Unilever.
Sarah Gilyard is an in-house consultant at Micron Technology. She provides statistical support to engineers and other semiconductor professionals.
At Aera Energy, Matt Kedzierski mentors, teaches and leads engineers and technicians who perform data analytics and Six Sigma improvement projects.
Trish Roth, a molecular immunologist turned data scientist, manages a statistical process control program for diagnostic product manufacturing at Abbott Laboratories.

Those are simply some of the brilliant people who pick the contributed content. Ready to register now?

But what about all of the interactive papers and posters that make the conference so rich in content? There’s too much information to include here. So we’ll just drop a dozen of the world-class organizations represented at the 2021 Discovery Summit Americas:

Abbott Molecular
Applied Materials
AstraZeneca
Dow Chemical
Johns Hopkins University
Microsoft
Procter & Gamble
Samsung
Sandia National Labs
Syngenta
Thermo Fisher Scientific
US Army

Satisfied? You will be when you hear real-world stories about overcoming obstacles with statistical data exploration. Register now to attend our all-online Discovery Summit Americas.

P.S. It's free!

在JMP中進行常態檢定與變異數同質性檢定

Michelle_Wu — Tue, 14 Sep 2021 13:29:33 GMT

在上一篇文章《一個神奇的 JMP 功能表，實現資料的所有組間比較》，我們介紹了組間比較常用的方法以及 JMP 操作的基本特色，本篇文章開始，我們將分別對幾種常見的組間比較方法及其操作過程一一介紹。今天的文章將從連續變數的組間比較說起。

對於連續變數的組間比較，如果要選擇合理的分析方法，有幾個前提條件必須先考慮，即常態性和變異數同質性 (Homogeneity of variance)。

什麼是常態性和變異數同質性？

一般常見的資料分佈形態，有常態分佈、泊松分佈、指數分佈等，常態檢定指的是判斷總體是否符合常態分佈的檢定。變異數同質性檢定，則是對各組樣本間的變異數是否相同進行的檢定。

在連續變數組間比較的實際應用中，很多人常犯的錯誤就是不考慮資料是否符合常態或是變異數，而是直接使用 t 檢定或變異數分析 (ANOVA) 對資料進行統計分析，而正確的分析流程是，在選擇方法前，先看一下你的資料分佈到底是什麼樣子。

本文的內容就是介紹在選擇統計方法之前需要考慮的兩個重要條件：如何判斷資料是否滿足常態分佈、如何判斷組間的變異數是否相等。只有確定了這兩個條件，才能選擇合理的分析方法。

在JMP中進行常態檢定

接下來，我們來看一下如何在 JMP 中使用「分佈」平臺實現常態檢定，以 (圖1) 的資料為例，探索吸煙與不吸煙人群的 BMI 值是否存在差異，首先需對 BMI 值進行常態檢定，然後根據資料分佈情況判斷應該使用 t 檢定還是秩和檢定。

圖1- 本文所用的數據

需要注意的是，在進行組間比較時，需分組進行常態檢定，即分別檢定吸煙與不吸煙兩組人群BMI的分佈，而不是檢定所有人 BMI 數值的分佈情況。接下來為大家講解如何進行具體操作，首先選擇「分析」→「分佈」(如圖2)。

圖2 - 常態檢定操作示意圖第一步

因為我們要探索吸煙和不吸煙的兩組人群的 BMI 分佈情況，因此 BMI 為結果，吸煙為分組。所以在對話方塊中將 BMI 放入「Y，列」；將吸煙放入「依據」(圖3)。輸出結果見圖 4。

圖3 - 常態檢定操作示意圖第二步

圖4 - 輸出結果

JMP中經常會看到「依據」，它表示「分別」或「分層」的意思，例如將吸煙拖入依據，表示分別對吸煙人群和不吸煙人群執行操作。這與「分組」意思不同，分組表示組間比較，如比較吸煙人群和不吸煙人群的差異。在後續的文章中，我們還會對這一概念進行解釋。

如果對這種豎向放置的長條圖不習慣，想將長條圖橫過來，可在結果介面點擊左上角的三角形按鈕，在下拉式功能表中選擇「堆疊」(圖5)。

圖5 - 堆疊操作示意圖

輸出結果如圖 6 所示。這裡分別給出吸煙與不吸煙兩組人群的 BMI 的各類統計量，包括平均值、標準差、中位數、第一&第三四分位數等。

圖 6 - 輸出結果

點擊「BMI」左側的紅色三角形按鈕，在下拉式功能表中點擊「連續擬合」→「擬合常態」(圖7)。

圖 7 - 常態檢定操作示意圖第三步

常態擬合的結果見圖 8。

圖8 輸出結果

點擊左側的紅色三角形按鈕，在下拉式功能表中點擊「擬合優度」(圖 9)，便會顯示常態檢定結果。

圖 9 - 常態檢定操作示意圖第四步

常態檢定結果如(圖10) 所示，p 值等於0.9128，大於 0.05，不能認為不滿足常態分佈，即無法得出「資料不服從常態分佈」這一結論，因此可將資料視為常態分佈進行分析。

圖10 - 常態檢定結果

不吸煙人群 BMI 的常態檢定操作方式同上，結果見 (圖11)，為非常態分佈。

圖11 - 常態檢定結果

根據上面的常態檢定結果，我們應使用秩和檢定進行組間比較，因為在兩組資料，中只要有任意一組資料為非常態分佈，就必須使用「秩和檢定」進行組間比較。

在JMP中實現變異數同質性檢定

如果在進行組間比較時，發現兩組資料均為常態分佈，並不意味著就可以直接進行 t 檢定，還需要進行變異數同質性檢定，來確定接下來應該使用 t 檢定還是校正的 t 檢定（t’ 檢定）。

什麼是變異數同質性檢定？

變異數同質性檢定，是對組間方差是否相同進行的檢定。為什麼要做變異數同質性檢定？因為在進行組間比較時，如果組間方差差別太大，將會掩蓋掉均值的差異，導致錯誤的結論。

變異數同質性檢定常用的方法有：

F 雙邊檢定、Brown-Forsythe 檢定、Levene 檢定、Bartlett 核對總和 O’Brien 檢定。其中 F 雙邊檢定用於兩組資料的變異數同質性檢定；其它用於多組資料的變異數同質性檢定，實際中以 Levene 法和 Bartlett 法較為常用，但 Bartlett 法只能用於常態資料，Levene 法還可用於非常態資料，應用範圍更廣一些。

以 (圖1) 資料為例，若分析不同呼吸困難程度的人軀體健康評分是否有差異，則首先進行常態檢定，若檢定結果發現兩組資料均為常態分佈，接下來則進行變異數同質性檢定。

變異數同質性檢定作為組間比較的重要步驟，其操作在 JMP 功能表「以 X 擬合 Y」模組中實現。在前一篇文章中我們已經對組間比較的操作進行了簡單介紹。

選擇「分析」→「以 X 擬合 Y」。本例中軀體健康評分為結果，呼吸困難程度為分組。所以在對話方塊中將軀體健康評分放入「Y，回應」，將呼吸困難程度放入「X，因數」，操作見 (圖12)。

圖12 - 差異性檢定操作

點擊「呼吸困難-軀體健康評分」單因數分析旁邊的紅色三角形按鈕，在下拉式功能表中選擇「不等方差」(圖13)。

圖13 變異數同質性檢定操作

輸出結果見圖 14，若變異數同質性檢定的 p 值大於 0.05，可以認為變異數具有同質性。

圖14 - 變異數同質性檢定結果

本案例為兩組變異數同質性的檢定，可選擇任一檢定結果（因為兩組是多組的特例），各檢定結果均顯示 P<0.05，則後續應採用校正的 t 檢定 (t’ 檢定)。

注意：如果是多組資料的變異數同質性檢定，則結果中不會出現 F 雙邊檢定的結果。

以上就是今天分享的基於 JMP 的常態性核對總和變異數同質性檢定的一些方法與注意事項。在下期文章中，我們將介紹如何在 JMP 中進行 t 檢定。敬請期待！下載 JMP，開啟你的資料分析旅程。

原文：如何在JMP中实现正态性检验和方差齐性检验？

【推薦閱讀】#資料分析必學10技巧

一文學會3種常用的t檢定 - 獨立樣本t檢定、單樣本t檢定和配對t檢定

Michelle_Wu — Tue, 14 Sep 2021 13:29:10 GMT

在過去幾篇的文章中，我們已經介紹了組間比較的基本操作、常態檢定和變異數同質性檢定。今天這篇文章將介紹最常用的 t 檢定：獨立樣本 t 檢定、單樣本 t 檢定和配對樣本檢定，並舉例說明如何使用 JMP 完成分析。

常見的三種t檢定

t 檢定主要用於檢定某一樣本統計量是否與總體參數相等，在實際應用中，最常見的有三種場景：獨立樣本 t 檢定、配對樣本 t 檢定和單樣本 t 檢定。

獨立樣本 t 檢定：用於檢定兩組獨立樣本的平均值是否有統計差異
單樣本 t 檢定：用於比較樣本資料與一個特定值之間是否有統計學差異
配對 t 檢定：用於檢定配對樣本的平均值是否有統計學差異。

本文主要用到兩個資料，一個是軀體健康評分的資料 (圖1)，另一個是皮膚含水量的資料 (圖2)。

圖1 軀體健康評分資料

圖2 皮膚含水量數據

1. 獨立樣本t檢定

所謂獨立樣本，是指樣本所來源的總體之間是相互獨立的，如兩組軀體健康評分的比較，張三的評分與李四的評分並無關係，是相互獨立的。

例1：在軀體健康評分資料中，我們想比較吸煙人群與不吸煙人群間的軀體健康評分是否存在差異。

首先通過點選 JMP 功能表「分析」→「以 X 擬合 Y」，調出組間差異比較的介面(圖3)。

圖3 獨立樣本t檢定操作——功能表選擇

本例中軀體健康評分為結果，吸煙為分組，所以在對話方塊中將軀體健康評分放入「Y，回應」，將吸煙放入「X，因數」(圖4)。

圖4 獨立樣本t檢定操作——變數選擇

結果如圖 5 所示。

如果大家看過上一期的文章，應該知道，雖然圖 5 並無任何統計學結果，但我們可以通過點擊左上方的紅色三角形按鈕，顯示我們所需的統計分析結果，這些結果的選擇，需要結合資料的常態性和變異數同質性來判斷。

檢定結果顯示該資料呈常態分佈且變異數具有同質性，因此可以考慮採用獨立樣本 t 檢定。

圖5 輸出結果圖

點擊「吸煙-軀體健康評分」單因數分析旁邊的紅色三角形按鈕，在下拉式功能表中選擇「平均值/變異數分析/合併的 t (圖6)」。

圖6 獨立樣本 t 檢定操作——方法選擇

結果如圖 7 所示， t 值(即結果中的t比)為 0.696，同時結果給出了 3 個 P 值，分別為：概率>|t|、概率>t和概率 <t。其中概率 >|t| 表示雙側 P 值，這也是我們最常報導的，概率 >t 和概率 <t 為單側 P 值。除非你在設計時刻意設計為單側檢定，否則一般都選擇雙側 P 值，本例為 0.4878。

結果提示，吸煙人群與不吸煙人群間的軀體健康評分差異無統計學意義。

圖7 獨立樣本 t 檢定結果

如果本例資料滿足常態性但兩組變異數沒有同質性，則可使用校正 t 檢定 (Satterthwaite t檢定)。點擊「吸煙-軀體健康評分「單因數分析左側的紅色三角形按鈕 (圖8)，在下拉式功能表中選擇「t 檢定」，即可獲得校正 t 檢定的結果。

其結果顯示形式和解讀同圖 7 的 t 檢定。

圖8 校正 t 檢定的選擇

**這裡需要注意：

在 JMP 的選項中，「t 檢定」輸出的是校正 t 檢定的結果，而「平均值/變異數分析/合併的 t」選擇輸出的才是 t 檢定的結果。大家在做分析時避免勾選錯誤。

2. 單樣本t檢定

單樣本 t 檢定，通常用於檢定一組資料的均值與指定的目標值之間是否存在統計學差異。如想瞭解某特定職業運動員的紅細胞均值與標準的紅細胞值是否有統計學差異。

例 2：在軀體健康評分資料中，假定軀體健康評分的正常值為 60，擬瞭解患者的軀體健康評分值與健康人群的正常值是否存在差異。

本研究資料為常態分佈，可以考慮採用單樣本 t 檢定。

由於單樣本 t 檢定並沒有組別因素，因此不在「以 X 擬合 Y」功能表中完成，而是在「分佈」功能表中進行實現。

圖9 單樣本 t 檢定——功能表選擇

選擇「分析→分佈」(圖9)。在圖 10 的介面中，將軀體健康評分放入「Y，列」。點擊確定後進入到結果介面(圖11)。

圖10 單樣本 t 檢定——變數選擇

點擊軀體健康評分左側的紅色三角形按鈕，在下拉式功能表中選擇「檢定均值」(圖11)，這裡的「均值」即我們需要指定的目標值。

圖11 單樣本 t 檢定——方法選擇

在彈出的對話方塊中的指定假設均值中填寫 60 (圖12)，即檢定樣本人群軀體健康評分的均值與目標值60是否存在統計學差異。如果已知標準差，也可在圖 12 中輸入標準差，如果未知，則基於樣本資料估計。

圖12 單樣本 t 檢定——指定目標值

結果如下圖 13 所示，提示差異有統計學意義 (t=-5.7130，p<0.0001)。

圖13 單樣本 t 檢定結果

3. 配對t檢定：兩組配對資料的比較

與獨立樣本的概念相對的是配對樣本，即樣本不是獨立的，如某高血壓人群治療前後的血壓值，對於張三該人而言，其治療前和治療後的血壓值會有一定關係，如果治療前為 180，通常治療後不大可能突然降為 120，很可能比 180 會稍低。

例3：在含水量資料中，擬探索患者干預前與干預後皮膚含水量是否有差異。由於本研究探索的是同一批患者不同時間點檢查結果是否有差異，屬於配對資料，若資料服從常態分佈，則使用配對 t 檢定。若不滿足常態分佈，則採用做非參數配對檢定，非參數檢定將在後續的章節為大家詳細介紹。

配對t檢定的統計分析思路為，計算干預前後差值，將差值的均值與 0 比較，從而獲得干預前後的差異是否有統計學意義。所以，在 JMP 中匯入配對資料時，應注意匯入格式需如圖 2 所示，每一個患者的兩次測量值在同一行的兩列上，這樣才能求出每個患者測量值的差值。

配對 t 檢定的實現不是在「以 X 擬合 Y」功能表中，而是在「專業建模」這一功能表下。選擇「分析→專業建模→配對」(圖14)。

圖14配對 t 檢定——功能表選擇

這裡需要提醒一下：本文是以 JMP 15 為例介紹，在不同的 JMP 版本中，配對功能表的位置可能不一樣，在早期的 JMP 軟體中可從「分析」選項下直接找到「配對」模組。

在配對 t 檢定中，至少需要兩個結局變數，本例分別為皮膚含水量(干預前)和皮膚含水量(干預後)，將這兩個變數放入「Y，配對回應」(圖15)。JMP 進行配對 t 檢定的計算時，默認放入的第 2 行變數減去第 1 行變數，在操作時需注意放入變數的先後順序。

如圖 15 中，干預前在第 1 行，干預後在第 2 行，因此結果是基於干預後-干預前。本案例中想要瞭解干預後皮膚含水量是否上升，因此先放皮膚含水量 (干預前)，再放皮膚含水量 (干預後)。

圖15配對 t 檢定——變數選擇

配對 t 檢定結果見圖 16。結果首先給出差值的計算方式為皮膚含水量 (干預後) - 皮膚含水量 (干預前)，結果部分給出了干預前和干預後分別的均值，以及均值差為 6.32，說明干預後的皮膚含水量高於干預前，皮膚含水量平均升高了 6.32。患者干預前後皮膚含水量差異有統計學意義 (t=12.595，p<0.0001)。

圖16 配對 t 檢定結果

以上就是本期為大家帶來的基於 JMP 的三種常見的 t 檢定應用場景、應用思路及分析的注意事項。在後續的文章中，我們將陸續為大家帶來變異數分析及兩兩比較、秩和檢定及其兩兩比較、卡方檢定和趨勢檢定。敬請期待！

原文：一文学会t检验的3种常用方法

一個神奇的JMP功能表，實現資料的所有組間比較

Michelle_Wu — Tue, 14 Sep 2021 13:28:51 GMT

組間比較，是統計分析中比較基礎的內容，簡單來說就是對兩組或多組之間的指標進行比較，這個指標可以是連續變數，也可以是分類變數。凡是進行統計分析，幾乎都離不開組間比較。如隨機對照試驗中，主要目的就是比較兩組的療效有無差異；多因素分析中，組間比較通常也做為基線結果需要展示給大家。

今天就要來介紹 JMP 有一個神奇的功能表，幾乎可以實現所有的組間比較，這個功能表就是 --以 X 擬合Y。

什麼是以x擬合y？

簡單來說就是，把組別因素看作 x，把要分析的變數作為因變數 y。根據 x 和 y 的變數類型，JMP 軟體會自動判斷合適的方法。

為什麼這樣功能會如此神奇？本文就從幾個方面分別說明：

常見組間比較方法的選擇；
從宏觀上把常見組間比較方法串起來；
通過一個案例介紹如何用 JMP 的一個功能表就實現所有的組間比較。

至於具體的組間比較方法，我們會在後面的系列文章中陸續介紹。

如何選擇組間比較方法？

組間比較的方法很多，取決於組別數、分析變數類型等。

連續變數組間比較的統計方法主要有 t 檢定、方差分析、秩和檢驗等。大家在選擇連續變數組間比較統計方法時可以參考圖 1，如果你在進行資料分析時認真參考這張圖，就不會出現用 t 檢定分析非正態資料，或忘記做方差齊性檢驗這樣的錯誤。

圖1 連續變數組間比較方法選擇思路

在選擇統計方法時，主要考慮的因素包括：

研究設計（完全隨機還是配對設計）
分組的數量（兩組還是多組）
正態分佈（正態還是非正態）
方差齊性（方差齊還是不齊）

分類變數組間比較的統計方法主要是卡方檢定和秩和檢驗。方法選擇見圖 2。

在選擇分類變數組間比較統計方法時，主要考慮的因素包括：

研究設計（完全隨機還是配對設計）
比較組數（兩組還是多組）
結局指標的類別（二分類還是多分類；有序還是無序）

方法的選擇主要與結局指標（注意不是分組變數）的性質有關：

若為二分類結局，則使用卡方檢驗；
若為多分類結局，則需要考慮結局指標是有序的還是無序的，若為無序變數則使用卡方檢定，若為有序變數則使用秩和檢驗。

圖2 分類變數組間比較方法選擇思路

JMP中的組間比較功能表

在 JMP 中，大部分組間比較均可在以 X 擬合 Y 功能表中實現 (圖3)，也就是說不管是 t 檢驗、秩和檢驗、方差分析還是卡方檢定，都通過該模組實現。

圖3 以 X 擬合 Y 菜單

以X擬合Y功能表的介面如圖4所示，大家可以先有一個直觀印象。

圖4 以 X 擬合 Y 功能表介面

這一功能表對習慣了具體某一種方法（如 t 檢定、方差分析）的人可能有點疑惑。

大家可以注意到，在圖 4 左下方，根據 x 是分類還是連續、y 是分類還是連續，組合成四類方法：

① X 是分類變數，Y 是連續變數，可輸出單因數分析結果，包括 t 檢定、方差分析和秩和檢驗等；

② X 是分類變數，Y 是分類變數，輸出列聯，即卡方檢定的結果；

③ X 是連續變數，Y 是連續變數，輸出單因素線性回歸；

④ X 是連續變數，Y 是分類變數，輸出 Logistic 回歸。

所以，這就是 JMP 功能表的特色，它把統計方法從目的上歸類了，因為不管是 t 檢定、卡方檢定、方差分析等，其結構都是一樣的，就是一個分組變數，一個分析指標，也就是對應上面提到的①②這兩種情形。也就是說，組間比較是包含在以 X 擬合 Y 這一功能表中的，但以 X 擬合 Y 這一功能表的功能並不僅限於組間比較，後面文章中我們還會繼續介紹。

如何使用「以X擬合Y」完成組間比較

接下來，我們通過一個案例來看下如何使用「以 X 擬合 Y」實現組間比較。

假定我們有如下資料 (圖 5)。我們的目的是想探索不同呼吸困難程度人群的軀體健康評分是否有差異。換句話說，呼吸困難程度是分組變數 X (也叫引數、解釋變數)，軀體健康評分是分析變數 Y (也叫結局變數、因變數)。

圖5 本文所用數據

點擊「分析」→「以X擬合Y」進入操作介面，如圖 6 所示。呼吸困難為分組變數，因此將其放入「X，因數」，軀體健康評分為結局指標，將其放入「Y，回應」，點擊確定，則可進入結果介面。

圖6 組間比較操作示意圖

結果如圖 7 所示。一看這個結果，你可能很納悶，因為結果中只有圖 7 顯示的一幅圖。

這也是 JMP 的一大特色，即：在結果介面中可以根據你的目的顯示所需的結果，而不是在功能表中選擇。在結果中選擇有個好處，你不用來回切換到功能表操作介面。

圖7 輸出結果圖

如果你點擊圖 7 左上角的紅色小三角形功能表，就可以看到可選的結果是如此豐富 (圖8)，這裡包含了連續變數組間比較中各種可能用到的方法，如 t 檢驗、方差分析、各種非參數檢驗、等效性檢驗、方差齊性檢驗等。你可以根據實際情況進行勾選，以顯示相應部分結果。

圖8 連續變數組間比較結果介面

如本例中，分組變數呼吸困難分為兩組，我們可以回頭對照圖 1 的思路，對於兩組的連續資料比較，可以考慮的方法有 t 檢驗、校正的 t 檢驗 (t’ 檢驗) 和非參數檢驗。

至於最終到底需要選擇哪種方法，這就不是統計軟體的問題，而是分析者本人的分內工作，你必須先明確兩組軀體健康評分是否滿足正態分佈，是否方差齊，如果這兩個條件都滿足，你可以勾選 t 檢驗的結果，否則可能需要從選擇非參數檢驗方法。

比如，我們想瞭解兩組是否方差齊，可以通過勾選圖 8 中的「不等方差」，勾選後，結果介面會多出關於方差齊性檢驗的一部分結果 (圖9)。結果顯示，P 值遠遠小於 0.05，不能認為方差齊，也就是說，不能直接採用 t 檢驗。

結果還同時給出了方差不齊的條件下的兩組比較結果，即圖中最後一部分的 Welch 檢驗結果，結果中已經很明確地說明了，該方法可以用於兩組方差不相等的情形。從該結果可以看出，兩組軀體健康評分是有統計學差異的。

圖9 方差齊性檢驗結果

JMP組間比較功能表4大特色

通過以上的初步介紹，相信大家對 JMP 實現組間比較操作已經有了一定瞭解，從中可以發現 JMP 統計分析的 4 大特色：

1. 操作極其簡單

JMP 的操作功能表很簡單，只需要選擇 x 和 y 就行了，也就是說，只要你能分清楚，哪個是分組變數，哪個是結局變數就可以了。這一點對研究人員來說不是什麼難題。

2. 探索式自主分析

JMP 的結果介面賦予分析人員很大的主動性，你可以根據自己的需要，勾選出或勾選掉某些結果。

這其實是一個好事，因為這要求你必須先明確你要採用什麼方法，而不是每逢連續變數，必用 t 檢驗。你需要先弄清楚連續變數的正態性和方差齊性，然後根據結果，勾選你想要的結果。

3. 無處不在的互動式視覺化

作為一款以互動式視覺化為特色的軟體，JMP 分析結果中能夠以圖形展示的部分，都會以圖形呈現結果，這可以方便分析人員更直觀地發現組間差異大小。可以說，互動式視覺化分析在 JMP 中無處不在。

4. 在結果中二次探索分析

JMP 結果介面中，除了提供各種方法，還提供了各種可供修改的選項，你可以隨時在結果中對圖形、結果進行修改，靈活地開展探索性分析，十分方便。

透過一個簡單的案例對 JMP 組間比較的強大功能進行了介紹，在後續的文章中，我們將會帶領你針對具體的組間比較方法及其如何在 JMP 中進行操作詳細說明，下載 JMP，開啟你的資料分析旅程。

原文：一个神奇的JMP菜单，实现数据的所有组间比较

【推薦閱讀】#資料分析必學10技巧：

So little to lose, so much to gain at Discovery Summit Americas

Diana_Levey — Tue, 31 Aug 2021 13:47:04 GMT

It's time to sign up! Registration is open!What do you say when junior colleagues find classes they want to take? You say, “Take them!”

What do you say when senior colleagues see ways to sharpen their skills? You say, “Sharpen away!”

What do you say when teammates want to benchmark best practices with other organizations? You say, “Go for it!”

You encourage others to follow their passion, to continue their education, to learn and grow every single day. But what about you? We’ve heard a lot about self-care these last 18 months. Maybe it’s time you put some thought behind it. Maybe it’s time you invested in yourself and in your own analytic journey and signed up for Discovery Summit Americas.

Join us for conversation-starting keynotes, simu-live papers, poster presentations, time with experts and networking opportunities. So little to lose, and so much to gain.

“Regardless of whether or not you think a plenary, paper, or poster topic directly applies to you and/or your work, you will ALWAYS learn something either useful, interesting, or both.” That's what an attendee from last year’s event said.

Here's what another attendee from last year’s event said, “If you want to deploy the full potential of JMP, it is a must to attend to the Discovery Summit.”

A third attendee said the conference is more than learning about JMP: “It's a clinic in how to think analytically and apply analytical thinking throughout your organization. This is so necessary in today's world.”

Like you, we believe in the power of analytics...and in empowering people with tools and best practices to do their best. In the spirit of democratizing data exploration, we’re making it easy to attend Discovery Summit Americas. It’s online and free of charge.

資料清理之「拆分」：在 JMP 快速實現一欄拆分為多欄

Michelle_Wu — Mon, 30 Aug 2021 15:20:55 GMT

在資料分析的過程中，你可能需要將一欄或多欄指標細拆成多欄指標。例如在人口普查結果中瞭解男女性在不同年齡區段的分布情形；又或者，出於分析或做圖需要，依據性別將身高拆分別兩列等。今天就要帶你瞭解如何在 JMP 實踐將一欄資料拆分成多欄。

假設我們有一份重複測量資料，時間點和測量值是作為兩個變量記錄的，例如大家常見的男女對比金字塔圖，也需要將男和女的資料分列成兩個變量後再進行繪製。今天我們就依據這兩個例子，給大家介紹 JMP 的另一個實用功能——拆分。

首先看一下重複測量的範例資料，共 15 個受試者，分為三組接受不同處理，重複測量三次，資料紀錄（截取部分）如圖1。

圖1

為了進行重複測量方差分析，我們需要將三個時間點的測量值根據時間點分成三列。拆分的對話框通過點擊工具欄的表 (Tables)→拆分 (Split)，如圖2.

圖2

我們將時間點放入拆分依據框中，然後將測量值放入拆分欄，即根據時間點對測量值進行拆分，拆分為與時間點個數相同數量的欄。其餘欄我們可以選擇全部保留，如果後續分析用不到，也可以選擇全部刪除，或者手動選擇想保存的欄，如圖3。

圖3

因為拆分資料欄必然形成新的資料表，可將新的資料表重新命名。以下我們用簡單 gif 進行動態示範，見圖4。

圖4

拆分之後生成的新資料表如圖 5 所示。原資料為 45 列，每個被試者三個時間點各佔一列，拆分之後，新數據為 15 列，每個被試者佔一列，三個時間點分別在 t1 - t3 三欄顯示，這樣就可以滿足重複測量方差分析的資料格式要求了。

圖5

再給大家列舉一種需要用到 JMP 拆分功能的情形，例如我們常見的性別金字塔，如圖6

圖6

若想做出這樣的圖，男性和女性的資料必須單獨成列，做為兩個變量，但是在正常情況下，性別往往是作為一個獨立變量存在的，這時就需要用到今天講的拆分功能。

我們以某個資料犯利用動圖進行展示，假設我們想做圖展示男女糖尿病人的 BMI 值分佈情況，該怎麼拆分呢？如圖7動圖所示。

圖7

當新的資料表生成後，我們就可以做圖了。至於如何製作金字塔圖，敬請關注本系列續的 JMP 做圖專輯。

欄的拆分和堆疊，是大家在 JMP 中清洗資料是非常常見且實用的功能，#資料分析必學10技巧系列文章持續推送中，敬請期待！下載 JMP 試用，跟著文章練起來吧！

推薦閱讀：

原文連結：數據清洗之“拆分” — 快速實現一列拆分為多列

註：本文為此系列文章的第六期。點擊 #資料分析必學10 技巧，即可回顧其他文章。

資料清理中的「堆疊」：多行資料合併處理的神器

Michelle_Wu — Thu, 19 Aug 2021 14:23:26 GMT

在資料分析的過程中，有時候我們經常需要將多欄指標合併為一欄，比如將一個 100 人 5 個觀察時間點生成的 100 列 5 欄數據表，轉換成 500 列 1 欄的數據表，從而滿足作圖、分析的進一步需求。那麼，在 JMP 中如何快速實現呢？今天這篇文章我們將介紹：如何在 JMP 將多欄資料合併處理的技巧。

在分析資料時，我們經常會遇到這種情形：一個人同時有多個主要分析指標（如隨訪資料中的多個時間點資料），在匯入資料時，通常是將其分別匯入在不同欄（如 5 個時間點資料，分 5 欄匯入）。這種匯入雖然看起來直觀，但在分析時，有時卻需要將這幾欄指標放置在同 1 欄中。

如果僅僅是 5 欄還好，逐一複製也花費不了多少時間。但如果是 20 欄、甚至更多的欄位呢？不僅費時費力，複製貼上也容易發生錯誤。今天我們便要來認識資料清理中的「堆疊」該如何實現快速將多欄合併為 1 欄，同時新增一個標識變量，以顯示不同指標。

下面我們就用 JMP 自帶的示範資料，來示範一下如何實現資料堆疊，您可以在最上方的選單「Help」--> 「Sample Data Library」中找到示範資料，如下圖 1

圖1 ：JMP幫助文檔及示範數據

這次我們作為講解示例的是樣本數據庫中的 Blood Pressure.jmp 數據文件。數據的整體情況如圖 2 所示。

圖2 示範資料

20 個被測者分為四組，變量 BP 8M 的含義是周一的八點血壓測量值。所以該數據記錄的是每個被測者分別於週一、週三和周五每日三個不同時間點的血壓測量值。沒錯，這是標準的重複測量數據，臨床上有很多醫生收集的數據都是這樣的格式。

第一步：將多欄指標堆疊成一欄指標

首先，想像一下這樣的情境，得到數據之後，你想比較一下四組被測者的血壓測量值是否有差異，這時，血壓測量值變成了因變量 y，測量時間點和分組變成了自變量 x1 和 x2，目前的數據格式，顯然不符合我們接下來分析的要求。

我們需要將 9 個時間點的欄進列堆疊，形成兩欄，一欄時間點，另一欄測量值，且一一對應。在 JMP 裡該怎樣操作呢？只需一步！

圖3 - 在選單「Tables」中選擇「Stack」，會出現堆疊的對話框

Stack columns – 選擇你想要堆疊的欄位
Output table name – 輸入堆疊後的table 名稱
Stacked Data Column / Source Label Column - 由於我們堆疊完成後會形成兩個新變量，一個是時間點（源標籤欄），一個是血壓測量值（堆疊數據欄），我們可以分別為其設定變量名，如果不做修改，則新變量會採用系統默認的【Data】和【Label】作為變量名；

操作過程如圖4：

圖4 - 欄堆疊操作過程演示

於是，新生成的資料表，如圖 5 所示：

圖5 - 欄堆疊後新生成數據表

20 個被測者每人測了 9 個時間點的血壓值，所以新生成的數據表有 180 列觀測，標籤和數據分別為堆疊前的時間點和血壓值。這樣，我們就可以進列後續的分析了。

第二步：將多欄指標堆疊為多欄指標

出於不同的分析目的，你可能在想：操作與之前相同。

不過，不同的是分析目的的需要

圖6 - 多序欄堆疊

因為這裡是連續的三欄變量堆疊為一欄，共堆疊成三欄，所以我們將序欄數寫為 3，並勾選「連續」，點擊確定即可，如下圖 7：

圖7

生成的資料表，如圖 8 所示。每個被測者的每日數據為一欄：

圖8

日常資料整理與匯總中，除了存在將多欄堆疊為一欄的情況，還有將一欄拆分為多欄的情況。我們將在之後的文章中為大家介紹「堆疊」的反向操作--拆分，即拆分究竟有何意義？又有哪些注意事項？敬請期待後續的 #資料分析10技巧專欄！

原文連結：數據清洗之“堆疊”—多列數據合併處理的神器

推薦閱讀：

註：本文為此系列文章的第五期。點擊 #資料分析10 技巧，即可回顧其他文章。

Predictive models save the day(ta): Well log interpretation and prediction are easier than you think

Peter_Polito — Wed, 18 Aug 2021 12:42:41 GMT

My favorite class in high school was Earth Science. I recall taking daily weather readings during our meteorology section. Every morning, we went outside, measured wet and dry bulb temperature, wind speed and direction, and estimated cloud cover. Once or twice, we returned to the classroom only to realize a reading was off or didn't make sense. No problem, we went back outside and redid the measurement – it was no big deal to walk the 50 paces and retake the measurement.

The Problem

Fast forward about 18 years, and I am working on the Helix Q4000, attempting to acquire methane hydrate samples from sediment over a mile beneath my feet. It took over 24 hours to acquire our first core. When it came to the surface, nothing had worked properly: The sample was depressurized, and the 10% or so of recovered sediment was just soup. Unlike my temperature readings in high school, we couldn't go back and try again – that sediment sample was forever gone.

This is the challenge geologists and engineers regularly face working in the harsh subsurface environments. The data is expensive to acquire, and you only get one shot. So what do you do when the data doesn't come back as it should? Maybe the tool fails at depth or worse yet, gets stuck and the entire well has to be abandoned? How can we take the little information we have and make useful sense of it? We can use JMP Pro's powerful predictive tools, of course!

The Experiment

Input parameters of Gamma (blue), Formation Density (red), Compressional Velocity (green), and Shear Velocity (purple). Interval one is >2064 ftkb and is above the black upper black line, interval two is between the two black lines (2065-2127 ftkb), and interval three is below the black line (2128-2206 ftkb).

Let's compare a few different methods of predicting a poorly acquired variable from more robust ones. In this instance, we are trying to understand hydrocarbon saturation (methane hydrate), which is sparsely available in this particular well (Mt Elbert well, North Slope, AK). What we do have is complete sets of four variables: Gamma Ray, Formation Density, Compression Velocity, and Shear Velocity. In the subsurface, there are three intervals of hydrate bearing sands, and we want to use the first interval (2003-2064 feet below the kelly bushing, ftkb) to predict hydrate saturation in the lower two intervals, 2065-2127 ftkb and 2128-2206 ftkb, respectively. We will test for different predictive models: Standard Least Squares, Generalized Regression, Neural Network, and XGBoost. We will use R2 values as a first pass of model quality but will then qualitatively evaluate the actual vs. predicted well logs to assess overall quality of each model.

Results

Actual hydrate saturation (blue) compared to predicted hydrate saturation for Standard Least Squares (red), Generalized Regression (green), Neural Network (purple), and XGBoost (burnt orange).

Standard Least Squares just doesn't cut it. It robustly models the Interval 1 but completely fails to predict with any accuracy the hydrocarbon saturation in the lower two intervals. Generalized Regression (Adaptive; Elastic Net) is marginally better but generally overpredicts the lower two intervals by 0.1 to 0.2 saturation units. The machine learning models do a much more robust job, but I give the edge to XGBoost. Neural Network produces a much "noiser" signal in that there are numerous kicks that account for 10-20% over and under prediction, while XGBoost provides a much more stable prediction at both low and high hydrocarbon saturations.

This is a common theme I observe when working with geologic and/or downhole data: The natural systems are incredibly complex, fraught with heterogeneities, and machine learning models (Bootstrap Forest, Neural Network, and XGBoost, etc.) tend to provide better predictive models overall.

Model results for validation data sets of XGBoost (top) and Neural Network (bottom).

Moving Forward

Most folks working in the industry are going to have substantially more data at their fingertips than a single well. A strong practice is to use clustering to group wells together using geologic parameters (lithology, porosity, permeability, etc.) and construct predictive models by cluster. This method can enable you to achieve better outcomes because the model is purpose-built for a particular play, or even a particular basin within a particular play. Beyond that, a great practice is to take a visual walk-about with your data in Graph Builder – try to get a sense of what is going on with individual parameters to identify a best path forward.

Lastly, democratization of data analytics in your organization is going to allow you to achieve greater ends quicker: get powerful tools in user's hands, show them how to build and interpret models, and then sit back and enjoy!

Data Source

This data is was collected through a cooperative agreement drilling between DOE and BP Exploration Alaska (BPXA), in collaboration with the US Geological Survey (USGS) and several universities and industry partners, to evaluate whether natural gas hydrate from the Alaska North Slope could be viably produced either technically or commercially. The Mt. Elbert well was drilled in 2007, and the data is available through the Lamont Doherty Earth Observatory at Columbia University. I subset the data, cleaned it and made it available in a JMP table on JMP Public (Note: Without JMP Pro, the GenReg, Neural w/ KFold, and XGBoost scripts will not execute. Furthermore, the XGBoost script requires JMP Pro and the XGBoost add-in).

Back to class with JMP: Easy access to academic teaching resources

curt_hinrichs — Tue, 17 Aug 2021 13:00:00 GMT

Our academic goals are simple, to help faculty teach more modern and data-driven courses, to help make concepts and their application clear and concrete, and to make getting, learning and using JMP as easy as possible for students. We think these goals better support the practice of statistics and help students see the value and relevance of statistics in their lives and their future endeavors. If you share these course goals, we want you to know about some resources that should help you along the way.

The JMP academic team is also excited to share some good news about these teaching resources. To streamline access to key resources we have redesigned and reorganized the academic website at JMP.com/academic. Faculty will now find teaching resources (e.g., step-by-step guides, videos and case studies) organized by course area at www.jmp.com/courses. This is an easy button where each course collection contains all of the relevant JMP resources organized by the common sequence and curriculum.

Each course link brings up a consistent set of Essential resources including textbooks that integrate JMP for that specific course area, one-page step-by-step guides and short videos (updated for JMP 16), and case studies that provide real-world scenarios and solution paths along with exercises that can be assigned. We have many new case studies that have just been posted (you can see a list of all cases at www.jmp.com/cases), and all exercises in the cases now have solutions that are available to adopting faculty only. We consider these resources to be more foundational to modern data-driven courses that seek to teach concepts through applications and by example.

In addition to these “essentials,” we also provide course Enhancements such as teaching applets, lab activities and additional data sets. These are designed to augment and enrich a class where appropriate. Most course collections also include Additional Resources such as online courses and content. These are helpful when teaching online or where you might want to offer additional content in a course. All resources at JMP are free to use and distribute to students by faculty adopters.

In addition to these teaching resources, the JMP academic team provides regular live webinars for faculty and students at www.jmp.com/academic-webinars where you can also register for our first webinars of the fall, "JMP 101: Teaching Statistics with JMP" on Aug. 25 and "Resources for Teaching with JMP" on Sept. 8. You can also access and view many webinar recordings from past seasons. Finally, if you have any questions about getting, using or teaching with JMP, do not hesitate to reach out to us at academic@jmp.com.

拆分資料集有煩惱？這些實用妙招試試看！

Michelle_Wu — Thu, 19 Aug 2021 14:24:03 GMT

在日常分析數據的過程中，我們往往只需要整個數據集中的一部分，比如只關註一部分觀測/行（男性或女性、某個年齡段的患者或者是患有某種疾病的患者）或者一部分變量/列等等這些可以稱之為子集的數據集，而拆分原始數據的過程也同時是生成子集的過程。

那麼如何選擇符合條件的觀測值？如何一鍵拆分數據為多個子集？JMP中又有哪些簡便快捷的隨機抽樣方法？今天就帶大家一起學習資料清理的一個重要部分。

在 JMP 中生成子集

圖1 生成子集的主要操作頁面

那麼具體該如何拆分呢？

這個對話框裡的每個選項都有何意義？

又該如何利用好這些選項呢？我們先從最簡單的說起。

01如果只需要一部分變量（列）

比如在下圖的數據集裡，只需要Y和年齡兩個變量（當然實際情況不會是這樣，一般情況下，除非變量特別多，都不需要單獨把一些變量拎出來），你只需要在數據表上選中你想要的列，多列的時候記得按住Ctrl鍵，然後打開子集對話框，如圖2操作。

圖2 將部分列生成新的子集

這樣就得到了由所有觀測（行）和選定變量（列）組成的新數據集。

然而更多的時候，我們更想選擇符合特定條件的觀測（行），對變量不做要求，或者是選擇一部分觀測和一部分變量，那下面的幾種拆分行的操作就能派上用場了。

02 如果只需要一部分觀測（行）

(1) 選擇符合條件的行

這個標題既是我們的目的，又是我們實現目的的操作。比如我們想選擇數據集中年齡＞40歲的觀測，我們可以通過【行】→【行選擇】→【選擇符合條件的行】，然後編輯篩選觀測的條件，點擊確定，數據中符合條件的觀測就會被選中。

本例中選中320行觀測，如果希望將這些觀測生成為新的子集，那麼只需要點擊【表】→【子集】，不需要做更改（當然，如果你在這之前已經選中了部分變量，那麼需要選中【選定列】，而非【所有列】），點擊確定即可將選中觀測生成新的數據集，如圖3。

圖3 選擇符合條件的行

小貼士：選擇符合多重條件的觀測

很多時候我們需要選擇的觀測要滿足≥2條規則，比如我們要選擇大於40歲的女性，同樣的流程，只是在編輯規則時稍稍複雜一點，見圖4。條件之間的關係是“和(and)”時，選擇【若符合所有條件】，是“或（or）”時，選擇【若符合任意條件】。

圖4 選擇符合多重條件的觀測

(2) 隱藏不想分析的行

上一個方法介紹的是如何將想要的觀測生成新數據集，以便我們進行分析。這裡介紹如何從另一個角度得到我們想要的行，那就是隱藏掉不想要的行。

同上面講述的選擇行的方法一樣，你可以從【選擇符合條件的行】選中你將要分析的行，然後反向選擇進行【隱藏和排除】操作，見圖5（圖中已經完成了選擇符合條件的行）。

小貼士：這裡建議大家直接選擇【隱藏和排除】，因為只是被【隱藏】的觀測同樣會進入到後續的統計分析中，只是在作圖表時不出現，而只是被【排除】的觀測不會進入到後續的統計分析中，但會在作圖時出現。一般情況下，對於這些不符合篩選條件的觀測，我們既不想讓它們出現在圖表中，也不想讓它們參與後面的統計分析，所以直接選擇【隱藏和排除】就好啦。

圖5 隱藏和排除不想分析的行

另一種方法就是直接反向編輯選擇條件，直接篩選出不想納入分析的行，再點擊【隱藏和排除】。再次點擊【隱藏和排除】即可取消。

(3) 一鍵拆分成多個子集

如果出於分析需要，我們想根據某變量的不同值將數據拆分成多個數據集，例如拆成男、女兩個數據集，或根據示例數據中的Y（有序型）拆成high、medium和low三個數據集，這在JMP裡的操作有多簡單呢？

我們以後者為例，見圖6。在勾選【取子集依據】後彈出的變量名列表裡選擇你想拆分數據集的依據，本例以Y（有序型）為依據，JMP將數據集自動拆成三個子集，分別是Y（有序型）=low，Y（有序型）=medium以及Y（有序型）=high三個子集。

圖6 按照變量值拆分子集

小貼士：

以變量為依據，JMP會將變量中的每一個值單獨生成一個新的子集。舉例來說，如果你不小心將連續型的年齡變量作為拆分依據，那你的屏幕將會彈出上百個子集，每個子集的觀測具有相同的年齡，這點要小心哦。

(4) 隨機抽樣

作為拆分數據集的一個特例，隨機抽樣其實也可以看作是數據集的拆分，相當於把隨機選中的觀測拆分出來，就形成了隨機抽取的樣本。

在JMP中進行隨機抽樣有多種方式，這裡介紹的是最簡單的一種隨機抽樣方式具體更詳細的隨機抽樣方法我們會在後面文章中有專門介紹。

這種方式仍然在【表】→【子集】的對話框中，如圖7所示，紅框內的部分就是進行隨機抽樣操作的關鍵選項，兩種方式任選其一，

圖7 隨機抽樣

圖7所示，點擊「確定」後，你會發現一個新的數據集就出現了，這就是你想要的隨機抽取的樣本數據。這大概是史上最簡單的隨機抽樣方式了，有了這一工具，還用擔心什麼抽樣問題呢？

當然，這種隨機抽樣只是簡單隨機抽樣，實際中的隨機抽樣方式也有多種，如分層隨機抽樣等，這在JMP中實現起來也非常容易，我們將在之後隨機抽樣的文章中專門介紹。敬請期待！

What you can (and should) do with all that text data

anne_milley — Thu, 05 Aug 2021 16:10:24 GMT

We were very excited to feature Manya Mayes, a natural language processing (NLP) expert and Head of Data Science at 1440 Consulting, as the plenary speaker in our first Statistically Speaking on text analytics. We were also delighted to host her as a panelist with two other savvy text analytics practitioners: Weida Tong, Bioinformatics & Biostatistics Director at the Food and Drug Administration; and Jeff Swartzel, Scientist at Procter & Gamble.

The practical expertise these statistically minded experts convey and the interesting examples they share are inspiring. We had so many questions for the panel during the livestream that we couldn’t get to them all. Our featured guests have kindly agreed to answer them via this blog post.

How do you measure and demonstrate ROI for text analytics?

Manya: As you have anticipated, measuring ROI for text analytics is not always easy. When identifying early warning of a product issue, it may never be possible to identify the true ROI for a problem that is fixed before it becomes a Ford Explorer/Firestone Tire issue (which had an estimated cost of $7.5B) or the Toyota sudden acceleration issue (which incurred $1.2B in penalties from the Department of Justice and $2B in recall costs). Had both companies taken action on text analytics results (which were easy to find using the National Highway Traffic Safety Administration [NHTSA] data and good text analytics capabilities), they may never have truly known how much they saved. The use of text analytics in a labeled and supervised prediction model provides a way to validate the ROI, but this is not always possible. In addition, for the analysis of data that informs future company business decisions and strategy, it is possible to measure the return per dollar invested via a driver/sensitivity analysis.

Jeff: This is tricky, and I often struggle with this question because the impact can be inconsistent. Sometimes very quick and easy work gives valuable insights, whereas other times, a high-effort analysis can yield results that are less impactful. The things I typically focus on as the value from text analysis are: 1) time saved, 2) the ability to intervene ahead of problems, 3) the uniqueness of insights that might not have been possible without the technique, and 4) increased ability to focus on topics of importance.

How was the plot on the screen (whompy wheel vs. time) generated from the Text Explorer window?

Manya: Using Text Explorer on the sample of complaints about electric vehicles, I removed a set of low information terms by adding them to the stop list, then I removed terms that appeared once or twice, and followed that with a latent class analysis. The results of the latent class analysis (in JMP Pro) include a Cluster Probabilities by Row window. In that window, it shows the most likely cluster for each document. It is possible to right-click on Make Into Data Table. The next window contains the results as a data table that can then be joined back into the original sample of vehicle complaints by selecting Tables > Join and then joining these results with the original data, matching by row number. I selected just the rows for the whompy wheel cluster. The original data has a faildate field that can then be plotted using Graph > Graph Builder. In Graph Builder, I used a histogram, with faildate as my X variable, and the Response Scale set to Count. The remainder was mostly cosmetics, where I changed the color by clicking on the faildate variable in the legend (see attached image for more details). To get the information by model, I dragged and dropped the vehicle model variable on the Group X portion of the graph and used the vehicle model as a color variable.

Why are scores for negative sentiments lower than positive sentiments (98 v. 100)?

Manya: The sentiment scores range between -100 and 100. I just happened to pick negative terms with scores of -98 and positive terms with scores of 100.

These images show that, on the whole, there were more positive documents than negative ones. The scores show the sentiment calculated across the whole document (review), so a document could have an overall positive score, even though it has parts that are negative and parts that are positive.

Is the text analysis for the FDA review process used more in targeting review efforts, or is it supplementary to reading documents?

Weida: Text analytics has been used in a broad way in the FDA, including both research and review. Among them, information retrieval and document classification have a strong presence in the review process. In addition, we also investigated text summarization, named entity recognition, and sentiment analysis, most of which is a regulatory science-centric endeavor.

What synergies do you gain with text analytics and other analysis methods?

Manya: The combination of text-based information into other analysis methods is common. The text is essentially given a structured representation. The incorporation of the now structured text can help with increasing model accuracy for machine learning supervised models when applied to a whole host of use cases (customer acquisition, retention, fraud, risk, etc.). It can also help with model explanation, and it can inform the right language to use for branding, messaging, etc. It can also help with unsupervised classification/clustering, where the terms, in conjunction with structured data such as transactional and demographic information, can help describe the clusters.

Jeff: JMP is great for directly connecting the output of text analysis to other relevant variables (time, customer rating, meaningful categories, etc.) for modeling (term selection and discriminant analysis). Relevant keywords and topics can also inform more qualitative analyses elsewhere.

What do you think about the bag-of-words approach vs. more advanced NLP?

Manya: The bag of words approach does not take context or directionality into account, although phrases can be included, making the analysis simpler and quicker, while still providing value. The more advanced NLP does provide more context, although it still requires pretrained models and the context can be much more localized (more likely to be within a sentence, for example). The more advanced NLP provides greater automation, but potentially misses the nuances involved when a data scientist gets to know the data in detail. Essentially both techniques have their pros and cons.

Jeff: I sound like a broken record, but I always say, “Connect the analysis technique to the question.” Frequently, I’m able to get fast and useful insights with basic methods (especially when the question is clearly defined). People will come to me and ask something along the lines of “We don’t know what we’re looking for, but we want to know everything about everything. Can we NLP our way to omniscience?” :grinning_face_with_sweat: I’ll either help them focus on question definition or start exploring the text with latent class analysis, latent semantic analysis, or sentiment analysis. Also, the bag-of-words approach has the benefit of being very easily conceptualized and interpretable, and it’s a great starting point. In my limited experience, methods like BERT are great for classification questions with well-trained machine learning models.

What does it mean to move toward “advanced regulatory science”?

Weida: The FDA defines regulatory science as “the science of developing new tools, standards and approaches to assess the safety, efficacy, quality and performance of FDA-regulated products.” In a sense, advanced regulatory science means taking emerging technologies and methodologies to improve the FDA’s operation.

How does JMP facilitate your collaboration with others and with text analytics in particular?

Manya: The JMP Community is an excellent platform for collaborating with others. Text analytics is part of the JMP Community, as well as SKP (the JMP Statistics Knowledge Portal), which is about to become more active with text analytics content and collaboration.

Jeff: At P&G, we have a very large JMP user base, and many people are familiar with the basics. It’s easy to share the output from a JMP analysis and have a working session with someone using Text Explorer. Sharing scripts and journals makes it very easy.

Do you think we have just scratched the surface of what can be done with all the textual data we are collecting?

Manya: Recently, text analytics and NLP have made some huge jumps in capabilities, but there is still more to come. There is a good scratch in the surface, but with additional capabilities, text won’t be considered any differently than structured data. For now, the human element is still very valuable.

Jeff: Yes, and I think that building a culture of people who are familiar with the fundamentals of this kind of work is the best starting point for building an understanding of analyzing text data

What else do you share with the people with whom you analyze text data?

Jeff: Sometimes I will make it a point to remind them that it’s not smarter than we are, meaning the analysis techniques aren’t going to be able to look at a sentence like “I stopped using your product because it smelled funny” and understand it on a human level. The funny smell might be an indication of a big problem, or something people typically say about this product, or even part of an intentional change that was understood to be received this way, or even something sarcastic and ironic. Text analysis and NLP always need to be paired with domain knowledge and curiosity around the questions that we want to answer. We can’t (yet) push a button and know all what we want to know (and even what we don’t know that we might want to know).

If you missed the live event, you can watch it on demand for more insights. In addition to the above questions, we had many other excellent questions from our viewing audience. Learn more about “whompy” wheels, funny smells, and other insights you could gain from some of the text data you have collected.

Visualize your well survey data in 3D with the NEW Well Trace Add-in

Peter_Polito — Fri, 23 Jul 2021 14:49:04 GMT

Just here for the Well Trace Add-in? Click here. Want to learn about the challenges of drilling oil wells? Read on...

I spent nearly a month on the Helix Q4000, where we successfully acquired 21 pressure core samples of methane hydrate from a reservoir located below a mile of ocean and 1300' below the seafloor. Regulation required we complete a directional survey as a part of the operation.Drilling for oil is a complicated process. Effectively, drillers are trying to jam a tiny straw through thousands of feet if not miles of rock to punch into a reservoir that might be only tens of feet thick. The Macondo Prospect — made infamous by the terrible tragedy aboard the Deepwater Horizon and subsequent oil spill — is 41 miles offshore, in about 5,000 feet of water and through an additional approximately 13,000 feet of ocean sediment and rock to reach a sandstone reservoir with an average thickness of a mere 25-43 feet (Pinkston & Flemings, 2019). Onshore, drillers often drill vertically for a short distance and then drill horizontally to increase the surface area within the reservoir. If we remove for the moment concerns about greenhouse gases, politics, concerns about the environment, and many global economies, drilling for oil and gas is truly an engineering marvel.

When operators devise a well plan, they have an ideal well trace in mind: "Straight down for 4 miles, then turn 90 degrees and drill 2700' at 230°." I know it's more complicated than that, but you get the gist. The reality is that the drill bit wobbles, the drillers might have to make unplanned accommodations in real-time, and the geologists may review the cuttings and decide to alter the plan — basically, a lot can happen that leads to minor or major deviations from the well plan. That is where the directional survey comes into play.

I had the privilege of taking part in a scientific drilling operation in deepwater Gulf of Mexico in 2017 where we used a core sampling technique called Pressure Coring to acquire methane hydrate samples and return them to our research lab for study while maintaining in-situ temperature and pressure (~5°C and ~2500 psi). As a part of this project, we had to complete a directional survey. Surveys can be completed using magnetic or mechanical-based tools that are lowered into the wellbore or drill string periodically throughout the drilling process. Regardless of the mechanism used, the tool provides three key parameters:

Total length — the sum of the distances between all survey points
Inclination — angle of wellbore between survey points where 0° is vertical and 90° is horizontal
Azimuth — the direction of the wellbore relative to north (0°-360°)

From these values and with the help of a little trigonometry, a well trace can be calculated and plotted in x, y, z space. Why is this important? A few reasons. The first is regulatory — governmental agencies tasked with monitoring the oil and gas industry require this information. The second reason gets back to the idea mentioned above that there is almost always a difference between the well plan and the drilling execution and, quite frankly, it is important to know where these wells actually are.

And now, with the Well Trace Add-in for JMP, it has never been easier! Whether you need to quickly visualize several well bores initiated from the same platform or multiple completion stages from a single wellbore, the Well Trace Add-in can accommodate.

Well trace of GC045-B005 drilled by Mobil Oil Exploration and spudded on June 1996 (t) and a fictitious multi-stage completion of a single well (b).

Using the Average Angle method, the add-in will calculate the Delta East, Delta North, and Total Vertical Depth (TVD) and generate a visual trace of the well using the input values of Measured Depth, Inclination, and Azimuth acquired through a directional survey. Additionally, the optional Well Name and Color By inputs allows you to plot multiple wells from a single borehole and/or color multiple completions. You can choose to work in feet or meters and degrees or radians. Upon plotting the well trace, you can add your preferred units to the plot with a simple button-click. This add-In creates the well trace using the Scatterplot 3D platform, providing you with additional options found in that JMP platform.

Average angle calculation method (courtesy of rigworker.com)

The add-in is a great resource for our oil and gas, environmental, and hydrologically inclined customers. To learn more about this add-in and download a copy, please click here.

A look at the Well Trace Add-in inputs.

A cat named Chabi takes over a group chat

sabrinafong8 — Fri, 23 Jul 2021 15:28:19 GMT

Recently, a cat named Chabi decided to stay at my grandparents house. The cat has been the center of attention in my family's Whatsapp chat group.

I wanted to analyze how often his name was mentioned in the chat group. So did a quick analysis using Text Explorer in JMP and generated a Word Cloud. (I did this with permission of my family, of course!)

The word "Chabi" was mentioned 259 times this year alone, along with typical Malaysian grammatical fragments or slang words such as "AH" , "ALSO", "ONE", "YA" and "LA".

Have JMP? Try to analyze something fun today!

Greenhouse Gas Emissions Episode 3: Facts on renewable energy transition with simple graphing in JMP

Valerie_Nedbal — Mon, 20 Sep 2021 19:17:37 GMT

The first episode in this series on Greenhouse Gas Emissions focused on temperature deviation. In the second episode, I discussed trends on nuclear energy trends worldwide. Particularly interesting was the dismantling of nuclear reactors in Germany since 2011 following Japan’s Fukushima Daiichi nuclear disaster. In this episode, I illustrate the renewable energy transition trends in Europe and globally.

After deciding to shut down nuclear reactors, Germany aimed to close the gap in production with energy transition (renewable energy), also called “Energiewende.” Germany invested greatly in solar and wind energy. Using data from Wikipedia that examined solar energy trends over the last 15 years in the EU, I could clearly see that Germany has been the leader in the EU since 2011.

Wind power continues to be a growing trend. Since 1998, Germany constantly invested in building new wind parks or wind farms. Using data from Wikipedia, the graph below demonstrates this trend very nicely, showing Germany in the lead, followed by Spain, UK and France.

This trend not only exists in Europe but also worldwide. Data from ourworldindata.org nicely shows this trend extending to every continent. I have to confess that I did not show hydro or biomass and geothermal energy generation. In fact, in some countries (such as Iceland), those renewable energies are the most predominant. In this snapshot of worldwide renewable energy generation since 1965, we actually see that hydro energy generation is the worldwide leader in renewable energy generation, followed by wind power.

Finally, how much of our primary energy generation comes from renewables? The chart below shows the overall share of primary energy consumption that comes from renewable technologies – the combination of hydropower, solar, wind, geothermal, biofuels and other. The top countries are Iceland and Norway, with a respective mean of share of 59.9% and 66.5%. By hovering over those countries, we can clearly see the trend.

It also tells me that Germany still has a long way to go. Its percentage of primary renewable energy consumption was 17.5% in 2019 (using the Data Filter on year), which is far behind the top two countries: Iceland (79%) and Norway (66%).

Finally, it was interesting to determine the yearly overall trend of the percentage of primary renewable energy. Are there countries with a steady increase or decrease of primary renewable energy consumption? I put some indicators in the data table to make it easier to read, such as:

Colorizing the years from red (1966) to black to green (2018).
Marking the year 1965 with a black “-“ minus sign.
Marking the year 2019 with a black “^” caret sign.

Displaying the data in the Graph Builder as shown below, I could clearly see many countries with positive trends (with the caret on the top); however it was a little hard to distinguish countries with a positive versus negative trend.

Therefore, I did a final analysis, splitting the initial data to make a column for each country. I further analyzed the data with a multivariate analysis to obtain the Pearson correlation coefficients from the percentage of renewable energy consumption of all countries based on the year. A positive trend has a positive correlation coefficient, while a negative trend has a negative correlation coefficient.

I generated a new data table with the Pearson correlation coefficients and joined it to the initial data set. To get clusters of positively versus negatively evolving countries, I used the binning formula (new in JMP 16) that allows the correlation coefficients to be binned into four different groups: -1 to -0.5; -0.5 to 0; 0 to 0.5; 0.5 to 1.

I can now use this cluster column as a Local Data Filter to view positive or negative trends of primary renewable energy consumption. Below is an example of countries with positive renewable energy consumption trends.

I was very happy to see that Germany, where I live, is one of the countries in the category of very positive renewable energy consumption trends.

In some wealthy countries, like in Switzerland and New Zealand, production of renewable energy varied and sometimes decreased (see below). In Switzerland, based on the estimated mean production level, hydropower still accounted for almost 90% of domestic electricity production at the beginning of the 1970s, but this figure fell to around 60% by 1985 (following the commissioning of Switzerland's nuclear power plants) and is now around 56%. In Switzerland nowadays, electricity is mainly generated by hydropower (59.9%), nuclear power (33.5%) and conventional thermal power plants (2.3%, non-renewable).

New Zealand has the third-highest renewable primary energy supply after Iceland and Norway. New Zealand was an early adopter of several alternative energy technologies, particularly hydroelectricity and geothermal energy. It has achieved a level of 60% of total electricity generation from such sources. As hydro energy is largely impacted by rainfall, the amount of energy produced may vary from year to year, hence the trending up over time. In recent years, New Zealand’s development of renewables has lagged that of other countries, particularly in wind power. However, the number of wind and geothermal developments began to increase rapidly from mid-2000s forward, mainly due to declining costs of renewable technologies and the downgrading of natural gas reserves. Total solar energy generation still remains a small proportion of total primary renewable energy.

All the charts in this blog series have been published on JMP Public. I encourage you to go have a look!

You access all the posts in this series here.

Greenhouse Gas Emissions Episode 2: Facts on nuclear energy trends with simple graphing in JMP

Valerie_Nedbal — Mon, 20 Sep 2021 19:14:44 GMT

In the first episode in this blog series on greenhouse gas emissions, I discussed trends of global warming and the reduction of emissions over the last 25 years.

Which led me to ask: What about the impact of nuclear energy on renewable energy trends both in Europe and globally? (Renewable energy trends will be discussed in the third episode in this Greenhouse Gas Emissions series.) First, I looked at Europe, and I found an interesting data from Wikipedia on power stations that are in service, permanently shut down and under construction.

I used the Internet Open function in JMP to download the three data sets and concatenated them. Worldwide, Europe has the greatest number of nuclear energy plants, with 64 plants in service, two under construction and 13 shut down. By hovering over on the bar chart, specifically over the 13 plants shut down in Europe, I see that seven nuclear power stations have been shut down in Germany. In fact, Chancellor Angela Merkel’s coalition announced on May 2011 that Germany’s 17 nuclear plants will shut down by 2022, in a policy reversal following the disaster at Japan’s Fukushima Daiichi nuclear plant.

Luckily, geographical coordinates of all the nuclear power stations were included in the data table so I could nicely display the locations of all the plants independent of their status (green dots represent plants in service, black dots are plants permanently shut down, and red triangles are plants under construction). When I zoomed in on Europe, I could see most of the nuclear plants that have shut down are in Germany (see graphs below).

How old are the operational nuclear power stations? I found a data set on the IAEA PRIS website, representing the number of operational reactors by age and the total net electrical capacity (MW). In the chart below, the size of the dots reflects the number of reactors, and the color reflects the total net electrical capacity. The chart clearly indicates that most operational nuclear power stations are 30-40 years old.

All the charts in this post have been published on JMP Public. You can interact with them there.

Greenhouse Gas Emissions Episode 1: Facts on global warming and energy transition with simple graphing in JMP

Valerie_Nedbal — Tue, 19 Oct 2021 19:04:13 GMT

Worldwide global warming is a fact. It is directly attributable to human activity, specifically to our burning of fossil fuels such as coal, oil, gasoline, and natural gas, resulting in the greenhouse effect. The largest sources of greenhouse gases are transportation, electricity production and industrial activity. Curbing dangerous climate change requires very deep cuts in emissions, as well as the use of alternatives to fossil fuels. In fact, countries around the world have committed to lower their emissions as part of the 2015 Paris Climate Agreement.

With the help of publicly available data, I used different analysis techniques in JMP to find evidence of how countries around the globe are working on their energy transitions to achieve their commitments. If you want to follow along, I have attached my findings and data to this post. This is the first of a series of three episodes on greenhouse gas emissions; check back tomorrow and the following day for episodes 2 and 3.

The first thing I looked at was how temperature deviation has progressed year after year back to 1880. I used the Global Temperature Times Series data from the Datahub.io, a collection of thousands of data sets that are free to use. This temperature data included the global component of Climate at a Glance (GCAG) and the GISS Surface Temperature (GISTEMP) from 1880-2016.

I used the cumulative sum (CUSUM) control charts in JMP, which are useful for detecting shifts that occur over time, such as gradual drift. In my case, I was interested in detecting shifts of temperature anomalies over time. I used mean temperature deviation as my Y, year as my X and GCAG and GISTEMP as a By Variable. The two vertical lines on the CUSUM chart below indicated that shifts in temperature occurred in 1940 and 1977. Below is a CUSUM chart representation of mean temperature source GCAG (identical result was obtained with GISTEMP).

Using the same data set in Graph Builder in JMP, we can clearly observe temperature anomalies occurring in the 1940s and really picking up in 1970 until now.

But what caused the warming in the 1940s and the subsequent cooling? Although humans were not burning very large amounts of fossil fuels or emitting large amounts of carbon dioxide (CO2) in the early 20th century relative to later in the century, CO2 emissions were non-negligible and did play a role in the early century warming. Actually, CO2 and increased solar activity played the largest roles at that time, but other factors played a part as well. This warming period was followed by a period of slow or no warming until 1970s, which may have been partly caused by aerosol cooling.

The main culprit is likely to have been an increase in sulphate aerosols, which reflect incoming solar energy back into space and lead to cooling. This increase was the result of two sets of events:

Industrial activities picked up following World War II, which, in the absence of pollution control measures, led to a rise in aerosols in the lower atmosphere.
A number of volcanic eruptions released large amounts of aerosols in the upper atmosphere.

The introduction of pollution control measures reduced the emission of sulphate aerosols. Eventually, the cumulative effect of increased greenhouse gases started to dominate in the 1970s, and warming resumed. One final point: It should be noted that in 1945 the way that sea temperatures were measured changed, leading to biased measurements and to a substantial drop in apparent temperature. To learn more about this, see the references at the end of this post.

The chart below showcases the steady, worldwide increase of CO2 in the atmosphere since 1958. A time series of the average CO2 released into the atmosphere is on the left Y axis, and annual CO2 atmospheric growth rate is represented by red points and the blue bar chart scaled on the right Y axis.

But what has been happening in Europe more recently? I found more recent data and other greenhouse gas emissions on the European Commission Eurostat website. In particular, the data set on air emissions in European countries from 1994-2009 was quite interesting. Was there any decrease in the most abundant greenhouse gas emissions, such as methane (CH4), carbon dioxide (CO2), nitrous oxide (N2O) and others in more recent years? The answer was “yes”! The three area graphs from Graph Builder shown below demonstrate the decrease of the three most abundant greenhouse gas emissions in Europe over the last decade, with the most significant being methane emission (CH4).

All the charts in this blog post have been published on JMP Public. Stop by there and interact with the graphs.

Additional reading:

Edit: I am performing a Khoros test for a support case that is open with us. By only submitting this for review, the test should be complete. Please reject/dismiss this change. Thank you.

Illuminating results from a failed analysis

roger_alford — Fri, 30 Jul 2021 18:09:46 GMT

A couple of years ago, I began to look at how I could automate the outdoor flood lights on my house. I wanted to schedule when they would turn on and off. There were two primary reasons.

First, changing them is a real pain. My backyard slopes down quite a bit and even though I have an 11-foot pole, it’s still only long enough to reach two of the bulbs. For the other four, I have to use an extension ladder. And though I’ve gotten more accustomed to it over the years, it still makes me nervous every time I climb up.

Second, my wife and I love to travel. Not wanting the house to be dark while we’re gone, we’d leave the spotlights on 24/7. Which made it a little obvious that we weren’t home. It also required me to change the bulbs more often (sometimes as soon as we returned). A two-pack of bulbs cost $9.99, which certainly adds up over time.

For curiosity’s sake, I started keeping a diary of how often I changed each bulb. My initial plan was to compare this information to the data collected after I implemented a home automation system. The objectives of my analyses were as follows:

How much longer would the bulbs last by having them automated?
What were the resulting cost savings?

Let's take a look at how my plan turned out....

Before Home Automation

Prior to implementing automation, I collected 20 months of data. There are six bulbs on my house (three fixtures of two bulbs each). Two bulbs for the front yard, and four on the back. I used timeanddate.com to calculate the number of days in each interval.

I teamed up with Julian Parris () to do the analyses in JMP. The first thing we noticed was that I changed FrontBulb1 the most often. If you look at the scatterplot below, you will see that four of the intervals last only 40 days. There doesn't seem to be any predictable reason for this, because FrontBulb2, in the same fixture, is almost a reverse image. And while most of the intervals for FrontBulb1 range from 40 to 94 days, we have one outlier of 169 days. In fact, many of the bulbs have a fairly consistent range of 40 to 100 days, with at least one outlier. BackRight1 had the highest interval of all at 214 days.

If we look at the median life for each bulb, we see that FrontBulb1, BackRight1 and BackRight2 all have a tight range of 50 to 56 days. This isn't surprising for BackRight1 & 2, as they're both in the same fixture. However, FrontBulb2 has a significantly higher median average of 72 days. BackLeft1 and BackLeft2 are both in the higher range of 77 to 86 days, with BackLeft2 having the highest median interval of all.

So, in implementing home automation, my goal was to improve bulb life from a general range of 40 to 100 days. And to then compute the resulting cost savings.

After Home Automation

In May of 2019, just before our summer vacation, we had our automation system installed. I programmed the outdoor spotlights to turn on at sunrise and turn off at sunset. To get a clean data set for comparison, I changed all six bulbs on the house just before we left in June (the final interval for each bulb was eliminated from the data set above). I was very excited to start collecting the new data and eventually compare it to the previous data set.

However, I ran into a couple of unexpected hiccups:

When I purchased the six new bulbs, the standard spotlights were no longer available. The store only carried new LED bulbs.
The LED bulbs were slightly more expensive ($14.99 for two as opposed to $9.99).

These differences would prevent an apples-to-apples comparison, so I’d have to adjust accordingly. But that wasn’t the half of it.

When we returned from our trip, I was happy to see that every bulb was still burning. I was sure I’d have to change one after another month or two, but that wasn’t the case either. My automation system was working great!

The Unexpected Results

But then something else happened. Or rather, didn’t happen.

Not a single bulb burned out after four months. Or six. Or eight.

I eventually realized that the automation hadn’t made any difference at all. Or at least not one that I could measure. It was the new LED bulbs.

It has now been over two years (longer than my original data set) since I changed all the bulbs. And they are still burning. I haven’t changed a single one.

Obviously, my cost savings in bulbs has been significant. Despite the higher price, the LED bulbs are far more efficient.

When you factor in the costs of the new switches and installation, I've basically broken even. But I can see where I’ll eventually come out ahead. And the automated switches gave me something worth even more: peace of mind.

In the end, I achieved my goal of having to change the bulbs less often. I just got there a completely different way than I expected.

My original analysis may not have worked as planned, but the results were still rather illuminating.

You can explore the interactive results of this analysis at JMP Public.

Automating and extending JMP Clinical: The preamble

Chris_Kirchberg — Wed, 07 Jul 2021 19:28:17 GMT

JMP Clinical was released 11 years ago specifically for the analysis and review of ongoing clinical trials centered on standardized and structured data. The intent was that a reviewer would receive data for review in regular intervals (every week or every month) to look for safety and/or efficacy signals using pre-programmed reports, enabling the reviewer to self-serve and dig deeper into the data on-demand. The hope was to expedite the review process as well as uncover anomalies that might otherwise be missed when looking at static tables, listings, and figures.

In addition, a goal of the software was to alleviate the burden placed on programming teams who produce the requested data and standardized reports for the reviewers. This required a lot of time to run programs or edit programs/scripts/macro to accommodate the reviewers' needs and on a frequent basis. Programming and biostatistics groups typically want to focus on data and reports that are unique to the study instead of the usual table or graph that is pretty consistent from study to study, allowing them to give more robust and undistracted attention to the study-specific anomalies that require more statistical or algorithmic knowledge.

For the most part, JMP Clinical met those goals unless we're talking about organizations that have many studies that need to be monitored and/or have frequent data updates. This puts a new burden on data management and biostatistics teams to update the studies within JMP Clinical or puts the burden on the reviewers who were not as proficient in the process or have the time. Also of interest was the ability to modify some of the reports within JMP Clinical and/or run the same review template on a scheduled basis. The demand for automation and extension of JMP Clinical’s capabilities became a frequent request.

Enter the APIs of JMP Clinical. Around JMP Clinical 6, we introduced the framework and infrastructure to make API-based access to common functions/processes programmatically possible. In JMP Clinical 7, the APIs were released and made available using the JMP Scripting Language (JSL). In version 8 (the latest one), more capabilities were released. This post does not go into the details of the development of the APIs. Instead, it focuses on the practical use of them so you can take advantage of them.

I'll share a series of posts on how to invoke and use the APIs. We start simple by introducing the basic syntax and their use. Then we progress into more sophisticated uses while using JSL. And finally, we combine the two categories of APIs invoked within some JSL.

For those not familiar with the terms Study, Review, Report and Review Template within the context of JMP Clinical, please see this link to become acquainted. Also, those not familiar with JSL should read and learn more at this link. Knowledge of both is required to understand the use of the APIs.

For this post, let’s start with the two main categories and what is possible. The two main categories are: Study Management related functions (JMPClinicalStudyManagerAPI) and the Review related functions (JMPClinicalReviewAPI). The idea is to replicate what can be done within the user interface (mainly the button clicks) where user a decision is not needed (like refreshing the metadata for Studies with new data or opening a Review Template and running it for a specific Study). Below are a couple of images that highlight buttons that relate to the Study Management section and the Review section.

Study Management Functions that Relate to JMPClinicalStudyMangerAPI

Review/Review Builder Functions that Related to JMPClinicalReviewAPI

The API is meant to take routine tasks and automate them, like refreshing studies with new metadata or to add/remove content from a Report within a Review Template. Here is a link for the JMPClinicalStudyManagerAPI functions and another link for the JMPClinicalReviewAPI functions for reference.

So, what can this JMPClinicalStudyMangerAPI API do for us? It provides us with two sets of functions: get functions (ones that get information about the studies) and manipulate functions (ones that add, remove or update studies). Here is a table of what is possible with the basic syntax to call that function. The colon (:) separates the API call and the function call. Some of the functions have input and/or returns. See the above links for details on what is expected as an input and return.

Study Management API

Get Information Functions

Manipulate Study Functions

Get list of Study names:

JMPClinicalStudyManagerAPI:getStudyList

Add Study:

JMPClinicalStudyManagerAPI:addStudy

Get SDTM folder path:

JMPClinicalStudyManagerAPI:getSDTMFolder

Refresh Study metadata:

JMPClinicalStudyManagerAPI:refreshStudies

Get ADaM folder path:

JMPClinicalStudyManagerAPI:getADaMFolder

Delete Study:

JMPClinicalStudyManagerAPI:deleteStudies

Get Date Study was added:

JMPClinicalStudyManagerAPI:getInitialDate

Update Study with a new Snapshot:

JMPClinicalStudyManagerAPI:updateSnapshot

Get Date Study was lasted updated:

JMPClinicalStudyManagerAPI:getLastUpdatedDate

Change Study data folders:

JMPClinicalStudyManagerAPI:changeFolders

Get Snapshot Enabled Status:

JMPClinicalStudyManagerAPI:getEnableSnapshot

Pre-compute Patient Profile data:

JMPClinicalStudyManagerAPI:precomputeProfileData

Get Snapshot Number:

JMPClinicalStudyManagerAPI:getSnapshotNumber

Get list of domains recognized:

JMPClinicalStudyManagerAPI:getDomainList

Get file size on disk:

JMPClinicalStudyManagerAPI:getSizeOnDisk

Get Windows OS user ID who created/added Study:

JMPClinicalStudyManagerAPI:getCreatedBy

Get Windows OS user ID who last updated Study:

JMPClinicalStudyManagerAPI:getLastUpdatedBy

Now let’s look at the second category, Review functions. This one also has a set of get and manipulate functions but for Reviews and Reports.

Review API

Get Information Functions

Manipulate Report/Review Functions

Get a reference to the Review Builder:

JMPClinicalReviewAPI:getReviewBuilder

Close Review Builder:

JMPClinicalReviewAPI:closeReviewBuilder

Get the Review Viewer/Builder ID and Title:

JMPClinicalReviewAPI:getReviewViewerIDsAndTitles

Open Review Template:

JMPClinicalReviewAPI:openReviewTemplate

Get a reference to the Review Viewer:

JMPClinicalReviewAPI:getReviewViewer

Add a Report to Review Builder:

JMPClinicalReviewAPI:addReport

Get Table of Contents of the Review Viewer/Builder:

JMPClinicalReviewAPI:getReviewTableOfContents

Remove Report form Review Builder by Title:

JMPClinicalReviewAPI:removeReportByReportTitle

Get a list of Report IDs:

JMPClinicalReviewAPI:getReportIDList

Run Report in Review Builder by Title:

JMPClinicalReviewAPI:runReportByReportTitle

Get a list of Report Titles:

JMPClinicalReviewAPI:getReportTitleList

Run all Reports in Review Builder:

JMPClinicalReviewAPI:runAllReports

Get a reference to a Report by ID:

JMPClinicalReviewAPI:getReportByID

Save Review Builder contents as a Review:

JMPClinicalReviewAPI:saveReview

Get a reference to a Report by Title:

JMPClinicalReviewAPI:getReportByTitle

Open a Review in the Review Viewer:

JMPClinicalReviewAPI:openReview

Get Report Details by Title:

JMPClinicalReviewAPI:getReportDetailsByTitle

Insert Section into a Report:

JMPClinicalReviewAPI:insertSectionIntoReportByTitleAndSectionName

Get a reference to the Report Results:

JMPClinicalReviewAPI:getReportResultsByTitle

Remove Section from a Report:

JMPClinicalReviewAPI:removeSectionFromReportByTitleAndSectionName

Get a reference to the Report Section:

JMPClinicalReviewAPI:getReportSectionByTitleAndSectionName

Add to end of Report Section:

JMPClinicalReviewAPI:appendDisplayObjectToReportSectionByTitleAndSectionName

Get list of data tables used in Report:

JMPClinicalReviewAPI:getDataTablesForSectionByTitleAndSectionName

Add to beginning of Report Section:

JMPClinicalReviewAPI:prependDisplayObjectToReportSectionByTitleAndSectionName

So now we have the different functions and what they can do listed. Now it is time to show the basic syntax of their use. The basic form is this:

JMPClinical<Category>API:<function> ({argument(s)}, {namespace}, completeListenerFunction());

I have already introduced the basic call to the Study Management related API functions above when I referred to JMPClinicalStudyManagerAPI. So, if I want to refresh the metadata of the registered studies, it will look something like this:

JMPClinicalStudyManager:refreshStudies ({studyList},{},{});

We will dissect this more and talk about common uses in the next set of posts. For now, I wanted to introduce what is available and how to generally invoke it within a JSL script. The next post will be about an example of the Study Management API, talking more about namespace and completeListener parts of the API Call and start to wrap it in JSL as well as use JSL within it.

Until the next post, take care!

‘Stuck in an Excel nightmare’

J_Marquardt — Thu, 01 Jul 2021 16:00:00 GMT

Anders Reinhardt drives business intelligence modernization at Coloplast, which includes storytelling with data and giving employees self-service access to data.Anders Reinhardt, Director of Business Intelligence at Coloplast, talks about the benefits of using a more visual tool like JMP. You can watch the full conversation with Anders and his fellow panelists at your leisure.

“We were using Excel as a primary source for distributing data, and we tended to communicate with data via huge lists of information. I think we’ve managed to turn that around and say, ‘You have to build a story around your data.’

“Sending a list to somebody or a huge table with a lot of information is not necessarily telling a story, or it’s trying to tell a lot of stories at the same time.

“Even something as simple as saying, ‘How would you actually segment your table? Would you show all the companies in a geographical order? And if you were, why?’

“You might have Greece on top of the list; we don’t sell anything there. And you would have to somewhere in the table structure find the UK…and maybe it’s more important that the market is performing, relatively at least to our performance. We’ve worked on lots of stuff like that.

“It has benefited us to go away from Excel and toward more visual tools. And thinking a lot more about ‘How do I present the message to the audience?’ Excel is a horrible tool for a lot of people.

“We were not thinking about our recipients either. Our message ended up not actually going across. I don’t think we got a lot out of it.

“We’ve set the data more free, given people easier tools to work with. And now we see a much wider engagement than when everything was in Excel spreadsheets.”

Pride Month: Seeing diversity as the gift that it is

anne_milley — Wed, 30 Jun 2021 14:16:16 GMT

Diversity and variation — what is the difference? We can think of diversity as the state of variation within a larger population. Statistics is used for many things, one of which is to understand variation and the sources that cause it.

Some variation comes from the measurement process itself. Measurement and other sources of variation are important to understand so that we can better understand our world and draw useful inferences from our data.

Because JMP is used by many in industrial process applications, we often talk about understanding and controlling sources of variation to realize more consistent processes and outcomes. But when we more broadly consider variation in our world, the goals may simply be to better understand it and appreciate it. For instance, in soil composition, we have yet to understand or fully appreciate its variation with so much in the microflora group yet to be classified.

More relevant to this post, what variation do we understand and appreciate in sexuality or gender? What is gender?

As a child, I leafed through many of my parents’ scientific books. With my father a chemistry professor and my mother a nurse, there were a variety of topics to explore. I will never forget learning about the fact that some people are born with atypical genitalia that seem to be in between what we consider to be male and female. A different term was used then, but today the term is intersex.

It is the term used to express the natural biological variation that occurs in the gender spectrum (it occurs in other mammals as well). The Intersex Society of North America (ISNA) defines it as “a general term used for a variety of conditions in which a person is born with a reproductive or sexual anatomy that doesn’t seem to fit the typical definitions of female or male.”

Several factors are motivating me to write this blog post. I recently facilitated a fascinating discussion for JMP Discovery Summit with these esteemed award-winning individuals:

Statistician and author David J. Hand, OBE FBA MMC, Emeritus Professor of Mathematics at Imperial College London and author of the recent Dark Data: Why What You Don’t Know Matters.
Feminist author, campaigner, journalist, and activist Caroline Criado Perez, OBE and author of Invisible Women: Exposing Data Bias in a World Designed for Men.

The examples in Criado Perez’s book are various examples of dark data, thus the overlap (by the way, I highly recommend both books!). Given recent examples of gender bias and dark data related to COVID, we explored other topics, one of which was “one-size-fits-all medicine.” How should the biological variation that occurs in nature related to what we term “sex” be taken into account in the development of drugs and vaccines and their dosage?

Another set of motivating factors is that it’s Pride Month. Inclusion is important to me, since we are all unique and there is value in that; I want to live in a world where people can be comfortable in their own skins for who they are.

While intersex is one aspect of sexual identity and LGBTQ+ is much broader, intersex does shine a light on the wider gender spectrum, reminding us that we shouldn’t constrain our thinking to binary male/female. I am grateful to the ISNA for providing such an informative website and for citing the research of Anne Fausto-Sterling, author of the recently updated Sexing the Body: Gender Politics and the Construction of Sexuality(which I am reading and it is most interesting!) and Alice Domurat Dreger, author of the article, Ambiguous Sex—or Ambivalent Medicine? Ethical Issues in the Treatment of Intersexuality. The ISNA includes some summary statistics from Fausto-Sterling for her research into the approximate number of sex variations and has made these approximations from her research available on its web site. I took the liberty of creating a JMP graph:

In this graph, we are looking at some nonbinary gender variation — something most of us have not seen. Given that the intent of Fausto-Sterling’s research was to review the medical literature from 1955 to 1998 (in hopes of estimating the frequency of sex variations), and that there is still no widespread agreement on what counts as intersex, these conditions are likely underrepresented.

What about other sex variations that we don’t see or that may not have a medical label? Or attempted measurements to determine if there are more as yet unknown variations that ultimately affect a person’s sexual identity? While science evolves, what we were taught many years ago may not have. According to ISNA:

”So now we have genes on the Y that can turn females with XX chromosomes into males and genes on the X that can turn males with XY chromosomes into females. . . wow! Maleness and femaleness are NOT determined by having an X or a Y, since switching a couple of genes around can turn things upside down.”

Wow is right! The ISNA cites that in the US there are some 7,500 men without a Y chromosome and an equal number of women who have XY instead of XX.

This Gallup poll estimates that lesbian, gay, bisexual or transgender identification has risen to 5.6% in the US (percentages total more than 100% because respondents could choose more than one category):

In Thomas Page McBee’s brilliant New York Times article, My Decade in American Manhood, he writes, “We are, all of us, in a constant stage of negotiation with the political and the cultural forces attempting to shape us into simple, translatable packages.” This affords us the opportunity to further question how our ideas about gender have been shaped.

Variation is ubiquitous, even if we can’t always see it. Diversity is a gift. In recognizing it as such, may it help us more greatly appreciate the gift of our many differences.

How to be strategic about quality

anne_milley — Thu, 24 Jun 2021 17:45:28 GMT

There were many excellent takeaways in a recent episode of Statistically Speaking focused on quality, featuring Martha Gardner, Chief Engineer of Quality at GE Aviation; Wende Wilson, Lead Data Scientist of Schweizer Engineering Laboratories; and Bernard Goguelet, Senior Consultant at Deloitte. Martha’s opening plenary on Becoming a Strategic Statistician was brilliant, setting the stage for the engaging discussion that followed.

Though the topic for the event was quality, the strategic perspective on statistical problem solving was the overriding theme.

These quality leaders recognize that quality is a strategic pursuit; when it’s done well, it can affect the entire organization because it requires quality data, processes, technology, and most critically, people, to create value.

Other tips and advice included the importance of asking the right question, effective ways to share results, and ways to persuade others to upskill so they can make the scientific discoveries they love even faster.

If you missed it, you can watch the plenary and panel discussion for this episode on-demand.

Gain accuracy and precision for model estimates from limits of detection (LOD) control

statpharmer — Mon, 26 Jul 2021 19:05:35 GMT

Researchers commonly use complex laboratory methods to evaluate critical quality attributes (CQA) of drugs in development and throughout the product lifecycle. Instruments have validated ranges for obtaining data, and researchers often get results that are beyond the limits of detection (LOD). If you're a technical person running predictive models that include responses with limits on the range of values that can be obtained, this post is for you. JMP Pro now includes an easily deployed technique to appropriately deal with limits of detection within statistical models and reduce the risk for error.

What is the practical value of utilizing the new LOD controls in JMP Pro?

Let's consider an example: We are planning a set of experiments to determine the maximum amount of a solid impurity that can be extracted and measured with a laboratory method utilizing three inputs. The measurement method includes a lower detection limit of 1 unit. We've tried different techniques for dealing with LOD in the past, including using missing values, creating a value that is very small but not zero to allow for a logarithm to be used, imposing the limit for the observations affected by the LOD, and using the LOD control included in JMP Pro 16.

We use a generalized linear model with a log normal distribution utilizing forward selection to model the amount of impurity that can be extracted and measured. Running the model without LOD control results in predictions that are highly suspect to include significant error.

The maximized model using missing values provides an estimate of 137.8, which is hugely beyond the highest impurity amount of 34.7 realized in the experimentation. Note the extremely wide spread of the 95% confidence interval limits.

The maximized model using values that are close to zero provides the unrealistic estimate of 472.4.

The maximized model using the 1-unit low limit seems to underestimate the optimum amount of impurity at 16.6, which is roughly half of the maximum amount realized during the experimentation.

We use LOD controls in the final model. The optimum amount of impurity that can be measured is estimated to be 36.7. Note that the realistic estimate comes from a model that includes all three method inputs. There are local maxima for each of the three method inputs, which makes sense to subject matter experts.

The comparison clearly illustrates how LOD controls mitigate error in prediction estimates for models. It is always a best practice to create confirmation runs in the space surrounding the amounts for the inputs identified as optimum.

What are limits of detection?

Instruments typically produce values from a matrix without an analyte, which are known as blank readings. A group of blank readings form a distribution of “noise” that is inherent with the measurement system. This “noise” is the random variability that you can expect from taking multiple readings of a homogenous sample tested by the instrument.

The limit of detection is the smallest concentration/quantity that is statistically different than a blank reading[1].

Smallest quantity = mean of blank measures + (confidence level x standard deviation of blank measures)

Using a 99% level of confidence, the limit of detection is the mean of the smallest signal that is roughly 3 standard deviations (blank) above the mean of the blanks.

A limit of quantification is often utilized to mitigate a false negative result (that is, we assume the measure is outside of the limit when an actual measure is available).

Some measurements have bi-lateral limits. Load cells that measure force typically have a zone of accuracy within bi-lateral limits.

Scientists and engineers deal with many different types of limits of detection including: Instrument detection limit, method detection limit, limit of quantification, and practical quantification limit. Regardless of the type of limit used, it is up to the scientist or engineer to identify the cut-off values for censoring with LOD to meet the goals of experimentation.

Why do limits of detection affect model estimates?

Parameter estimates are typically made with the cumulative distribution function (CDF). With limits present, the result is a shifted CDF and altered parameters. The plot below illustrates the effect of an LOD for a symmetric distribution.

The shift of the CDF due to limits within measurements is especially profound for skewed distributions, which is seen below as a significant amount of potential data that is not included in the CDF that creates a big shift in parameters.

When we use limits of detection appropriately as censor values, estimates are made from the probability density function. The shape of the PDF mitigates the shift of the parameters since the missing portion can be Parameters are more accurate yielding realistic model estimates.

How can you utilize LOD controls in JMP Pro?

Let's consider a similar scenario: We are planning a set experiments to find out the maximum amount of a solid impurity that we can extract and measure using a laboratory method. We use the DOE> Custom Design menu option to set up the structure of the experiments. JMP 16 includes the ability to add detection limits as part of the design process as shown below.

Another way to set up LOD controls is to add them to the column properties.

JMP Pro will automatically control for LOD without you needing to create additional columns required to censor appropriately for the limits within models.

Realize practical value by utilizing LOD controls

Use of this LOD controls in JMP Pro 16 yields significant improvements in the precision and accuracy of model estimates to enhance quality by design (QbD) efforts. Technical people now have a great new tool to deal with measurement methods that are known to have limits.

Reference

[1] IUPAC. Compendium of Chemical Terminology, 2nd ed. (the "Gold Book"). Compiled by A. D. McNaught and A. Wilkinson. Blackwell Scientific Publications, Oxford (1997). Online version (2019-) created by S. J. Chalk. ISBN 0-9678550-9-8. https://doi.org/10.1351/goldbook.

How happy are we here in Europe? A data visualization exercise in JMP

eurvan1 — Thu, 10 Jun 2021 16:39:01 GMT

I’m a curious person, and I wanted to analyze survey data coming from the European Social Survey. Especially, as I’m based in Germany, I was interested to see how Germany was rated compared to the rest of Europe. The European Social Survey (ESS) is an academically driven cross-national survey that has been conducted across Europe since its establishment in 2001. Every two years, face-to-face interviews are conducted with newly selected, cross-sectional samples. The survey measures the attitudes, beliefs and behavior patterns of a diverse population in more than 30 nations.

The survey began gathering data in 2002 and continued every two years until 2018. The data ultimately was compiled into nine different data sets. The ESS questionnaire consists of a collection of questions that can be classified into two main parts: a core section that focuses on a range of different themes that are largely the same in each round and a rotating section that is dedicated to specific themes and can differ from round to round. The ESS data is available free of charge for non-commercial use and can be downloaded from this website after a short registration.

These data sets can be downloaded either as SAS, SPSS or STATA. I’ve chosen SPSS. JMP is very versatile in opening data coming from different sources, and SPSS is one of them. After concatenating and cleaning the data, including recoding and removing nearly empty columns, I got to the fun part: analyzing the data.

Since I was able to save the different data preparation steps in the script coming from the Action Recorder in the log view, which is new in JMP 16 (see Data Cleaning.jsl), it wouldn’t take much time if I needed to repeat the data preparation steps.

I first had a look at how many responses were obtained in each country during the different years. Graph Builder is the perfect platform to execute many different types of graphs. The graph below shows that Germany completed its survey every year with close to 3,000 respondents.

In each country, sets of questions were asked on politics, education, religion and life satisfaction. An interesting outcome was that a happiness trend in the different countries of Europe was revealed by asking such questions as “How happy are you?” and “How satisfied are you with life as a whole?” To answer these questions, I used different analysis techniques in Graph Builder and in multiple correspondence analysis.

Using geographic mapping of the countries involved in the survey and by coloring the question “How happy are you?” it was very clear to me that the happiest and most satisfied people live in Scandinavian countries and Switzerland (in dark red). The highest rated country was Iceland, with 8.27. In contrast, countries with the lowest scores are in Eastern European countries and Portugal (blue to dark blue). The lowest rated country was Ukraine with 5.79. Germany had an overall rating of 7.43, which is good!

By plotting survey question scores on the Y axis and Years on the X axis, and then changing the graph display to a mosaic plot, it was easy to see the trend of happiness year over year in the different countries. Using the Column Switcher, I could easily switch from one question to another and see the percentage of the scores, as seen below with Germany as the example. Interestingly, the trend over the years is an increase in happiness and life satisfaction as a whole, again which is good!!

How happy are you? 2002-2018

How satisfied are you with life as a whole? 2002-2018

Next, I wanted to verify my results using a multiple correspondence analysis (MCA) in JMP. MCA is an analysis technique for nominal categorical data, used to detect and represent underlying structures in a data set. It takes multiple categorical variables and seeks to identify any association between the levels of those variables. MCA is frequently used in survey analysis to identify question agreement. Therefore, this type of analysis was a perfect fit for my purposes. I used the question “How happy are you?” as my response and “Country” as my factor. The results are below. We can see the scores nicely distributed on the horizontal axes, from Extremely Happy on the left to Extremely Unhappy on the right. Again, it is no surprise to see the Scandinavian countries on left part of the graph and Eastern European countries on the right part. Germany was positioned on the left and scored between 7 and 8, which is the overall score spanning 16 years.

But is there a positive happiness trend for Germany over the years, as was observed in the mosaic plot? To check this, I recreated the multiple correspondence analysis from above but now used Year as a by variable so that I could split my analysis into the corresponding years. I saved the coordinates for X and Y from all nine years. I concatenated the different X and Y coordinates and used the final data set in a bubble plot to see the trend for each country year by year, and specifically, for Germany. I colored the dots: red dots for scores 7, 8, 9 (extremely happy) and the blue dot for Germany. The gray dots are for European countries other than Germany. When animating the bubble plot, the dots begin to move. The trend for Germany was particularly interesting. In 2002, the dot was close to 7, but year after year, the dot got closer to 8 and 9, which again is a good sign.

My findings were verified using different techniques in JMP. Many other questions could have been analyzed for different countries using these different techniques. All the graphs and results in this blog have been published on JMP Public: How happy are we here in Europe? | JMP Public

Analyzing spectral data: Multivariate methods and advanced pre-processing

Bill_Worley — Wed, 25 Aug 2021 14:04:50 GMT

A quick reminder of where we left off in Part 1 of this series on spectroscopic data: We were analyzing a popular spectroscopy data set by Martens et al. "Light Scattering and light absorbance separated by extended multiplicative signal correction. Application to near-infrared transmission analysis of powder mixtures" Analytical Chemistry 2003 Feb1;75(3):394-404. A few simple pre-processing steps allowed us to dramatically improve the signal-to-noise ratio in the data.

In this post, we use the same data to demonstrate the utility of the multivariate platforms in JMP. In spectral analysis, multivariate statistics are used extensively as one iteratively tries different pre-processing steps and assesses the impact on the data. We focus on unsupervised learning, or exploratory data analysis, at first. Then we demonstrate how the functional data explorer (FDE) can be used to build multivariate calibration models. Finally, we introduce a more advanced pre-processing method -- the extended multiplicative signal correction -- and show how this can further improve our multivariate calibration model.

Multivariate Methods

Principal Components Analysis

The first exploratory method we demonstrate is principal components analysis (PCA). Interestingly, when a PCA is performed on the raw spectra, the known subgroups can be identified in the score plot without the need for any pre-processing (Figure 1). Remember from Part 1 that the samples are mixtures of gluten and starch, and that the colors indicate the gluten fraction. While there is good separation between groups, there is still a considerable amount of variation for spectra with the same gluten fraction, which will cause problems for multivariate calibration later. This error improves considerably by the end of this post, after we find the optimal pre-processing workflow.

Figure 1. The score and loading plots for the PCA model. The loading plot indicates a high degree of correlation across wavelengths, as expected. The colors indicate the gluten fraction present in each sample from 0 (blue) to 1 (red). The eigenvalues indicate that variation in this data set can be explained with 2 principal components.

Model Driven Multivariate Control Chart

Typically, some type of outlier analysis is performed to identify and remove influential outliers from the model. JMP provides outlier analysis tools within the PCA platform. However, with JMP 15 came the Model Driven Multivariate Control Chart (MDMCC), which is a more full-featured tool for outlier analysis that allows for deeper investigation into root cause analysis and comparing spectra for differences.

Figure 2. The Model Driven Multivariate Control chart identifies two spectra (13 and 14) as outliers according to the T² and SPE charts. Mean contribution proportion plots demonstrate the wavelengths with largest contribution to the out of control signal.

For the spectra in this analysis, MDMCC identifies two spectra (13 and 14), as outliers based on their PCA derived T² and SPE charts (Figure 2). The T² chart indicates that these points are far from the data within the model plane, while the SPE chart indicates that the points are distant from the model plane. Another way to conceptualize this is that these spectra are 1) distant from the average spectra and 2) poor fits in the PCA model. Since both of these conditions are satisfied, the points are potentially influential outliers, meaning that they may pull the PCA model plane toward them when the model is fit. If these are truly erroneous data, there is potential for a poor PCA model. Note that there are other points besides 13 and 14 that are outliers in the T² chart, but not in the SPE chart. Since these points are distant from the average spectra, but still fit the PCA model, they are less likely to be influential outliers. To see a good illustration that distinguishes T² and SPE outliers, see Figure 4 of this paper.

To get a better sense of why the points 13 and 14 are outliers, we construct mean contribution proportion bar charts for the T² and SPE charts (Figure 3). These show the wavelengths with the largest contribution to the outlier status in both charts. You can also hover over individual bars to see univariate control charts for a specific wavelength.

For this example, it is helpful to plot the spectra in Graph Builder to get a better sense of why the observations are outliers (Figure 3). One noticeable anomaly is that these two spectra have abnormally large absorbances at longer wavelengths.

If we decide these outliers are suspect, it is easy to refit the model with the outliers removed. To do this, turn on Automatic Recalc in the MDMCC report, select the outlier observations in either the T2 or SPE control chart, and then use right click >> Rows >> Row Exclude.

Figure 3. Graph Builder view for spectra 13 and 14 as selected in MDMCC via dynamic linking. This graph shows how the two selected spectra could be perceived as outliers when compared directly to the rest of the data.

The Score Plot for MDMCC can also be used to compare subgroups (Figure 4). Group A is the top “red” subgroup and Group B is the bottom “blue” subgroup. The Relative Score Contribution Plot shows where there are significant differences between the two subgroups by wavelength.

Figure 4. Group comparison between spectra with 0% and 100% gluten. The wavelengths with largest absolute contribution in the bar chart show the largest difference between groups.

Functional Data Explorer

The PCA platform in JMP is a good general-purpose tool but was not designed for data with a functional form like spectra. Fortunately, in JMP Pro, there is a tool that is specifically designed for functional data. The Functional Data Explorer (FDE) platform in JMP Pro can be used for deeper investigation of spectral data. FDE is used to analyze curve data over some continuum, for NIR spectra the continuum is wavelength. Because all spectra are measured on the same equally spaced grid, we can fit a direct functional PCA model, which performs a functional PCA without fitting a basis function (Figure 5). Note that we did not exclude the outliers that we identified in the previous section (13 and 14) from the FDE model. While one might choose to exclude these outliers, we include them in the fit because in a later section we will compare to a FDE model that is fit to pre-processed data where these observations are no longer outliers (see Figure 14).

In Figure 5, the eigenvalues corresponding to the functional principal components (fPCs) show how much variation is explained by each fPC. In the case of this data, the variation is characterized by the first two fPCs, as was the case with PCA. You also see a mean curve associated with the fPCs. If your data tracks the mean function exactly, all fPCs would be zero. If the fPC value is positive, the shape of the function varies from the mean in a shape similar to the corresponding eigenfunction, while a negative fPC indicates the reverse effect. The eigenfunctions are comparable to a common plot used in the PCA analysis of spectral data called the spectral loadings plot, where the loadings are ordered by wavelength.

Figure 5. The direct functional PCA model fit summaries. Two functional principal components explain 100% of variation in the data.

The individual spectra are fit and the resulting score plot is shown in Figure 6. A handy feature in the functional principal components platform is the ability to hover over points in score plots and see graphlets for the corresponding spectra. This is very useful in this example, as the score plot organizes the data into subgroups of similar spectra, and then the hover help shows the differences in the functional form. This can be used to discover differences between subgroups without relying on any more sophisticated analyses.

Figure 6. The score plot for the FDE model fit. Differences between subgroups can be explored by hovering over points in the score plot, revealing the functional form of the spectra.

The fPC Profiler allows for easier interpretation of the score dimensions for the individual fPCs (Figure 7). Moving one of the fPC sliders shows the effect on the predicted function of moving along one of the fPC directions, with the other fPC values fixed.

Figure 7. Eigenfunctions (A) and the fPC Profiler (B-D) for the FDE model fit. The fPC profiler with the fPCs set to their mean values (B), fPC2 set to its minimum value (C), and fPC2 set to its maximum value (D). When the fPC2 value is positive, the shape of the spectra varies from the mean in a shape similar to the corresponding eigenfunction, while a negative fPC indicates the reverse effect.

Functional Design of Experiments (Multivariate Calibration)

Since these data are from a designed experiment, a Functional DOE analysis is relevant (Figure 8). The FDOE Profiler allows for the prediction of the functional form of spectra at starch and gluten fractions not observed in the experimental data. In spectra analysis, this is called a multivariate calibration model. Since starch and gluten are mixture proportions that sum to 1, we apply a mixture constraint in the Profiler.

Often it is desirable to build the inverse of this model, in which experimental factors (the constituent proportions, in this case) are predicted as a function of the spectra. Confusingly, this is also called multivariate calibration. For clarity, we adopt Brereton’s terminology and refer to this as inverse multivariate calibration. To fit these types of models, one can output the functional principal components and provide them to any predictive modeling platform in JMP. This enables one to fit flexible models, such as neural networks, to the function summaries. These models may be able to model nonlinearities and dependencies better than PLS. We will illustrate such an approach in the next post.

Figure 8. The functional DOE profiler. A linear constraint has been applied to starch and gluten so that they sum to 1. Spectra are predicted at experimental factor combinations not observed in the original data.

Hierarchical Clustering

Hierarchical clustering is another unsupervised machine learning tool popular in spectral analysis. The goal of hierarchical clustering is to identify subgroups in the data, where differences are small within clusters and large between clusters. In previous sections, we identified subgroups but did so in a subjective way. Hierarchical clustering provides an objective criterion for defining these subgroups.

The cluster dendrogram shows a hierarchy of clustering that goes from the tips, where every spectrum is in its own cluster, to the root, in which all spectra are in one cluster. The dendrogram shows how clusters are merged iteratively by the clustering algorithm. The length of the internal branches is proportional to the difference between clusters, where the difference between clusters is defined according to a linkage criterion.

The hierarchical clustering in JMP is a rich platform, with many features useful for spectral analysis that we do not have time to cover here. We provide the report in the attached journal so that you can discover the features on your own. Some features worth checking out are the interactive selection of the number of clusters, the output of cluster assignments, the ability to select and highlight clusters in the dendrogram, and the ability to launch parallel plots for clusters. It is also worth noting that hierarchical clustering has excellent performance for large n and large p problems.

Figure 9. Hierarchical clustering of the raw spectra (left) and the spectra preprocessed according to part 1 of this series (right). The raw data shows much more scattered clustering of the individual spectra resulting in poor sub-grouping while the pre-processed data shows better defined sub-grouping. One can save the individual cluster designations and use that as an input into a modeling process if so desired.

Advanced Pre-processing

Extended Multivariate Signal Correction

This section introduces a useful diagnostic plot called the scatter effects plot and a powerful pre-processing method called the extended multiplicative scatter correction (EMSC). The scatter effects plot is straightforward to create in JMP, and an example is provided in the attached journal. The EMSC will require some JSL knowledge to reproduce. We present the EMSC because it works well for these data, and the Fit Model platform is a useful pedagogical tool for teaching the method. Also, we intend to provide many pre-processing methods like the EMSC in the FDE platform in a future release of JMP, so stay tuned for that.

Figure 10 shows the scatter effects plot, which is useful when selecting pre-processing methods. There are patterns in these plots that indicate different sources of noise, and you can use this information to select what pre-processing methods are appropriate for your data. In the scatter effects plots, shifts in the y intercept indicate a constant baseline shift, or an additive effect. A pre-processing method like the 1^st derivative Savitzky Golay (SG) filter -- which we introduced in the previous blog post -- removes these effects. Multiplicative scatter effects occur when spectra have large variation at only certain wavelengths. These show up as shifts in slope, or “fanning”, in the scatter effects plot. Some pre-processing methods that can remove multiplicative effects are the multiplicative scatter correction (MSC) and the standard normal variate. We showed the standard normal variate in the previous post. In this post, we introduce the MSC. Chemical effects also show up as a distinct pattern in these plots. Interestingly, these show up as loops!

Figure 10. Scatter effects plot for raw spectra (A) and 1^st derivate SG filtered data (B). After the additive effects are removed by the SG filter, multiplicative effects remain. A pre-processing method that removes multiplicative effects will be necessary for these data.

The scatter effects plot for the raw data strongly suggests an additive effect (Figure 10A). There may also be multiplicative effects, but it is hard to tell because the additive effect is so dominant. So, we first remove the additive effect by applying a 1^st derivate SG filter on the data as we did in the previous post. In the previous post, we plotted the pre-processed spectra in Graph Builder and showed that multiplicative effects remained. The multiplicative effects are also apparent in the scatter effects plot -- note the considerable fanning (Figure 10B). So, there are both additive and multiplicative effects in the data. The EMSC can remove both sources of noise in one step.

In their original paper, Martens et al. introduced the EMSC and showed how this method could improve upon simpler pre-processing methods, such as those shown in our previous post. We are building up to the EMSC, but it is easiest to start with the multiplicative scatter correction (MSC) first. The MSC aims to correct for the various sources of noise in the data by making the data look more like a reference spectrum. Since there often is not a good standard available, the mean spectrum is frequently used. For each spectrum, a separate regression is performed. Let x = {x₁, …, x_p} be a vector of absorbances for a given spectrum and x^* = {x^*₁, …, x^*_p} the reference spectrum. The model is

And the spectrum is corrected as follows:

One problem with the MSC is that it does not account for the relationship that exists between scatter effects and wavelength. The EMSC address this by incorporating a polynomial relationship into the model. Typically, a quadratic relationship is assumed for wavelength. Another problem with the MSC is that by regressing each spectrum on the mean spectrum, the real chemical effects that are of interest can be removed. If the samples are known to be a mixture of a small number of chemical components -- and pure reference spectra are available for each component -- the EMSC can be extended further by incorporating prior chemical composition information into the model. In the data from Marten et al., there are pure gluten (x^g) and pure starch (x^s) samples in the data. As in the Marten et al. paper, we chose sample 1 and sample 93 to be the reference spectra. The EMSC model will modify the reference spectrum depending on the estimated chemical composition, which better preserves the true chemical signal in the data.

m = (x^g + x^s)/2 and k = (x^g - x^s) and l = {l₁, …, l_p} is the vector of wavelengths. The m and k model terms are expressed in this way to simplify the correction equation. The corrected spectra are computed as:

To perform the EMSC in JMP, we will first need to create columns for the x^g and x^s reference spectra. We do this by subsetting the data table to create tables for the reference spectra, and then join the reference tables back to the original table. We do this with the following steps:

Use Rows >> Row Selection >> Select Where to select all rows corresponding to one of the reference spectra.
Use Subset to create a table containing only the reference spectra.
Join the tables back to the original table, using wavelength as the matching column.

Next, we create the k and m columns, which can be done using formula columns that operate on the x^g and x^s columns. We then are ready to estimate the EMSC model. This can be done in fit model using the options shown in Figure 11. To perform the EMSC correction, we need to extract the model term coefficients and save them to columns in the data table. We need to use a jsl script for this, but the script is fairly straightforward. Finally, we perform the EMSC correction using the formula shown in Figure 12.

Figure 11. Fit model launch dialogue for the EMSC model.

Figure 12. Column formula for the EMSC correction.

The final result is shown in Figure 13. The spectra are remarkably free from scatter effects, and are a much more accurate representation of the true underlying spectra.

Figure 13. Spectra before and after the pre-processing steps applied in this blog. (Top) Before pre-processing, (Bottom) after EMSC correction.

Final Multivariate Calibration Model

The spectra are now ready for our final multivariate calibration model in FDE. We perform functional DOE using the method described in the FDE section. The resulting model allows us to accurately predict the spectral form for a given combination of mixture proportions. An example of such predictions is shown in Figure 14.

Figure 14. Functional DOE of EMSC corrected spectra before and after pre-processing. A linear constraint has been applied to starch and gluten so that they sum to 1. Note the large changes in spectral shape after pre-processing. Since scatter effects have been removed from the spectra, we can obtain more accurate estimates of the predicted spectral shape at factor combinations not observed in the data.

Conclusions and Future Posts

We demonstrated several multivariate platforms in JMP that are useful for identifying patterns in your spectral data. We focused on the subgroup analyses and outlier analyses common in spectral analysis. The utility of FDE for analyzing spectral data was on full display as we demonstrated the numerous exploratory tools available. Importantly, functional design of experiments allows one to easily build multivariate calibration models. We demonstrated how the Fit Model platform in JMP can be used to build an extended multiplicative signal correction (EMSC) model, which dramatically cleaned up the noise in our data. By building EMSC models from scratch in Fit Model, we gain more control and a deeper understanding of the model building process.

In the next post, we will cover supervised multivariate models. We adopt Brereton’s terminology and refer to these as inverse multivariate calibration models. These will predict experimental factors as a function of spectra. We will build models in the Partial Least Squares and Generalized Regression platforms. We will also show how functional principal components can be output from FDE and provided to any predictive model in JMP, enabling the construction of more complex calibration models

Your 5 most commonly submitted questions about design of experiments (DOE)

MikeD_Anderson — Fri, 04 Jun 2021 21:14:49 GMT

Recently, JMP partnered with the Royal Society of Chemistry and Chemistry World to hold a weeklong series of seminars on experimental design.

The series was, ostensibly, supposed to be for the eastern and western United States, but, as the attendee map below shows, well – things got a little out of hand on that point.

The seminar series included a panel of JMP experts who would answer questions submitted by the audience. With between 600 and 900 participants in each session, it was challenging to answer all the questions in the flow of the event. And there were a lot of them – on the order of 400 questions in the live session, with even more in the offline environment.

Once the team had recovered (and the joint liniment had done its magic), we got together and looked through the questions for trends to put in the series’ FAQs on the web. In true JMP fashion, we ran these questions through Text Explorer and came up with the top five. Here they are, along with our answers:

Question 1: What is the purpose of center points? When do we need to use them? How many do we need?

OK, we should probably first clarify this: A center point is an experimental treatment that exists in the center of the design space. If you have a three-factor design space, you could map it to a 3D plot, and it would generally look like cube. The center point would sit at the center of the cube, equidistant from all edges, faces, and vertices.

A center point’s primary purpose is to help alert us to curvature (or curvilinear behavior, if you want the $5 word for it) in the data set. A few center points together can also be used as a substitute for replication. That said, you really shouldn’t sweat the question of when to use them. If you focus on model-centric DOE (optimal designs) and other modern DOE strategies like definitive screening designs (DSDs), center points are automatically included when they are necessary. You just have to include model terms that need center points to be studied, and the algorithms take care of the rest. If want to add more center points to study variability (and you’ve already set aside material for a validation set!), then you can add them easily enough. You just have to do a power analysis to figure out the correct number.

Question 2: Do we need to replicate our experiments? Why? Can I use repeated measurements as replicates?

The answer to this depends a lot on the type of experiment you’re doing. If you’re more on the “screening” end of things, replicating is less important because you are looking for gross trends and not so much variability. You might add some center points to get an idea of the variation and do some screening for curvature if you’re not doing a DSD. But that’s about it. If you’re doing model building or refinement, then you probably should do replicates if you have the resources and time.

On the repeated measurements (e.g., running the same factor settings a few times one after the other or obtaining multiple measurements on the outcome of one run of the factor settings), those are not replicated runs. It’s a little confusing because replicates and repeats or repeated measurements sound like they should be the same thing, but there is an important difference in this case.

A replicated run (in the DOE sense) is a randomized repeat of the experiment. Say you were to take a DOE table and put it in a spreadsheet. Then if you were to put a second copy of the complete DOE below it on the same spreadsheet, that still would not be a replicated DOE...yet. Now, if you were to then randomize the row order of the two copies of the DOE, that would be a DOE with replicates.

If you are doing repeated measurements, you can take their average and then use the average as the value for that treatment combination, but they are not the same as replicated runs. The important point is that replication includes randomization and can therefore dilute the effects of lurking variables. Repeated measurements don’t have this property.

Question 3: How many responses can I have in my DOE? How do I work with them?

Adding factors to a DOE increases the cost, in terms of required runs; responses are essentially free. You can measure an unlimited number of response variables to understand how factor changes influence all the responses. When finding optimal process settings using the JMP Prediction Profiler, users can choose a weighting scheme that reflects the relative importance of different responses.

While there aren’t any theoretical limits to the number of responses you have in a DOE, it is to your advantage to define all your responses during the DOE set up. JMP will automatically pass them to the modeling platform after data collection with all the information you provide (upper/lower desirability limits, LOD values, targets, and importances). It also helps make sure you have considered how you are going to gauge success for your DOE and ensure that all your responses can detect differences at the levels you are planning for your factors.

Question 4: How do you know if your factor ranges are appropriate?

One rule of thumb is that if, based on your factor settings, you expect some of your runs to give undesirable results (i.e., end up in the trash), you’re probably right. But you don’t want your ranges to be so broad that you end up with no measurable results at all.

Another way this could be expressed is this advice: Be bold in setting your factor ranges. Set your lows low, and your highs high. There is a great temptation to only look at ranges that make you feel comfortable, because you suspect the optimum is achieved there. Avoid that temptation. The results from the model will be clearer and there will be more precision in the estimates when factor ranges are larger.

If you’re worried that your process will fail to produce usable data in some regions of the design space because of the wide ranges, this concern can often be addressed by defining factor constraints that prohibit runs from being assigned to those areas. Remember, you have the flexibility to use this strategy with optimal designs.

Sometimes experimenters are worried that by setting factor ranges too wide, they may miss an optimum setting that occurs in the middle of the range, due to a curved relationship between a factor and a response. In a custom design, this should be addressed by adding polynomial terms to the desired model, which will cause the generated design to include runs that are set at intermediate factor values between the endpoints.

More generally, you want your experiments to cover a wide enough range to clearly see differences in your settings. You also want your settings to be far enough apart so you can see any inflection points in your curved lines, allowing you to know where “good” and “bad” are. Otherwise, when you optimize, the result will be at the extreme high or low value in your design space. And, you won’t know if you’re close to a place where your system goes out of control.

Question 5: What is the difference between classical design and modern design?

Classical designs are fixed plans that the experimenter must follow, fitting the planned study into their requirements and constraints. These designs were developed before the era of ubiquitous and cheap computing. Modern designs use software algorithms to generate a unique plan for the situation at hand. Modern designs attempt to give the best answer possible for the statistical questions that are posed by the experimenter, while staying within the specified experimental budget.

Modern design is the natural extension of classical design. By running modern DOEs, you don’t lose anything you remember from classical design, but you gain a great deal of flexibility. Fundamentally, modern designs are model-centric in that they make the design fit a model made up of terms defined by the user. As a user, you are deciding what you want to learn in the experiment by selecting the model. Classical designs are design-centric in that they define the design, and the user is expected to make their questions fit into that predefined design. In many cases modern and classical approaches will lead to the same design because the classical designs are optimal for fitting standard models. Modern designs enable you to find the optimal design in situations where classical approaches would not work, like when you have an irregularly shaped factor space that you need to explore.

The adventure continues

And that’s it. When looking at the results, it was interesting to see how frequently these questions came up. It reminded me of something many of my teachers said: “Ask the question, because it’s very likely that others also have it but are too shy to ask.”

Anyway, do you think we missed one? Are there other burning questions that you have about DOE? Go ahead and put them in the comments, and we’ll see what we can do about getting them answered. Until then, happy experimenting!

Dark data and the pandemic, why tech investors love DOE, and more

J_Marquardt — Fri, 04 Jun 2021 20:49:43 GMT

Wicked problems.

Visit jmp.com/foreword to read the magazine.

I bet you can name one or two. They’re problems that are more than just complicated. They’re complex, with so many factors that one doesn’t know where to start.

The term “wicked problem” was coined by University of California, Berkeley, professors Horst Rittel and Melvin Webber to describe societal problems that are difficult or impossible to solve – for instance, pandemics and poverty, or sustainability and social injustice. These two first presented their paper on wicked problems to the American Association for the Advancement of Science in the late ‘60s.

Their work remains highly relevant. Our world is extremely complex, full of wicked problems. But here’s the upside – people and organizations are using their analytics skills to untangle challenging issues that will, in turn, help us solve some of our thorniest dilemmas.

In this issue of JMP Foreword, you'll find examples that show how leading science and engineering organizations use statistics to solve problems, make decisions and innovate.

Photo credit: Innogy

You’ll read about Heliatek’s pioneering organic solar film that is suitable for covering curved and irregular surfaces. This means we have another sustainable energy solution to bring us closer to ending fossil fuel dependency and reaching a goal of 100% green electricity.

Photo credit: Court Sandau

You’ll see how forensic chemists are using data visualization to help judges make decisions on important cases involving pollution and arson. This means we can bring offenders to justice even though contamination is notoriously difficult to prove.

Some of our columnists will share their perspectives on the vital role of statistics in tackling overwhelming questions and information overload.

Author David J. Hand writes about dark data and the COVID-19 pandemic. Researcher Galit Shmueli predicts what we’ll be able to accomplish with the deluge of data we’re creating in this age of virtual assistants, smart home devices and the Internet of Things. And Riffyn CEO Timothy Gardner explains how design of experiments has streamlined his approach to scientific innovation.

Riffyn CEO Tim Gardner describes how DOE can transform the scientific process.

While we don’t present JMP as an easy button to solve the world’s problems, our developers are always looking for ways to make analytics less frustrating and less burdensome. In one story, a crop scientist describes his JMP experience this way: “Tasks like compiling data over years were practically impossible in the past. We’ve cut the time from days or weeks to a few minutes or hours. And that opens up analysis that wasn’t possible before.”

What will you do that wasn’t possible before? Read JMP Foreword for inspiration.

What does it take to be a great quality engineer?

J_Marquardt — Fri, 04 Jun 2021 20:43:40 GMT

Greg Mattiussi of Siemens Healthineers speaks about the rise of the Industrial Internet of Things (IIOT) and how to be a great quality engineer in this modern environment.About this time last year, we hosted an event on how to use modern quality engineering methods to stay relevant and competitive. Experts from Los Alamos National Lab, Wolfspeed and Siemens Healthineers talked about embracing automation and the rise of the digital economy. Sound interesting? You can watch the on-demand video of the event.

Here's an excerpt from last year's panel discussion, where Greg Mattiussi begins a conversation about what it takes to be a great quality engineer:

“One thing that I look for is the ability to understand concepts that are relevant to the particular area that the candidate is in and the ability to use those concepts to create actionable ways forward.

“I think it’s important to not just have a set of tools that you’ve been exposed to and, therefore, would bring them up if you were in that situation. You have to understand why those tools and what they will get you. And that’s especially true for statistical methods.”

Modernization Updates in JMP Clinical: A follow-up

Chris_Kirchberg — Thu, 13 May 2021 20:07:44 GMT

It was great to see many people attend the recent Technically Speaking event to learn about the updates to JMP Clinical that help our users keep up with industry and regulatory agencies needs. It represented one step of many that will be coming in the next few years to not only modernize JMP Clinical, but also to help modernize how clinical trials are reviewed, analyzed and monitored.

During the presentation, we had lots of questions, and we couldn’t answer them all. So, we are addressing them in this blog post. To learn about what exactly was modernized in JMP Clinical, view the recording of the event.

General Questions

I am a medical writer – how easy is it to extract information for patient narratives?
- JMP Clinical will aggregate the information selected in the options dialog and automatically insert it into a patient narrative depending on the narrative template chosen within the application
Can JMP Clinical help with data quality and fraud detection?
- Yes, there are specific reports that look at data integrity from a site to site comparison point of view as well as a risk-based monitoring report that is at the site level as well.
Can JMP Clinical be customized? Or is what you see is what you get?
- Yes, it can be customized at a variety of levels and ways -- from choosing different options for the report using a point-and-click mechanism all the way to creating a custom report to appear within the product.
Can I easily share my work with my colleagues?
- Yes. It will depend on how you want to share with them. PDF, PPT, RTF, web browser (JMP Live) or JMP Clinical review/template (If they have access to JMP Clinical).
Can you create PDF, Doc and PPT formats for reporting?
- Yes. There are export options for creating static reports from an individual report or the collection of reports from a review
Can you view JMP Clinical reports on your phone or iPad?
- Yes. If you publish the reports to JMP Live, the content can be viewed on most web browsers on any device. But I do not recommend viewing such content on a phone because the graphs will not resize in a meaningful way.
Is JMP Clinical used by the FDA?
- Yes. In fact, it is required training for classification of a medical officer from level 1, associate reviewer to level 2, reviewer.
If you wanted to customize the color schemes used in reports, can that be modified?
- It is possible to change the color schemes used in an ad hoc fashion for a single graph, or you can set them for any report at the study level.
Does this run directly from standard SDTM data sets, or do you need to define/configure all of your own data sets and reports?
- Yes, standard SDTM-like and/or ADaM-like data sets. Custom domains following the SDTM/ADaM data structure can be used as well. You do not need to define or configure the data sets or reports to use JMP Clinical if the data sets follow the SDTM or ADaM data structure and rules.
Is it true that JMP is more about clinical data as compared to the post-market data?
- JMP Clinical was designed for ongoing clinical trials as well as completed clinical trials. It can be used in a post-market scenario if the data are formatted/structured like the clinical data. Not all domains are needed. Simple demography and adverse event data is really all that is needed to follow safety signals. The more data you supply, the more you can do (more reports and options). And with the inclusion of DSUR/PSUR report, one can use that for post-market reporting.

Presentation Specific Questions

Is DSUR/PSUR useful for ongoing studies?
- Yes. In fact, some sponsors use a DSUR like report for internal reporting on a regular intervals, so why not use this report for such activity?
DSUR and PSUR are at compound level. Does JMP Clinical handle multiple studies to generate DSUR and PSUR report?
- Yes. Within JMP Clinical, you can combine studies and then run the DSUR/PSUR report. All treatments/compounds will be combined if they have the same name but separated within the report if they are named differently (or spelled differently). Study identifiers are not included in the report except where adverse events are listed.
For JMP Live reports, can I use templates for publishing to groups or do they have to be specified each time?
- Groups cannot be saved to a review template. So each time you run a review template (any template), you will have to specify the group(s) if you publish to JMP Live
Can JMP Live be used to publish general SAS outputs from PC-SAS as well?
- JMP Live can be used with JMP, JMP Pro, JMP Genomics and JMP Clinical. These are the only tools that can publish to JMP Live. That means if you use JMP to get access to SAS and then create a report within JMP, it might be possible. But not directly from PC-SAS.
Is JMP Live available only for JMP Clinical 8, or can we still use it with an older version of JMP Clinical?
- It is only available in JMP Clinical 8 since that is the version that JMP Live publishing was first made available.
The JMP Live integration looks like a very useful way to share reports from JMP Clinical. What security options are available to limit access to those web reports?
- There are a variety of security options that range from limiting publishing to certain groups, different types of authentication to get access to JMP Live, and even how it is hosted. It is probably best to connect with your local JMP Sales Team to find out which security options you are interested in and if we support them.

Statistically Speaking: Demystifying Machine Learning and Artificial Intelligence answers to audience questions

HadleyMyers — Tue, 11 May 2021 19:48:22 GMT

Many people tuned in to watch David Hand, Imperial College London; Lene Bjørg Cesar, Novozyme; Francisco Navarro, Solvay; and Aziza Yormirzaeva, Corning Environmental Technologies, discuss machine learning and artificial intelligence.

The viewers asked a lot of great questions, but, due to time constraints, host Malcolm Moore from JMP wasn’t able to get to all of them during the event. We felt that many in the greater JMP community shared these questions and would be interested in the answers.

Q: While increasing the number of scientists who have data skills means faster innovation and delivery of products to market, it can also mean increasing the risk of making wrong decisions. Do you agree? If so, how can we mitigate that risk?

A: Analytics is a power tool that can be used to “see things that couldn’t be seen without it, much like a telescope or microscope,” as one panelist said, but like any tool, risks arise from improper use. In larger organizations, data scientists or statisticians within internal Centers of Excellence can guide process experts in best practices of analytic techniques. However, the greatest risk comes from bypassing the process experts entirely in favor of an overreliance on machines to make good decisions. Careful consideration of the results of any analytic endeavor, by process experts, is the best way to mitigate that risk. On the other hand, it is also equally important to consider the risks of doing nothing or maintaining the status quo, such as losing the opportunity to increase innovation capacity or to speed products to market.

Q: Am I the only one who finds that pro-innovation bias gets in the way of a sensible discussion of the basics, such as defining the problem, GIGO, DOE, and measurement systems?

A: This question received many likes during the event. The answer: No, you are not the only one. Evidently, this sentiment is shared among many. It would seem that, often, a rush to innovate causes experimenters to neglect the basics, as you’ve described them, in favor of a try-and-see or one-factor-at-a-time approach to development. When this happens, the best advice is to raise this point and emphasize the value of these basics during discussions with colleagues when planning experiments.

Q: I have been asked to predict commercially successful inventions from patent and nonpatent literature. Are ML and AI experts often asked do this?

A: David Hand addressed this question during the discussion. Data scientists are often asked to embark on this type of exercise. (In particular, patent data is a frequent subject for analytics, as seen in this presentation at a recent JMP Discovery Summit). David mentioned a similar situation in which he was tasked with creating a model to predict the success of start-up companies, which he described as “very, very difficult.” However, two new features in JMP Pro 16 – Sentiment Analysis and Term Selection – make just such an exercise significantly easier. (This short video demonstrates how these features work.)

Q: Many feel the true goal of machine learning and AI is to have the technology signal us on how to optimize and/or address signal deviations, such as in sensor data patterns. How far are we from meeting that goal?

A: This question was also touched upon by the panel. Each member emphasized the value gained by leveraging ML and AI to augment the understanding of processes or patterns that couldn’t have been seen without these methods, but also the importance of process experts to interpret these findings. We are at a point now where machines can optimize, detect, interpret, signal and even decide what actions to take, but there are massive risks in allowing them to implement these autonomously by removing specialists from the equation. By the way, Functional Data Explorer in JMP Pro is a great machine learning tool for analyzing sensor data.

Q: Have you experienced pushback over the fact that something was done a different way historically, specifically in the manufacturing environment? If so, what did you do to help persuade people to use the new tools and features available?

A: Pushback is a common response to anyone trying to implement change. The advice on overcoming it is to think big, start small, and broadcast your successes. Machine learning can be used to solve big problems or revolutionize entire industries, but it can also help with small things that are under the radar for most people. And while your company, department or team may be resistant to change, there may already be success stories elsewhere in your company or close to home that will resonate with your colleagues. You can find a collection of such stories here. Often, the fear of change may be driving the pushback. If that’s the case, STIPS, JMP’s free online course that can help scientists and engineers overcome their fears, may prove useful.

Q: I feel there is some confusion around the differences and overlaps between the terms AI and unsupervised ML, as well as supervised ML and predictive/inferential analytics. A clearer set of definitions would be welcome.

A: Indeed! To add to what the panelists discussed during the event, this page from SAS does a great job in explaining the differences and overlaps between these terms. In short:

Artificial intelligence: the broad science of building machines to mimic human abilities.

Machine learning: a specific subset of AI relating to the techniques by which machines are trained to learn.

Supervised learning: the development of models to classify, explain or predict outcomes in data using data with known outputs.

Unsupervised learning: comparing or differentiating data points based on patterns or underlying structures within the data itself, without the use of response variables.

Predictive/inferential analytics: using supervised or unsupervised learning to make predictions or infer knowledge and understanding about the world around us (manufacturing processes, product developments, weather patterns, stock markets, etc.).

Q: What’s the best way to bring the data experts and the “physical” experts (e.g., sensor development) together? Very often, the first lacks understanding of physical artifacts, while the other lacks understanding of what ideal data should look like.

A: Optimally, the data experts and physical experts are the same people. JMP delivers the power of analytics to the scientists and engineers that are in the best position to understand their individual processes, but who don’t necessarily have a background in data science. Where possible, statisticians and those with data science backgrounds serve in advisory or overseer roles to support those process experts in their efforts.

JMP 16の新機能を用いたデータ分析　Part.3 EWMA管理図　～少しの変化でも気づくように～

naohiro_masu — Tue, 11 May 2021 18:33:16 GMT

ニュース等で ”季節外れの暖かさ“と聞いても、最近はあまり驚かなくなりました。しかし、今年3月の東京近辺は特に暖かいなと感じた日が多かったです。近所の桜がいつもになく早く咲いていたので、あー、季節の変わり目が早いなあと感じました。

本ブログは、グラフビルダーの開発者であるXanさんの投稿から影響を受けています。その投稿とは、京都の桜の開花日を西暦812年から現在までまとめられたグラフです。（桜のマーカーはキレイですので、是非ご覧ください。）

Kyoto cherry blossom peak bloom day since the year 812

https://public.jmp.com/packages/J_rDc8JMMdjJBJDf7fMbF

このグラフに描かれている平滑線をみると、19世紀の後半から現在まで、桜の開花日が急激に早まるトレンドがあることがわかります。

桜の開花日は、その地域の2月1日からの日平均気温の合計が400℃になる日が目安となる「400℃の法則」が有名です。そのため、開花の早い遅いは、2,3月の気温が影響していると考えられます。

そこで、1970年から現在（2021年）まで、2,3月における京都の平均気温を調べてみました。

下図は、年ごとに2月の平均気温（オレンジ色）、3月の平均気温（赤色）を折れ線と平滑線（トレンドをみる線）で示したものです。（データの出典：気象庁）

平滑線より、近年2, 3月の平均気温は上昇傾向にあることがわかりますが、特に2010年以降、3月は急激に上昇しており、今年2021年3月の平均気温は11.6℃と、1970年以降で、最も高くなっています。

では、これら気温の年ごとの変化を調べるために、工程管理で用いられる管理図を用いてみたらどうでしょう。管理図はさまざまな種類がありますが、そのうちの一つであるEWMA管理図は小さいシフト（変化）も検出できるので、グラフで見た気温の上昇トレンドが検出できるかもしれません。丁度、JMP 16でEWMAはさまざまな機能追加があるので、使ってみるほか考えられません。*

（*同じように、小さなシフトを検出する管理図としてCUSUM(累積和)管理図があります。）

EWMA管理図とは

EWMA管理図は、指数加重移動平均（Exponentially Weighted Moving Average）管理図の略です。通常のX管理図は、その時点での値をプロットするのに対し、EWMA管理図では、その時点での値は、過去の値にも依存してきます。依存の度合いが直近であるほど大きく、過去に遡るにつれて指数的に依存の度合いを小さくします。

X_iをi時点での値としたとき、EWMA管理図のi時点での値 Z_i は、その1時点前の値Z_i-1と、ユーザが指定する重みパラメータλ（lambda）を用いて、次の式で計算されます。

Z_i =　λ X_i + (1-λ) Z_i-1

JMPのデフォルトではλの値は0.2ですが、このときEWMA管理図では、20%を現在の情報、残りの80%を過去の情報を用いて値をプロットしていることになります。

式の性質から、過去に遡るほど、用いる情報の量（重み）は指数的に減少します。下図はλ = 0.2 としたときの、現時点(0時点)に対する、過去の時点の重みを示しています。

現時点では0.2の重みがありますが、1時点前(-1)、2時点前(-2) と過去に遡るにつれて、重みが指数的に減少していることが分かります。

EWMA管理図を用いて工程管理をすると、工程における小さなシフト（変化）を検出しやすくなると言われています。そこで以降では、上記に示した2, 3月の京都の平均気温についてEWMAを描いてみます。

JMP 16のEWMA管理図

JMPでは以前からEWMA管理図を描くことができましたが、JMP 16ではインターフェイスが変更され、大幅に良い機能が追加されました。主な追加機能は以下の通りです。

EWMA管理図とともに、X管理図、残差の管理図も表示する。
レポートにあるコントロールパネルにより、λ(lambda) やσ（sigma）を変更し、インタラクティブに管理図に反映する。
変化の開始点を検出できる。

京都の平均気温に対するEWMA管理図（2,3月）

まずは2月のEWMA管理図です。

図は上から下にかけて3つ描かれていますが、上から順に、EWMA管理図、X管理図（気温の値そのものを用いた管理図）、残差の管理図です。

残差の管理図は、ある時点のX管理図の値から、1時点前のEWMA管理図の値を引き算で計算される残差をプロットしたものです。例えば、2010年の残差は、2020年の値から2019年のEWMAの値を引くことにより算出できます。

左側の「設定パネル」に入力されている値は、次の通りです。

目標値：データの平均値。(EWMA管理図、X管理図にある緑色の線)

Sigma : バラつきを示す値（σ）。データから推定される。

Lambda: 上記で説明した重み（デフォルトは 0.2）

赤色の線は、3σ（シグマ）で計算される管理限界線です。

真ん中のX管理図をみると、見た目ではそれらしきトレンドを発見することができません。しかし一番上にあるEWMA管理図をみると、70,80年代にかけての下降トレンド、2018年以降の上昇トレンドを確認することができます。ただ、どの年も管理限界線は超えていません。

ちなみにEWMA管理図の右上にプロットされている青色の点は、2022年の予測値です。

次は3月のEWMA管理図です。分かりやすいトレンドが見てとれます。

X管理図は2021年に上側の管理限界の外に出ています。EWMA管理図では2010年代に上昇トレンドが見てとることができ、2020年に管理限界の外に出ています。

EWMA管理図に引かれている紫色の線は、シフトが起こった開始点を示します。2月の平均気温では検出できませんでしたが、3月の平均気温では小さな変化を検出できているのです。

ここでは示しませんが、東京の平均気温でも同じような傾向を示す管理図が描かれます。近年3月でも暖かく、桜が咲くのが早いなあと感じる感覚は、EWMA管理図の上側に向かうトレンドが証拠となるでしょう。

JMP 16のEWMA管理図では、左にある「設定パネル」から、目標値、sigma, Lambdaの設定を変更すると、即座に管理図がその設定で描かれます。下図は、3月のEWMA管理図でLambdaを0.1に変更したときのものです。つまり、過去のデータの依存度を高めると、結果として今年（2021年）が上側管理限界の外に出ます。

環境問題として考えると

寒いのが苦手な方には、早く暖かくなるのは有難いことかもしれませんが、地球温暖化が進んでいる結果みなすと、手放しに喜んでいるわけにはいきません。

現在の国際的な環境の取り組みに関して”パリ協定”が有名ですが、それより前の協定として、今回話題とした京都で定めた”京都議定書” があります。京都議定書は1997年に地球温暖化対策として採択されましたが、あれから20年以上たった今でも、効果的な対策が取られていないと言わざる得ない状況が続いています。

アメリカでバイデン大統領が就任し、今年2月にパリ協定に復帰しましたが、これからが温暖化を食い止めるための重要な局面だなと感じます。

Is it OK to interchange terms like 'machine learning' and 'AI'?

J_Marquardt — Tue, 04 May 2021 19:43:27 GMT

Cameron Willden supports engineers and scientists across many different product lines at W.L. Gore, focusing on manufacturing and new product development.

Cameron Willden, W. L. Gore, touches on the differences between data science and statistics, and the differences between machine learning and AI. He responds to some of the learning objectives covered in a keynote by author David J. Hand and provides his own perspective based on years in industry. You can watch the keynote and the panel discussion at any time.

Need some help demystifying machine learning and artificial intelligence? We have upcoming livestreams and on-demand videos on this topic. Check them out!

“Yes, I believe there are some really important differences between, in particular, data science and statistics. I think that David explained the differences pretty well...

“As a statistician, we tend to have more of an emphasis on quantifying the uncertainty in our estimates, and data science tends to emphasize more the point predictions that come out of a model. And because of that, we tend to gravitate toward a different set of tools.

“Statisticians are looking at hypothesis testing and different types of statistical intervals – things like bootstrapping – where data scientists gravitate toward things like neural networks and support vector machines and tree-based methods. Not to say that they don’t cross paths with each other every once in a while, but there certainly tends to be more of an emphasis that aligns with things that they care more about.

“As far as machine learning and artificial intelligence, honestly, I’ll admit I haven’t heard it explained in quite the way that David explained that; that might be because he’s coming at it with a better understanding of some of the subfields of artificial intelligence.

“But the way I typically think of machine learning and artificial intelligence is that AI is the end goal. We’d like to have a machine that can perform a task competently, even when it is faced with an example we haven’t explicitly encountered before. And to get there, we use machine learning tools – a set of techniques and algorithms that we use to school the machine to help it to grow its intelligence. And hopefully, at some point, it’s ready to have a job.

“So that’s the way that I typically think about those terms. But I’ll say that they are nebulous terms, and you’ll very often hear people use AI and machine learning interchangeably. I don’t really see where that is causing major problems or confusion, so we may not converge on a common understanding of those terms in the near future. But maybe that’s OK.”

통계적 기법 - 지무지도(知無知圖) 시리즈 1: SPC 개념

ChulHee_JMP — Thu, 22 Apr 2021 20:33:48 GMT

SPC(Statistical Process Control; 통계적 공정 관리)라는 주제가 세상에 알려진 지 이미 80년이 넘었다.

WALTER A. SHEWHART 박사와 W. EDWARDS DEMING 박사에 의해 1939년에 발간된 'STATISTICAL METHOD - FROM THE VIEWPOINT OF QUALITY CONTROL'을 그 시작으로 볼 때...

SPC는 이제 전 세계의 다양한 산업에 광범위하게 적용되는 필수적인 개념이자 기법이다. 그러나 여전히 많은 사람들이 기존의 '상식'으로 인해 SPC의 올바른 개념에 대해 오해와 착각을 하고, 그로 인해 현장에서는 혼란이 발생하고 오히려 품질과 생산성에 악영향을 준다.

이 지무지도(知無知圖) 시리즈의 첫 번째 시간을 통해, SPC의 올바른 개념과 응용 방법에 대해 다시 고찰하고자 한다.

Happy Earth Decade!

Landra_C — Wed, 21 Apr 2021 17:19:02 GMT

For years, SAS has striven to make environmentally sustainable and socially responsible business decisions. Those of us in the JMP division have applauded those decisions, and we’ve built upon that foundation whenever possible. When we heard the United Nation’s recent rallying call to protect and revive the world’s ecosystems, we were compelled to answer.

JMP is joining the UN’s Decade on Ecosystem Restoration, and we hope you will, too. “Over the next ten years, every action counts. Every single day. Every country, company, organization, and individual have a role to play.” That powerful statement is from the UN Decade website.

We’re excited to do our part, and we’re inviting current – and future – JMP users to join us. Are you ready to commit to a decade of sustainability practices?

Here’s our three-part plan:

Analyze.

Data is power. But not if it sits unexplored with its mysteries unsolved.

That’s why we’re inviting you and other organizations to join us in exploring consumption data.

By exploring travel, energy use and carbon footprint data, we can identify specific changes to the way we work that advance conservation. Not sure where to start? Begin with the “Getting Started with JMP” webinar. To ensure that you understand the nuances of what you’re seeing and to sharpen your statistical skills, take our free “Statistical Thinking” course. And top it off with whichever “Mastering JMP” webinars you need to be completely up to speed in your approach to dynamic data analyses.

Act.

We’re using our data to take a hard look at our ecological footprint. Once we’ve audited our travel, events, marketing programs and more, we will make data-driven decisions that foster our own sustainable practices. We’ll also work with green vendors at every opportunity, align with corporations that are committed to restoration and assist learning institutions and non-profits with their green initiatives.

Knowing what your organization is doing well – and knowing what could be improved upon – is only the first step. Consider what you can do with what you learn. How can you implement environmentally friendly practices?

Amplify.

We believe that by amplifying our voices we can multiply the number of people working toward change. Start by making sure people within your own organization understand your analyses and support your actions. We’ll share our confidential data on our intranet, of course, but we’ll also put much of our data on JMP Public for all to see. We hope the organizations that align with us will also share data along their journey both internally and externally.JMP plans to use our communication vehicles for climate awareness at local, regional, national, and even international levels. This includes – but is not limited to – talks at Discovery Summit events in the Americas, Asia, and Europe.Find ways to integrate awareness into your organization’s communications and events – putting climate change efforts right up front. Certainly, in some fashion, you’ll have opportunity to celebrate and amplify the achievements of those around you.

At www.jmp.com/green, we’ve curated a few stories of researchers who have dedicated their careers to environmental stewardship. Read them there.

And when you’re ready to take the next step – whatever that might be – engage with us. We’ll be right there with you. It all starts at www.jmp.com/green.

Big data and the race to save coral reefs

J_Marquardt — Wed, 14 Apr 2021 17:30:57 GMT

Anderson Mayfield explains that predictive modeling can help determine which reefs are more susceptible to environmental stress - and which are most resilient.With climate change prompting rapid coral bleaching on global scales, marine biologist Anderson Mayfield uses predictive modeling to prioritize conservation efforts. You can hear more about his work in a recent episode of Statistically Speaking or read about it in a back issue of JMP Foreword magazine.

“I think it’s the time for a least a few of us to start making these transitions. We’re just explaining things that happened in the past. ‘Oh, this reef died because of an oil spill.’

“That’s all well and good, but if you have the opportunity or the data set with which you can start making predictions about reefs that haven’t yet died, you might actually be able to do something proactive and save that reef.

“So, basically, I went from just writing coral reef obituaries to kind of thinking of myself as a coral reef actuary. I’m not going to know the exact date this reef is going to die, but I might be able to tell you that this reef over here is in worse shape than this reef over here," says Mayfield, who jointly works for the US National Oceanic and Atmospheric Administration and the University of Miami.

“And I want to at least have this triage capacity to where we can go to a manager and say ‘Based on our data (which is all being fueled by JMP Pro), based on these predictive models we’ve made, we think this reef over here is the one that’s going to be a refuge. The corals, for whatever reason, are super resilient. This one over here is stress susceptible.’

“I’m not sure which one you might want to prioritize, but I least want to have the data in hand to be able to make these kind of decisions.”

Experts weigh in on how to build analytic cultures of excellence

anne_milley — Mon, 12 Apr 2021 19:06:27 GMT

Treating data as a core asset, cultivating curiosity, encouraging experimentation, continuing to invest in training and mentoring – these are just some of the many important takeaways from the plenary and panel discussion featuring Loren Perlman, VP of Science at Riffyn; Andre Argenton, VP of Core Research and Development at Dow; and Kumar Subramanyan, Director of Data Science at Unilever. Watch the on-demand version of this episode of Statistically Speaking.

Here's an excerpt on data as a core asset from Loren's plenary talk:

We had many good questions from the audience that we weren’t able to answer at the time. Our featured guests have kindly provided answers to some of the questions. We thank them for sharing more of their wisdom.

Can you talk more about FAIR data practices?

Loren: FAIR stands for Findable, Accessible, Interoperable, and Reusable, defined as follows:

Findable means that there are search and organization terms associated with data sets that make them findable when using relevant search terms, such as project, experimental purpose, etc.
Accessible means that once the data is located, it is obtainable for use – I can literally download it. This may come with associated authorization or authentication requirements, depending on a given organization’s data access policies.
Interoperable means that the data sets can operate with applications or workflows for analysis, storage, or additional processing.
Reusable means that the data is well documented with associated metadata to enable recombination in different settings. In other words, the data and associated metadata use shared vocabularies (onotologies) that enable an understood relationship between data sets – defining how are they related via the data or metadata contained within.

More thorough information about FAIR can be found here: https://www.go-fair.org/fair-principles.

How have you addressed pushback on purchasing JMP licenses over the “why can't we just use Excel” argument? Any tips for proving to non-users that JMP is worth it?

Kumar: This is precisely the argument many organizations are facing. Microsoft Excel is pervasive in organizations, but increasingly, you have folks getting trained in R and Python. The question is what is the value of investing in JMP? The latter point is relatively easy to dispel – R, Python and other open source analysis tools need a certain level of expertise and training. As a result, they are not the route to democratizing analytics, even though they may be a route to democratizing specific analytic solutions in organizations. The conversion from Excel to JMP is the right question to address, and the following points will help this journey:

Use Excel as a starting point to get people on the analytics journey, if needed. Basic data management/organization and analysis can be taught in Excel.
In the next phase of up-skilling, embed JMP as a key enabler. Showcase its unique capabilities and advantages over Excel (for example, design of experiments, visualization, handling larger data sets, significantly larger number of methods and models).
Build communities of JMP users around application areas; this works better than targeting individual users.

Andre: There are really two approaches, and the choice of the right one depends on many factors. One approach is the corporatewide one. It requires someone in the company’s leadership to understand the value and buy in to the idea that a transformation happens faster when there is top-down commitment and support. If you do not have a champion at the top, then you should consider the second approach, which is one of growing support through success cases. For this approach, you need to literally run an experiment and have a subset of the organization operating as a test case. This trial group would be given the chance to operate with the right tools, such as JMP, as well as the right support structure. Consider it a case study to show value. The success of this group would feed the future investments to expand.

Loren: It’s a fair question from management, as they are trying to illustrate ROI (return on investment) for any money spent on new tools or strategies for data management. Excel is admittedly a fast and easy tool for limited-scale data sets and basic analyses. The problem appears when either the data sets become substantial in scale (and substantial means different things to different people) or the modeling needs become significant. There is simply no comparison between the power of JMP as an analytics platform. I think Kumar and Andre covered the right approaches – establish a core capability and provide support so that users can demonstrate clear value from implementing the tool.

What are specific ways to incentivize change and growth toward a culture of analytics?

Kumar: I have a few suggestions:

Invest in the right tools (systems) to gain user conversion/adoption. Focus on a great user experience and seamless data access.
Identify early adopters/business champions and use them to drive further adoption.
Implement incentives, such as learning credits, that can contribute to future career opportunities. Redefine roles to include analytics as a key skill set.

Andre: Celebrate small victories and celebrate behavior. When a researcher uses new skills, a novel approach to analyze the data, or a different approach to visualize the data, that example should be celebrated as much as the discovery of new material.

Loren: As the cliché states: Money talks. Offering appropriate rewards for change in culture go a long way. Maybe offering a lunch for the team that demonstrates the cleverest use of analysis or a cash reward for problem solving that might not have been accomplished without a data- and analytics-first approach. I would say that it also starts during the hiring process – we need to hire people who have certain skills or understand they will be expected to develop them on the job. I do think we tend to silo skill sets a bit (for example, a data scientist does the modeling; a bench scientist does the experimentation). Why can’t people be expected to upskill and continuously grow in multiple dimensions?

I've been hearing a lot lately that we should make data a core asset, and I agree. My question is how do we do that, and what does that look like?

Kumar: This needs to be addressed at all levels of the organization to be fully realized:

At the top level, there needs to be a data strategy that defines what core data is, how it will be managed (acquired, stored, protected, accessed), and how it will be used to create value.
At the lower levels of the organization, two things are required: 1) instilling a culture of pride in data and data as a shared asset in the organization; 2) operational processes and tools to enable good quality data to be collected and managed as a shared asset.

Loren: We see this a lot in deployment of Riffyn Nexus. The short answer is that it needs to be easy to collect high-quality data and organize it according to FAIR data practices. If there are barriers to data collection and management, then users will simply revert to existing behaviors, such as siloed data in locally saved spreadsheets.

If your organization has a very minimal data culture – little to no alignment between business processes, data and business decision – what do you focus on first?

Kumar: Focus on business processes that are generating large amounts of data. Show the value of analyzing these data in terms of new insights that lead to better decisions or operational efficiencies from automating analysis/reporting. Once the impact is recognized, move to systematize data capture/data quality issues and show improvements in quality of insights/decisions. Lastly, identify opportunities for business process improvements that will invariably come out of the analytics. These improvements are usually much harder to implement unless their benefits are significant.

Andre: In alignment to Kumar’s point, I would focus on those areas where you can effectively make the most impact and where you can deliver value quickly. Say you have two opportunities: 1) a $100 million opportunity that will take multiple years to be realized and that requires massive effort, or 2) a $10 million opportunity where data is largely available, stakeholders are more committed, and analytics has an obvious and large role to play. Choose the $10 million opportunity. Deliver on it, gain credibility and trust, and work your way toward the point where the organization can address the larger opportunities.

Loren: This intersects tightly with the cultural change and incentivization question. It requires a top-down investment and vision, along with bottom-up tools and attitude to make it happen. I will note it is not easy, but it is deeply rewarding at every level within an organization. Management needs to create the space and incentive for this to happen, while providing the infrastructure to drive it.

Has there been any evolutions in HR to support building of a culture of analytics, such as job descriptions, incentives for up-skilling, etc.?

Kumar: Yes. HR needs to and is playing a critical role in building an analytics culture in organizations. It spans from including data and analytics as a specific skill, creating analytics roles, building a recruitment network, and championing up-skilling of employees through learning opportunities.

Many organizations have seen the concept of citizen data scientists used to promote/elevate the base of the organization to a higher level of analytics awareness and culture. They create incentives, ranging from certification, learning credits that help further career opportunities, and reverse mentoring for senior management.

How do you aggregate data across a large group? What tools can be used to take the data from one problem so that it has relevance to the greater, aggregated data set?

Kumar: For a group that has a common domain (such as life sciences or manufacturing), the key is to standardize a process for data capture, which will drive efficiency and, more importantly, data quality. The biggest hurdle to data sharing is poor data quality, like missing data or lack of metadata. Improving data quality can be done via a tool such as a standardized data capture template or a user-friendly workflow (Electronic Lab Notebooks work well). For groups across domains, it can be more challenging. It works best if there is a defined use case that needs to access data across these domains. Assuming this has been established, bringing together these data sets in a cloud environment for analysis is the standard operational method today. The key is to allow search and discovery of the data across these groups to facilitate analytics solutions.

Loren: I want to echo Kumar’s excellent comment about standardizing processes for data capture. This means providing the tools and ontologies that create consistency and alignment within groups. Driving data aggregation between groups can be achieved by understanding how data and materials flow between groups. It means aligning on simple things, like sample naming schemes, so that data can be joined in a data lake or a tool like Riffyn Nexus.

How do you get people to trust the results of analysis when they don't fully understand math or statistics?

Kumar: This is indeed one of the challenges to creating a pervasive culture of analytics in organizations; therefore, the importance of change management cannot be underestimated. We talk about the need for raising the floor of the organization, which refers to raising the awareness of data and analytics and its value across all levels of the organization. There is also a need to target the managers and leaders who need to make the business decisions. We are increasingly seeing data and digital transformation training for leaders. In doing the above, there is also a need to address the hype vs. reality dichotomy that exists among the non-experts. There are some who believe analytics (AI/ML) is a panacea to all business problems, while there are others who will cite the glorious failures to support their position that the value of analytics (especially AI/ML) is over-hyped. The truth is, of course, in the middle, so the goal of all training must be to showcase this as the art of possibility, laced with a sense of realism to drive the right behavior and outcomes in organizations.

Andre: I agree with Kumar’s point of raising the floor and that starts with the need for continuous education. I also believe that a center of excellence with highly respected individuals and subject matter experts in the field will provide the assurance for people to accept it. With that acceptance, results and trust will naturally build within an organization.

The key to generating value is taking action from insights and analytics. How should organizations drive action along with analytics?

Kumar: Clearly, a transformation to a culture of analytics will fail unless there is value created in turning insights into actions. Therefore, it is important that a very clear set of actions are agreed upon at the beginning of every analytics project. Every result or prediction must have an specific outcome for the business, whether it is a measurable improvement in efficiency or an impact on a business process or decision.

Could you please touch on how ethics and sustainability aspects can be addressed in an industrial setting?

Andre: I believe that the meaning of ethics and sustainability in this question is regarding the ethical treatment of data and long-term sustainability of the digital solutions proposed (as opposed to general ethics in industry and general sustainability challenges to the planet).

The ongoing debate of ethics in data and data treatment that is being discussed across academia and industry is a very important one. In the ideal world, data generated is transparent, and analyses from the data are reproducible, auditable and self-explanatory for future users. Professor Philip Stark from Berkeley talks about the concept of “preproducibility,” and it is one that addresses this topic well. I see industry and academia evolving toward a system of data transparency within a specific field and a specific group (that is, everyone in an organization who can create value from that data should have access to the raw data and its associated metadata).

Addressing the question of sustainability of systems and processes, industry always has to make a decision when developing a system and data generation and analysis workflow. Do you choose a rapid development that serves one project really well but is not scalable? Or do you choose a longer and more expensive development of a system and data generation workflow that is holistic and applicable across all of the workflows? The answer, in my view, is we need balance. Some of the fundamental pieces of data structure and processes in research that will impact most of the workflows should be treated holistically, since it is cheaper and better to do it this way. However, unique systems, unique projects and unique needs will always exist, so it is important to have the flexibility to accept those in your data architecture and data culture. In other words, a pragmatic approach should always be considered.

Watch this episode of Statistically Speaking via our website.

Two sides of the same coin

J_Marquardt — Thu, 08 Apr 2021 20:52:52 GMT

Tim Gardner, CEO of Riffyn, on his journey to a career in science.In a Statistically Speaking livestream earlier this year, we hosted panelists who are purpose-driven in their analytics. Tim Gardner, CEO of Riffyn, was among them.

In this clip, he shares how he came to dedicate his career to science. To learn more about his journey and passion for statistics and science, you can see the full discussion or watch a keynote by Tim at a different Statistically Speaking event on design of experiments.

"Well, the short story is I wanted to be an astronaut. And that passion ultimately evolved to engineering and engineering evolved to science because, the truth is, they are two sides of the same coin.

“Thermodynamics was born of the steam engine, and fluid dynamics was advanced through the Wright brothers' successful attempts to build an airplane. Signal processing was born from the development of radar, and virtually all the most innovative engineering in the world is derived from innovative science and experimentation.

“For me, I really followed the same path. As I wanted to advance as an engineer – in the engineering field – I had to become a scientist, almost by necessity, but also out of passion and joy for that.

“Ultimately, that led me from airplanes to robotics and to biotechnology and the life sciences that underpin it.”

Catch the event on demand.

How Netflix's crackdown on password sharing could use analytics

HadleyMyers — Fri, 09 Apr 2021 17:01:06 GMT

Recently, I came across an article on BBC World News titled “Netflix considers crackdown on password sharing” that sparked my interest and got me thinking. Apparently, some Netflix subscriptions are being shared by multiple households in violation of the terms of service.

This issue isn’t unique to Netflix; most streaming platforms allow for simultaneous viewings from multiple devices by “allowing users to create multiple profiles,” as the article describes and which anyone who’s used a streaming service in the last few years could confirm. What users aren’t supposed to do is let friends or neighbors who don’t live at the same address use their subscription.

I have to say that I’m a huge admirer of Netflix, as well as a customer. My account has three profiles: one used by me, one by my wife, and one shared amongst our three kids. It has always amazed me how well the shows recommended on my profile align with my personal tastes and how they differ from those displayed more prominently in the other profiles.

As far as analytics is concerned, Netflix is an extremely mature company, and I have no doubt it is employing many of the same techniques used in recommending shows to me as it is in identifying terms of service violations. However, having no insights into its inner workings beyond this 300-word BBC article and my own experiences using the platform, I can only imagine what those techniques might be. Here’s what I’ve come up with:

One option that clearly won’t work is to confirm sign-ins using local IP addresses that are linked to the physical addresses of the accounts. In the days before the pandemic, I used to travel quite frequently and would often sign in to Netflix using my profile from that very same iPad from hotels in various cities or even countries. I’m sure there are others who have multiple residences from which they watch Netflix. I’ve seen people watching Netflix with their phones on public transportation. This is all allowed under the terms of service, and taking steps to prevent it would certainly aggravate many loyal customers.
The current trial, which uses an email or text message to verify an account, may equally annoy some people. I can easily imagine scenarios where I am unreachable just at the moment my 3-year-old tries to watch Peppa Pig and is asked to verify her account through a text sent to my phone. The screaming that would ensue (not really, she’s a great and understanding kid, but stay with me on this one) would be placated by immediately switching to Disney+, and perhaps, never switching back.

There are situations that are more likely than not to be actual violations, such as consistent log-ins to the same account during prime time at several repeating geographic locations within the same city, or daily simultaneous use in different countries or continents. There may exist isolated cases of legitimate use even within these occurrences, but they are hard to imagine and would certainly be in the minority.

According to the BBC article referenced earlier that forms the entirety of my research into this topic, Netflix “now has more than 200 million subscribers around the world.” If it is going to make any meaningful attempt at tackling this issue, it will have to turn to analytics for help. A really great place to start may be one of several unsupervised learning algorithms available in JMP, such as hierarchical clustering.

Unsupervised learning allows for users with similar viewing habits to be classified into clusters without having to identify those habits in advance. In the example below, 15,100 internet users are grouped into five clusters using 51 usage metrics. As can be seen in Figure 1, not all clusters contain the same number of users. In fact, Clusters 2, 3 and 5 contain only 25 users between them. If this were my data set, I would take a closer look at these clusters to try to understand how they differ from the remaining 15,075 people.

Figure 1: Counts for five clusters for data set of internet usage habits of 15,100 (left) and dendrogram produced by the hierarchical clustering platform in JMP.

That information may be found in the parallel cord plot below in Figure 2, where each line represents a cluster. Looking at the leftmost extreme of the plot, it seems the three users in Cluster 5, on average, had a larger tot_HO (total hours online) than the users in any of the other clusters, on average. This may be one of the metrics that distinguishes this cluster from the others. There are others as well, as indicated in the portions of the plot where the lines deviate.

Figure 2: Parallel cord plot, showing how the usage habits differ for each of the five clusters.

The next step would be to consider these differences and think logically about which clusters are composed of users who are most likely violating their service agreements. The nice thing about JMP is that initially, if nothing concrete emerges, the platforms are designed to let analysts keep exploring the data until something meaningful pops out at them. Once that happens, they'll have a good idea of the accounts to target the warnings.

From there, a streaming platform like Netflix could monitor these accounts to find out whether there are differences in their usage habits. Doing so might indicate that some of the accounts had multiple households sharing passwords, and the warnings were heeded. A clear outcome would be to do this as a test in a localized region and see if there is a spike in subscriptions in the days or weeks that followed.

All of this could be the foundation for a training data set and the development of a model to estimate the chance that a user is committing infractions. Likelihoods greater than 50% may suggest violations, but the company may choose to be more conservative in its approach and only consider users above, say, a 70% chance as potential violators. This can be done by adjusting the probability threshold to optimize the confusion matrix – these options and others are found throughout JMP. The outcomes of this model could even be binned according to probability, with different actions taken on these bins. For example, users with habits indicating a 99% chance of violations may have their accounts suspended, while those above 80% may be sent warnings.

This is where a streaming platform that wants to stop password-sharing needs to make some decisions about what it’s hoping to achieve, where its priorities lie, and what it would absolutely like to avoid. Without careful consideration, it may use this process to inadvertently develop models that unfairly and incorrectly target minority populations. Or, bad guesses may lead to scores of legitimate users cancelling their service and switching to a competitor. This highlights the importance of including human oversight in every step of this process, including, and especially, having humans evaluate the models using logic and common sense. There are many examples of organizations learning the hard way what machines will do to their brand if left to their own devices.

Perhaps streaming companies have ways of monetizing viewership apart from the subscription fees, such as product placement, and would rather err on the side of more viewers rather than fewer. Of course, how many viewers gained versus subscription fees lost would be the type of information needed before making those decisions. JMP can be used at every step of this process. Our software enables domain experts to wade through large and complex data sets until insights emerge that only they, being experts, can understand. It further allows those experts to distill the insights and present them in a way that is easily digested by stakeholders, thus ensuring any resulting actions align with the strategic needs of the business.

It'll be interesting to see what actions the various streaming services take to address this issue moving forward. One thing is certain: They, like many other organizations, will have leveraged the power of analytics to inform and augment the decisions driving them.

JMP 16の新機能を用いたデータ分析　Part.2 データテーブルの比較　～データで変化を実感するために～

naohiro_masu — Tue, 06 Apr 2021 18:15:35 GMT

万物は常に変化していく。仏教でいう”諸行無常”の考え方ですが、別れと出会いが多いこの季節、変化をネガティブに捉えるのでなくポジティブに捉えることが重要です。世の中は日々良くなっているはずです。

新型コロナワクチンの投与は、日々世界が良い方向に向かっていると考えるべきでしょう。投与が進むことにより、感染が収束に近づいていく。そのことを日々実感するために、私は、国ごとのワクチンの投与状況（各国ごとに、人口100人中何人が投与したか）を可視化するレポートを公開しています。

COVID-19 Vaccinations Status by Country (by JMP Public)

https://public.jmp.com/packages/npMyYpl9Ns7m_SG7wJcLy

レポートを描くためのデータ　（https://github.com/owid/covid-19-data/tree/master/public/data/vaccinations）は日々更新されており、国、日付ごとに、ワクチンの投与数、投与人数、人口100人あたりの投与人数などが記録されています。

新しくデータを取り込んだとき、その前のデータとは何が変わったのかを知りたいことがあります。新しいデータでは、新しく投与開始した国が記録されている、ある国では、最新の日付に対する投与数が大幅に増加しているといった喜ばしい状況を知ると、世界が良い方向に変わっていることを実感できます。

執筆時点（2021-03-29）で取り込んだ最新のデータの一部を示します（以下、最新データと表記）。

このデータは、国(Country)と日付(Date)でソートされています。

一方、こちらは1日前（2021-03-28）のデータです。（以下、1日前データと表記）

データの行数は最新データが9,303件、1日前データが9,172件です。この1日の間に131件ものデータが追加されています。イタリア(Italy) に、2021-03-28のデータが追加されていることはわかりますが、当然、他の国々でもデータが追加されています。

私は、大人になってから新聞や雑誌にのっている”間違え探し”をやるのが億劫になってしまいました。何が違っているかを調べるには、JMP 16の「データテーブルの比較」を使ってみるしかありません。

JMP 16の「データテーブルの比較」

「データテーブルの比較」は、2つのデータテーブル（JMP形式）を比較して、相違点などをレポートする機能です。以前のバージョンからこの機能はありましたが、JMP 16では機能が大幅に刷新されました。

今回のようなデータでの比較をするのであれば、単純に2つのテーブルを行番号でマッチさせるわけにはいきません。国ごとのデータ行数は可変であり、得られた日付の数だけデータがあるからです。

以前のバージョンでは、このようなデータでの比較はできなかったのですが、JMP 16ではデータの揃え方についてID列を使用することによりできるようになりました。さらに、比較した後のレポートも非常にわかりやすくなっているので、実際にJMP 16で、最新データと1日前データを比較していきます。

データの揃え方を指定

比較する2つのデータテーブルをJMPで開いておき、最新データをアクティブにした状態で、[テーブル] > [データテーブルの比較]を選択します。

比較には最新データ、対象テーブルには1日前データが指定されています。「オプション」に表示される列の対応で、同じ名前の列が対応付けされているのがわかります。

その下にある「データの揃え方」が重要です。この例では、「ID列の使用」ボタンをクリックし、CountryとDateにチェックをいれます。すなわち、国と日付ごとに相違点を見ていくことを指定しているのです。*

指定後、右上の[比較]ボタンをクリックすることにより、比較のレポートが表示されます。

* ここでCountryだけ指定しても相違点としては同じレポートが表示されますが、日付にもチェックをしておくと、相違点のレポート上に日付も表示されるので比較のレポートが分かりやすくなります。

比較レポート

2つのデータテーブルを比較したレポートです。

赤色で塗られている行は、最新データには含まれていて、1日前のデータにはないものです。この例では該当する行が132行あることを示しています。

黄色で塗られているセルは、最新のデータと1日前のデータを比較したとき値が異なるセルです。この例では該当するセルが5,806あることを示しています。

上記のレポートで、オーストリア（Austria）を見てみましょう。赤色の行として 2021-3-27, 2021-3-28のデータが追加されていることがわかります。一方、2021-3-26以前のデータのセルの多くは黄色く塗りつぶされていて、データが修正されていることがわかります。レポート上で黄色のセルをクリックすると、レポート上側に「セルのデータ」に詳細が表示されます。この例では、オーストリアの2021-3-26の累積投与数は、1,492,002件から1,489,710件に修正されていることがわかります。

レポートをスクロールして他の相違点もみていきましょう。

こちらは、ガンビア(Gambia、西アフリカの国)のデータです。最新データで新たに追加された国です。2021-03-28に投与が開始され、人口100人あたり0.22人に投与されたことが分かります。ワクチンが国全体に行き渡るのはまだ先のことなのかもしれませんが、ガンビアにとっては大きな一歩です。

ちなみに、青色で塗られている行が1行ありますが、これは1日前データにはあるが、最新データにはない行を示します。

該当するのは、ルワンダ(Rwanda,東アフリカの国)の2021-03-25の行です。理由はわかりませんが、何かデータに不備等があり、最新データでは削除されたのかもしれません。

変化をデータで把握する

読者の方でも、今回のような日ごと、またはもっと短い間隔で更新されるデータがお持ちの方もいらっしゃるのではないでしょうか。製造の工程パラメータのデータ、臨床試験のデータ、時間ごとの特性値の推移をみる実験データ、大規模な消費者のアンケートデータなどさまざまな例が考えられ、今回紹介した「データテーブルの比較」がデータをチェックする用途でも役に立つかと思います。

万物は常に変化していく。だからこそ日々の生活は楽しい。そんなポジティブ思考を常に持っていきたいと考える筆者でした。

Model Screening — now you have it in JMP Pro 16

kemal_oflus — Mon, 29 Mar 2021 17:36:28 GMT

We were very happy to have had so much interest in the recent Technically Speaking featuring Model Screening, a powerful new feature in JMP Pro 16. When you’re building models, you never know which method will work the best on any given set of data.

With Model Screening, you can try multiple approaches at once, assess the model performance and choose the best-performing model, including customizing the decision thresholds according to your needs. This powerful new feature generated so many questions, we couldn’t answer them all. So, we are addressing them in this blog post. You can see the speed and power of Model Screening in this episode of Technically Speaking.

Kemal Oflus setting the stage to show the Model Screening platform.

General Questions

When we try to fit nonparametric models, we might achieve high predictability, but we lose interpretability of the model. How do we achieve that balance between predictability and interpretability in real-world manufacturing problems?

In this episode of Analytically Speaking, Galit Shmueli dives into the details of this topic with her much-cited paper, To Explain or to Predict. We have metrics like RSquare that summarizes model performance in terms of how much of the variance can be explained by the underlying model. A model with higher RSquare is technically better than a lower one. Model interpretability provides insight into the relationship between the inputs and the output. An interpreted model can answer questions as to why the independent features predict the dependent attribute. The issue arises because as model accuracy increases so does model complexity, at the cost of interpretability. For this case, the JMP Profiler helps us visualize the underlying model so we can use the subject matter expertise to verify the sanity and the feasibility of the model. A model with fewer parameters is easier to interpret. This is intuitive. A linear regression model has a coefficient per input feature and an intercept term. For example, you can look at each term and understand how they contribute to the output. Moving to logistic regression gives more power in terms of the underlying relationships that can be modeled at the expense of a function transform to the output that now too must be understood along with the coefficients. A decision tree (of modest size) may be understandable, a bagged decision tree requires a different perspective to interpret why an event is predicted to occur. Pushing further, the optimized blend of multiple models into a single prediction may be beyond meaningful or timely interpretation. Again, JMP Profiler is the perfect tool to visualize and interpret the models we build.

When assessing regression models, is there a standard range for a good RSquare metric that signifies a good model, or does it depend on the context?

This will depend on the context; if trying to predict the stock market, 55% RSquare will be more than adequate versus when you are trying to predict if a heart valve will fail.

What is the difference between the training set and the validation set? Can you go into more detail on what type of data scenarios would be in those sets?

The training set is what we use to learn from data (and build the model); the validation set is to validate what we learned from the training set. If there are enough data, it is a better practice to also have a test data set to further tune the model, then validate the results with the validation set.

Is there a reason to not use PLS?

No particular reason, just that it is harder to interpret and not really appropriate for the data sets that we looked at during the webinar.

Some assays have a larger error. Is there a way to enter expected error in the models?

If this is a question about using a classifier, by default the classification threshold is set at 50%, but depending on your circumstances, you can modify this threshold.

Regarding the sample on coffee, why did the neural boosted method have a lower sample size versus all the other models (which were identical to each other)? Did the RSquare value for neural boosted receive an advantage by selecting more appropriate data?

That is because a few of the selected input factors had missing values; some algorithms can handle the missing values and hence, use the rest of the information from the given observation. However, since neural networks cannot, a smaller number of observations are included in the sample.

How would one adjust and compare parameter settings of several models?

The model screening and comparison platforms provide a way to average/combine/aggregate multiple models so that you can create an ensemble model, which you can then use to optimize parameter settings.

It seems like this process might be prone to overfitting. How would you compare this with AIC?

AIC values are presented for the appropriate models in the individual model summary section of the model screening platform. The details in the online help cover this and more in the Model Comparison Report.

Doesn't the data scientist fill a critical role that JMP cannot? For example, the data scientist can balance between getting more true positives (TP) at the risk of increasing FP in a Classification problem. How does JMP deal with this tradeoff and communicate it to the modeler?

Software cannot replace the role of a subject matter expert (in this case, the data scientist), who is the bridge between the data and the applicable use cases. However, the tools and reports JMP provides with the model evaluation metrics (RSquare, adjusted RSquare, misclassification rate, confusion matrix and most importantly, the profiler) make communicating the findings to different levels of the organization extremely easy.

JMP Capabilities Questions

Can we do dose response logistic modeling in JMP?

Yes, this post in the Community and this one from JMP partner, Pega Analytics, cover more about how to do this in JMP.

Can we run agent-based modeling in JMP?

Not at this time.

Can the user specify values of model parameters or are default values assumed?

The Model Screening platform is designed to find the optimal parameter settings (for example, the number of trees, boosted layers, etc.). You can include interaction terms for linear models. Once the best model (or a set of models) is found, each model can be further customized.

Is there a validation column in your worksheet? Does this populate automatically?

If you have a validation column, you can utilize it. If not, JMP provides multiple ways to create a validation column. See options of how to do this.

Is there any way to get model analysis (such as Prediction Profiler) for a model created in Python if I put the data and predictions in JMP?

There is an Excel profiler in the Excel JMP add-in that can only be used for models created in Excel. However, JMP can execute your Python, R, or MATLAB scripts and/or exchange data back and forth, provided you have access to the software.

What about creating custom variables that might work best in a model created from raw data? Does this platform handle this or is it necessary to create these first to have them as inputs?

The Model Screening Platform uses the data as is; if you believe there should be some compound variables like Principal Components, they will have to be created separately. Simple interaction variables like a*b or quadratics can be selected via checkbox in the dialog box. Also, each node in the neural network also corresponds to a compound variable that might have a physical meaning (but again we do not have any control over those).

Can JMP incorporate a model created in R or Python?

If you have R or Python script that uses a model to predict results, you can have JMP run the R/Python script and use the results as a new column variable, if need be. This white paper, “JMP Synergies: Using JMP and JMP Pro With Python and R,” is a great resource.

Have you used JMP against textual data at all?

JMP has a built-in Text Explorer platform. This JMP Discovery Summit talk has some nice examples and the online help gives a good overview.

In Prediction Profiler, when we see behavior of y versus one of our x’s, is it established, considering that it is the only x or does it assume the behavior knowing there are other x's in the model as well?

The Profiler provides a way to investigate and optimize the relationship between multiple input variables to desired output variable. The input variables can be setup to be independent of each other or may be interacting with one another. This video shows how to use the Profiler. Our own Bradley Jones, Distinguished Research Fellow in R&D, wrote this JMP Foreword article in celebration of the 30-year anniversary of the Profiler, one of our most popular interactive visualizations.

What happened in Texas?

Peter_Polito — Thu, 25 Mar 2021 19:44:59 GMT

This past February, Texas experienced some of the coldest sustained temperatures on record. As many read in the news, this caused an unprecedented failure of Texas’ electrical grid, causing 4 million people to be without electricity at its peak. A cavalcade of problems resulted, ranging from municipal water systems failing, evacuations of hospitals, and most tragically, loss of life.

This article aims to focus on the data and shed light on what happened and when. But first, a few bits of information that will make it all easier to understand:

Texas’ power grid (which covers 90+% of the population in Texas) is not connected to the national grid.
ERCOT – the Electrical Reliability Council of Texas – manages the flow of electricity from producers to regional consumers (for example, Austin Energy, my power provider), who in turn control the flow to individual customers (for example, me).
Texas’ power grid is built to thrive in the summer. It gets hot here, really hot, and for a really long time. During the winter months, some of our power generation capacity goes offline because it just isn’t (normally) needed. Much of that generation takes weeks to months to bring back online.

Now that you have that background...

The storm

The cold weather began sweeping into the state on 10 Feb. In Austin, we had some freezing rain, blustery winds and falling temperatures. Schools closed because the roads were unsafe, but by and large, the grid held up. This first storm was a typical Texas winter front that we see maybe one or two times a year: windy, temps in the 20s, and a little ice.

The real hammer dropped on Valentine’s Day. In Austin, we started the day just below freezing (31°F at midnight) with temperatures dropping steadily all day to 14°F 24 hours later. It warmed up briefly into the low 20s but then dropped to 6°F by the morning of 16 Feb. This pattern, albeit with differing magnitudes and timing, was repeated throughout the state. On the morning of 15 Feb., Austin Energy customers began receiving calls that their power would be rolled, which means turned off for ~45 minutes and then back on for 15 minutes, to keep the grid from overloading yet allowing people to stay somewhat warm. However, in Austin and throughout the state, those who were rolled off were not rolled back on – for days.

Figure 1: A timeline of February storm in south-central Texas. Source: https://www.weather.gov/media/ewx/wxevents/ewx-20210218.pdf

It was not only cold, but it was cold for a long time. Austin Bergstrom airport recorded a temperature of 32°F or below for 164 hours (previous record was 112 hours)! Austin set a record for the coldest February temperature ever (6°F). Keep in mind that Austin is two times closer to the Tropic of Cancer than North Dakota!

Figure 2. Temperature readings at the international airports in Dallas-Fort Worth (DFW), Austin, and Houston.

The number of records eviscerated by this cold snap is quite impressive. The snowfall totals were unlike anything we’ve seen in decades. Austin received close to eight inches, just east of DFW received six and north of Houston – on the Gulf of Mexico – received up to four inches. Del Rio (a town west of Austin on the Rio Grande) recorded almost a foot of snow, beating its previous record by nearly 10 inches!

Figure 3: Snowfall on the morning of 15 Feb. as seen from my garage. By the end of the day, my kids had converted the exersaucer seen on the curb to a sled. Texans are gonna Texas.

At one point, the entire state of Texas was under a winter storm warning.

Figure 4: NOAA weather advisories on 14 Feb. 2021. Source: https://www.weather.gov/media/ewx/wxevents/ewx-20210218.pdf

Considering the magnitude of the cold, combined with its duration and the fact that much of Texas’ power generation is not winterized, it’s easy to see how and why things failed so catastrophically.

Failure to generate power

Figure 5: Four major sources of ERCOT power generation compared to total power generated. Major drops in in power generation occurred in the early morning hours of 15 Feb.

The proverbial wheels fell off at 2 a.m. on 15 Feb. Natural gas wellheads in the Permian Basin froze up. These failures, in conjunction with failures at coal-fired power plants, caused a 16% drop in statewide power generation. Three hours later, nuclear power dropped by 28% (2% of statewide power generation). These failures, coupled with an additional 2-3% statewide power generation drop due to frozen wind turbines, meant that Texans woke on 15 Feb. to some of the coldest statewide temperatures ever and with 20% less power. By the time temperatures rebounded statewide five days later, Texas had lost nearly 40% of its generation capacity (a peak of ~68 kMW on 14 Feb. and a min of ~42 kMW on 17 Feb.).

Figure 6: Temperatures at DFW, Austin, and Houston with power generation by type. Check out that Solar signal!

When we look at the power generated versus the load on the system in the JMP graph above, you can see how Texas came within less than five minutes of losing its entire grid. It was at the time of the two downward spikes on the early morning of 15 Feb. (Figure 7) that, statewide, the grid's frequency dropped dangerously below 60 Hz. Experts hypothesize that 4 minutes and 37 seconds longer at that low frequency would have fried ERCOT's entire network, taking weeks to months to bring it back online. (The Generation data was preliminary at publication time, which likely explains why the Total Generation minus Total Load is negative for so many days).

Figure 7: Power Load matched Power Generation nearly step for step. Large dropoffs in Load on 15 February were caused by loss of generation, not lower power usage.

How did we fare?

Oh my, we were so fortunate. We maintained electricity throughout the ordeal, and while we had to boil water for five days, we never completely lost water pressure. Maintaining our utilities made it possible for us to open our home up to friends without power (which translates to six adults, 11 kids, two dogs, and no COVID!) for four days.

Many in Austin and across Texas were much less fortunate. Reports of deaths caused by the outages have been trickling in over the weeks since. Jackson, MS, which received weather that was nearly as intense from the same storm, has been without water for a month! One hopes this will motivate the state (and other states) to improve the infrastructure to ensure this does not happen again.

Data presented above is available at JMP Public.

JMP 16の新機能を用いたデータ分析　 Part.1 アクションレコーダー～記憶でなく、記録に残るように～

naohiro_masu — Mon, 22 Mar 2021 20:03:17 GMT

私個人的に、“記録より、記憶に残るスポーツ選手”　は数字で表せないヒーロー性があって好きですが、今回の話題は、”記憶でなく、記録に残る” 重要性をお話します。私が社会人になったころ、新人研修で上司に呼ばれたときは、必ずメモをもっていき、言われたことを記録しろと教育を受けたものです。

データ分析作業では、データ加工の作業に手間がかかることが良くあります。JMPは加工する機能が充実していて有難いですが、しばらく経って、加工したデータを開いてみると、どんなデータ加工をしたのか記憶に残っていないことがあります。そんなとき、加工の一連の流れをメモにでも記録しておけばよかったなあと後悔します。

JMP 16では、アクションレコーダーという新機能があり、列名の変更、尺度の変更、列や行の削除などデータテーブルに対する一連の操作を記録として残しておくことができます。*

この機能を使った分析例を示します。

今年2月中旬から、日本でも新型コロナワクチンの接種が開始されました。厚生労働省のホームページには、接種の実績に関するデータがあります。（https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/vaccine_sesshujisseki.html ）

JMPでこのページにあるデータを読み込み、日付ごとに累積接種回数、施設数を示す棒グラフ、折れ線グラフを作成することが目的です。

*注意：記録として残せないデータテーブルの操作もいくつかあります。

データの読み込みとデータテーブル操作の記録

JMPでは、[インターネットから開く] メニューより、URLを指定してデータを読み込むことができます。下図は、この機能を使って読み込んだときの一例です。（注意：実際にこの例では、列名の行とデータの開始行を変更したスクリプトを実行しています。）

データが読み込まれましたが、このままでは目標とするグラフを描くことができません。グラフ作成用のデータにするために、次のようなデータテーブルに対する操作を行います。

列「施設数(*)」の列名を「施設数」に変更し、連続尺度に変更する。
最後の合計行（この例では23行目）を削除する。
列「1回目」、「2回目」は、その日の接種回数のデータのため、これらの累積接種回数を示す新しい列を作成する。
連続尺度の列の値の表示形式をカンマ区切りにする。

JMP 16では、上記のデータテーブルに対する操作を行うと、その操作に対するスクリプトが自動的にログに出力されます。下図のログには、データを開き、データテーブルに対する操作が記録されています。

上図で選択されている「列情報の変更：施設数」は、列「施設数(*)」の列名を「施設数」に変更し、連続尺度に変更する操作をしたときの記録になります。ウィンドウ下側には、この操作に対するスクリプトが表示されています。

一連の操作で、次のようなグラフ作成用のデータテーブルを作成できました。

分析結果の記録とスクリプトの実行

作成したデータテーブルから、グラフビルダーを使って、日ごとの累積接種回数、施設数のグラフを作成することができます。接種施設、接種回数とも大幅に増加していますが、日本全体の人口を考えると、まだまだこれからといった感じですね。

グラフビルダーのレポートウィンドウを閉じると、先ほどのログに、グラフビルダーに関するスクリプトが追加されます。

ここまでで作成されたログをすべて選択し、左上の赤い三角ボタンから、[スクリプトの実行] を選択すると、データを読み込む、データを作成する、グラフを作成する　といった分析の一連の操作を再実行することができます。

また、ログの左上にある赤い三角ボタンから [スクリプトの保存] > [スクリプトウィンドウへ] を選択すると、次のように、スクリプトウィンドウに自動作成されたスクリプトを書きだすことができます。

このファイルをJMPスクリプト形式（拡張子 *.jsl）で保存しておけば、データが新しくなっても、このスクリプトを実行することにより、新しいデータで、分析結果を作成することができてしまうのです。驚きですね！

作成されたスクリプトを少し変更して汎用性があるものに

ただし、作成されたスクリプトを新しいデータにそのまま利用できないケースもあります。今回のワクチン接種データは、日々データが追加されます。一連の操作には、最後の「合計」の行を削除するものがありますが、以下のスクリプトを参照すると、23行目を削除することとして記録されています。

もう1日分データが増えれば、ここを24行目にしないといけませんし、より一般的には、データテーブルの最後の行を削除するスクリプトにしないといけません。その場合は、どうしても分析者がスクリプトを汎用性のあるものに修正する必要があります。

一連の操作で作成したスクリプトを少しだけ修正して、汎用性のあるスクリプトにしたものを、このブログの最後に示しています。

データの読み込みから、データ作成、分析まで記録に残せる”アクションリコーダー" の機能があることを、是非とも記憶して頂き、JMPの操作を記録してみてください。

// ワクチン接種データからグラフを作成するスクリプト
//:　HTMLデータの読み込み */
dt = Open(
	"https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/vaccine_sesshujisseki.html",
	HTML Table( 1, Column Names( 1 ), Data Starts( 2 ) )
);

// 列情報の変更: 施設数
dt:"施設数(*)"n << Data Type( Numeric ) <<
Set Name( "施設数" ) << Set Modeling Type( "Continuous" ) << Set Field Width( 12 );

// 選択されている行を削除
// Data Table( "vaccine_sesshujisseki.html" ) << Clear Select << Select Rows( [23] )
//  << Delete Rows;
dt << Delete Row(Nrow(dt));

// 計算式列の新規作成
dt <<
New Column( "累積[内１回目]",
	Numeric,
	"Continuous",
	Format( "Best", 12 ),
	Formula( Col Cumulative Sum( :内１回目 ) )
) << New Column( "累積[内２回目]",
	Numeric,
	"Continuous",
	Format( "Best", 12 ),
	Formula( Col Cumulative Sum( :内２回目 ) )
) << Move Selected Columns( {:"累積[内１回目]"n, :"累積[内２回目]"n}, after( :内２回目 ) );

// 列属性の一括設定
Local( {old dt = Current Data Table()},
	Current Data Table( dt );
	For Each( {col, index}, {:接種回数, :内１回目, :内２回目, :"累積[内１回目]"n, :"累積[内２回目]"n, :施設数},
		col << Format( "Best", Use thousands separator( 1 ), 13 )
	);
	Current Data Table( old dt );
);

// レポートのスナップショット: vaccine_sesshujisseki.html - グラフビルダー
dt <<
Graph Builder(
	Size( 846, 555 ),
	Show Control Panel( 0 ),
	Show Title( 0 ),
	Variables(
		X( :日付 ),
		Y( :"累積[内１回目]"n ),
		Y( :"累積[内２回目]"n, Position( 1 ) ),
		Y( :施設数 )
	),
	Elements(
		Position( 1, 1 ),
		Bar( X, Y( 1 ), Y( 2 ), Legend( 4 ), Bar Style( "Stacked" ) )
	),
	Elements( Position( 1, 2 ), Line( X, Y, Legend( 6 ) ) ),
	SendToReport(
		Dispatch(
			{},
			"Graph Builder",
			OutlineBox,
			{Set Title( "日本の新型コロナワクチン接種実績" ), Image Export Display( 通常 )}
		),
		Dispatch(
			{},
			"累積[内１回目]",
			ScaleBox,
			{Format( "Best", Use thousands separator( 1 ), 12 )}
		),
		Dispatch(
			{},
			"400",
			ScaleBox,
			{Legend Model(
				6,
				Properties( 0, {Line Color( 70 )}, Item ID( "施設数", 1 ) )
			)}
		),
		Dispatch( {}, "Y title", TextEditBox, {Set Text( "累積接種回数" )} ),
		Dispatch( {}, "Y 1 title", TextEditBox, {Rotate Text( "Left" )} )
	)
);

Elements of mature cultures of analytics

J_Marquardt — Thu, 18 Mar 2021 19:09:00 GMT

Sarah Kalicin, Intel, joined a panel on building a culture of analytic excellence last year. Join us March 25 for more on this topic.How do Intel, P&G and Seagate embed analytics throughout the organization? Sarah Kalicin is a data scientist and industrial statistician at Intel -- and a proponent of spreading analytic excellence. You can watch Sarah and her fellow panelists have a complete discussion about this topic. Here's a preview of her perspective.

“It’s really about asking questions and how do you know that you have data that can answer those questions? So that’s analytic maturity.

"But in order to show people that they can do this, you need to have data discipline. Because nobody is going to get data and spend hours going through the data and making sure that it’s correct, it’s clean, you can trust it. In fact, if you have everyone doing that, it’s kind of a waste of time.

"You really need a central organization that’s focusing on that so essentially everyone can have access to the data and start asking questions.”

Want to watch the complete panel discussion? Find it here. We also welcome you to join us for different ideas, same topic, when we meet with experts from Unilever, Dow and Riffyn on March 25. Sign up to watch.

The Great Virtual Cookie Experiment: The results

Ryan_Lekivetz — Wed, 17 Mar 2021 16:36:52 GMT

The results are in!

At the last social event of our online analytics conference JMP Discovery Summit Europe 2021, attendees voted on a series of pairs of photos of cookies. Each pair of photos were taken by volunteers according to instructions based on a designed experiment. For more details about the experiment, there is a separate blog post outlining it.

The attendees needed to pick which of the two pictures tempted them more from each set, picture A or picture B.

While not immediately known to the attendees, the factors being changed by our photographers were:

Beverage: Milk / Tea / Coffee
Type of Plate: Solid / Pattern
Setting: Inside / Outside
Number of Cookies on Plate: 2 / 6
Image: Color / Black & White
Use a Prop: Yes / No

Many thanks to Clay Barker for helping collect data during the social and prepare the analysis. He set up the data set to be used with the Bradley-Terry model as has been seen in a couple of previous posts (here and here). Full disclosure: Since the data was being collected real-time and Clay had percentages available, the percentage of each pair of pictures was entered. So it pretends we had 100 respondents, when we had between 70 and 80 at any given time. Modeling this as the probability that picture A would be selected, during the social Clay showed us the main effects model in JMP:

What this tells us is that based on the respondents, six cookies are better than two (which came up in the discussion), color is preferable to black and white, and there might be a small effect by using a prop.

After the social, considering interactions and going through some model refinements, we ended up with this model:

Still much the same story – six cookies and a color image are the main drivers, but milk/coffee/tea and the outside setting can be, depending on what you are comparing it to. The interactions factor in what the paired picture has, so it can be a bit more difficult to see from here. If you want to dig more into the details, I have attached the original data table and the one set up for the Bradley-Terry model.

Clay was kind enough to set up the Profiler as well so that you can compare the probability of choosing picture A given settings as defined by pictures A and B. If you want to explore the model, the interactive Profiler is available on JMP Public.

According to our model, these two pictures should look tempting. Let me know in the comments if those plates of cookies indeed look good to you (or not so good).

Final thoughts

It would have been interesting to collect some demographic information to go along with the results, but because the event was virtual, we wanted to keep it as straightforward as possible.

If we ever contemplate an experiment like this again, I would change how we handle the prop, which was left very open-ended for this experiment. Being much more specific about the type of prop would give a better idea of whether using a prop makes a difference.

Attached to this post are a couple of versions of the data for analysis and a set of slides with the pictures, if you are interested.

The Great Virtual Cookie Experiment: The experiment setup

Ryan_Lekivetz — Wed, 17 Mar 2021 16:39:57 GMT

For the last social event of JMP Discovery Summit Europe online, we wanted to give attendees a fun interactive experience.

We decided to create an online experiment involving cookies. Attendees took part in an experiment exploring what they find more tempting when looking at a plate of cookies.

This post is focused on the setup of that experiment. If you do not want all the details of the experiment and just want to see the results, go here.

Inputs and Responses

Even after deciding to do a cookie experiment, there were still many different factors we could change. But we realized that we needed to keep it simple since we were going to be asking volunteers to help with baking cookies and taking pictures. We stuck with one type of cookie, and produced a list of different factors that could be changed when taking a picture:

Beverage: Milk / Tea / Coffee
Type of Plate: Solid / Pattern
Setting: Inside / Outside
Number of Cookies on Plate: 2 / 6
Image: Color / Black & White
Use a Prop: Yes / No

While we can define these inputs, a proper response is difficult. A continuous response is difficult, particularly if we give a large set of photos (that is, how do you rate this out of 10?). We knew that we were going to call out for help, asking for volunteers to bake cookies and take pictures according to our instructions. We were not sure how many volunteers would help. And there was also the issue that, even with the same set of instructions, individuals would take quite different pictures. On top of that, we did not want volunteers to be burdened with taking too many photos when they could be enjoying their cookies instead.

If the experiment was going to involve comparing pictures, they should be by the same photographer. Asking each volunteer to take a pair of photos and having attendees pick their favorite seemed reasonable. And since there is no preconceived belief of any effect sizes or directions, I liked the flexibility of using Custom Design to set this up, as we have done before.

In JMP, from DOE->Custom Design, we specified the factors and levels:

To create the “pairs of pictures” for each volunteer, we chose random blocks of size 2 that would give the settings for their two pictures. We initially set this up for 16 runs (that is, eight pairs of pictures), and main effects only, not knowing how many volunteers to expect.

Making the design, we had our original set ready to go.

The Curious Case of the Augmented Pairs

As it turned out, after a slow start, we ended up with many more volunteers than anticipated. This means we needed to augment the design. Because the first design was set up in Custom Design, Augment Design recognizes that random blocks of size 2 are required when selecting that column as one of the inputs when launching the platform:

In the original design, where main effects were specified, the pairs were effectively mirror-image pairs (except for beverage, with 3 levels); for each pair of photo instructions, whatever was used for the first picture would be the opposite for the second picture (that is, mirrored).

With the new set of volunteers, there were enough to specify the interactions. Even though we were looking at analyzing this as a choice model, this should still provide better exploration of the design space. To augment, we specifed 42 runs, which would provide photograph settings for the additional volunteers (there were a few rounds of augmentation, and a few batches that did not get made):

In the augmented design, what you may find interesting is that for many of the new pairs, only one or two of the factors change. That is, the photo instructions to a volunteer were quite similar. Why is it so different from the first set? The key is in the interactions. If all the factors are switched in the same set, then the interaction will have the same sign (think of this from the -/+ 1 perspective). Viewed in this way, it makes sense that those extra random blocks are different than the first set since we have now asked for the interactions.

Attached are the original design and the final design used (that needed a few modifications based on what the photographers had on hand).

Thanks for reading!

Sample Size Explorers: A new approach to an age-old question

calking — Thu, 04 Mar 2021 18:42:31 GMT

It begins innocently enough. You’re sitting in your posh office, happily typing away on your computer, powering through your latest awe-inspiring statistical analysis on your favorite statistics software (which is JMP, of course). You’re in your happy place.

There's a knock at the door. You look up to see a client standing with a stack of papers by their side. Always eager for a new problem, you ask politely, “Hey, there! How can I help?”.

“I have a quick question,” they reply.

“Go for it!” you quip naively.

And then it happens. The words shoot from their lips like arrows toward your unwitting ears.

“How many samples do I need to-?”

“GAAAAAHHHHH!!!!”

"I can only afford five. Is that enough?"

"GYAAAAAAAAAAHHHHHHHHHH!!!!!!!"

OK, so maybe I’m being a bit over-dramatic here, but many of us know this seemingly simple question – how many samples do I need? – can be a very difficult one to answer. There’s a lot of extra information you need to factor in. What effect size are you looking for? How “noisy” do you think the population is? What type of data will be collected and how?

While the answer may not be as straightforward as researchers would like, it is a vitally important one to ensure that resources are used efficiently. Too little data means you can’t make valid inferences. Too much data is wasteful. So how do you find the right balance?

The most common tool for answering this question is a power calculation. As a refresher, statistical power is the probability of detecting an effect of a certain size assuming it actually exists. One important tool available in JMP to help with this question is the Power and Sample Size Calculator. It gives you calculators for common scenarios, enabling you to generate sample sizes and/or power with ease. In some cases, you can even generate a power curve to explore the power/sample size relationship.

The Power and Sample Size Calculator is certainly a useful tool. But, to borrow a phrase from a recent film character, while it is good, it can be better. To start, it is a calculator. It's purpose is to get you an answer quickly. But power calculations should almost never be one-and-done. As with design of experiments in general, a key component to power and sample size calculations is exploration of several options. The best test plans are the result of considering different scenarios, ensuring your study is robust to unforeseen circumstances. That's not necessarily easy to do with a calculator. Something new is called for...

Introducing the new Sample Size Explorers in JMP 16!! Take a look (ooh's and aah's are welcome):

So what's so new? Well, for one, it's there in the title. Sample Size Explorers are created not just to help with power calculations, but to encourage exploration of the relationship between power, sample size, and other key metrics involved. Let's dive in and see what else is new!

It's All About Interaction

First, there's an improved layout that's all about interactivity. On the leftmost side, you have your control panel. This area contains all the information you need to set up your calculations. Are you doing a one-sided test or two-sided? Is this a two-sided interval or a one-sided bound? What's the alpha? Sigma? Demonstration time? If you know it, this is where you put it. And if you don't know it, that's OK. You can start with the default values provided and change them later.

Next, just to the right of the control panel, is the interactive graph. Yes, you read that right. This isn't just a simple visualization; it's OK to touch this picture. See that white square attached to the red line? You can click that "handle" as we call it in the biz and drag the crosshairs up and down the curve. The power and sample size boxes will update accordingly. Or, if you're the daring, adventurous type, you can click anywhere in the graph and watch the handle come to you! The curve will also change based on the other inputs you give into the control panel. In short, you have multiple ways to explore different scenarios, each with their own level of interactivity.

Finally, on the far right is the saved settings table. "But Caleb," you might say, "there's no table there!" True, there's nothing visible...yet! If you look in the bottom left corner, you'll see a button labeled "Save Settings". As you can probably guess, if you were to click that button, you'll get the following result:

Yep, anytime you want to save a particular scenario, simply click that button, and it will add another row to that table. Which, by the way, isn't just any ordinary table. You'll notice that one of the rows is highlighted. If you click another row, it becomes highlighted, and everything changes to reflect that scenario! So now, you can save several scenarios of interest for exploration later. You can even export the scenarios into their own data table through the red triangle menu!

A Wide Array of Possibilities

With Sample Size Explorers, power exploration is just the tip of the iceberg. There are explorers for interval calculation and even reliability demonstration. In fact, here's a list of all the available Sample Size Explorers coming out in JMP 16:

Power
- Power for One Sample Mean
- Power for One Sample Proportion
- Power for One Sample Variance
- Power for One Sample Equivalence of Means
- Power for Two Independent Sample Means
- Power for Two Independent Sample Proportions
- Power for Two Independent Sample Variances
- Power for Two Independent Samples Equivalence of Means
- Power for ANOVA
Confidence Intervals
- Margin of Error for One Sample Mean
- Margin of Error for One Sample Proportion
- Margin of Error for One Sample Variance
- Margin of Error for Two Independent Sample Means
- Margin of Error for Two Independent Sample Proportions
- Margin of Error for Two Independent Sample Variances
Reliability Demonstration
- Parametric Reliability Demonstration
- Nonparametric Reliability Demonstration

At JMP, we're all about statistical discovery, and Sample Size Explorers are yet another tool to help you along the way. So what are you waiting for (besides your chance to update to JMP 16)? Get out there and start exploring!!

Outliers Episode 4: Detecting outliers using jackknife distance

JerryFish — Wed, 17 Mar 2021 16:59:53 GMT

Welcome back to my blog series on Outliers. In previous episodes, we’ve looked at:

Today after a quick (but necessary) review of Mahalanobis, we will look at the jackknife distance.

Quick review of the Mahalanobis distance

Remember the three-dimensional, 10-point data set from Episode 3? The multivariate Scatterplot Matrix looked like this in JMP:

Figure 1: Scatterplot matrix of 10 data points with one outlier (red)

The red point appears to be an outlier when compared to the other nine points. This is particularly clear in the highlighted plot.

Yet when we performed a Mahalanobis distance calculation, the red point did not appear to be a “terrible” outlier:

Figure 2: Mahalanobis distance for 10 data point sample shown in Figure 1

In Figure 2, the red point is above the line of significance, but not by much. This seems counterintuitive based on the Scatterplot Matrix of Figure 1 where the red point appears to be an obvious outlier.

But remember, the Mahalanobis calculation involves measuring the distance from the whole when taking the covariance structure of the data into account. If we take the highlighted scatterplot from Figure 1 and fit a straight line to it, we get something like this (from Graph Builder):

Figure 3: Simple plot of X1 vs X2, including best-fit line (from ordinary least squares)

The straight-line fit is a visual representation of the covariance structure between these two variables (X1 and X2). Note that the red outlier point is a heavy influence on the position of the line; it pulls the line up toward it, making the distance from itself to the line smaller than it should have been.

In addition, the poorly fitting line also makes it appear that other data points are farther from the line than they might really be if the red point hadn’t influenced it so strongly. For example, data point 6 (labeled in Figure 3) appears to be far from the line and might be on the border of being an outlier if this particular best fit line was chosen.

Is there a better way to detect the outlier?

The jackknife technique

The jackknife technique is very simple, yet very powerful, relying on calculations using the “leave one out” technique. Conceptually, if we left the red outlier out of the line fit calculations, it would generate a scatterplot with best fit line that looks like this:

Figure 4: Simple plot of X1 vs X2, including best fit line based on all points EXCEPT for red outlier

Now it is clear that the red outlier is truly considered an outlier, warranting further investigation.

In general, recall this equation for computing the Mahalanobis Distance from Episode 3:

To compute the jackknife distance, use the “leave-one-out” technique and calculate the

vector and the

matrix for all points except for the point of interest. Next, simply put the point of interest into the

vector and compute the Mahalanobis distance, which is now called the jackknife distance.

Repeat the calculations for all points, each time leaving the point of interest

out of the calculations of

and

The jackknife plot then looks like this:

Figure 5: Jackknife distance plot for 10 point sample

Now it is quite clear that the red point is an outlier.

History of jackknife

Per Wikipedia, John Tukey came up with the name “jackknife” because it is a robust multipurpose tool that can be used in a variety of circumstances, though the technique was originally developed by Maurice Quenouille in the early-mid 1950s.

Parallels with other techniques

If you’re familiar with other validation schemes (like K-fold validation) or things like bootstrap forest and boosted tree techniques, you’ll recognize that leaving one (or more) observations out of an analysis is a very common and powerful way to get more out of your statistical analyses.

Next episode

Way back in Episode 1, I mentioned that the Episode 5 would involve T²distances. However, I already pointed out in Episode 3 that T² is simply the square of the Mahalanobis distance, so we don’t need another episode to discuss that.

Instead, my fifth and final blog post on outliers will address multivariate K-nearest neighbor detection of outliers. Stay tuned!

The Great Virtual Cookie Experiment: Coming to a screen near you

Ryan_Lekivetz — Wed, 17 Mar 2021 16:38:47 GMT

In case you haven’t heard… our live events for Discovery Summit Europe are coming on March 8-12. You can even start looking at the papers now (shameless plug).

While the social events are always one of the highlights of any Discovery Summit, the virtual nature of this year’s events make the interaction different. We wanted to finish this Discovery Summit Europe with a fun experiment, but how to do it virtually? The idea of cookies came up but trying to taste cookies (or anything) virtually seemed a bit difficult. How about determining what makes cookies look more tempting?

How can we do that?

We gathered a fantastic set of volunteers who were willing to bake cookies and take a pair of photos according to a set of instructions that were determined using a designed experiment. For those of you into design of experiments, in a future blog I’ll provide more details on how we created the design. The idea is that we picked a set of factors to change in each photo.

However, to get the most information from this experiment, we need YOU. The more data we gather on picture preferences, the more interesting our results for seeing which factors drives preference in the cookie pictures.

Do I need to know about designed experiments or statistics?

Absolutely not! All we need is some willingness to join us for the final social and look at several sets of cookies. And be able to tell us which plate you would rather grab a cookie from. For example, would you rather take from A or B?

If you want to join, make sure you’re registered for Discovery Summit Europe here and attend the Social on Friday, March 12 at 17:00-18:00 CET!

Analyzing spectroscopic data: Pre-processing

Bill_Worley — Mon, 07 Jun 2021 21:26:28 GMT

My colleague Jeremy Ash () and I realize that analyzing spectroscopic data has many nuances and potential pitfalls that can make analysis difficult and messy at best. So we wanted to write this blog series to describe how you can import, visualize, clean, and analyze spectroscopic data with JMP software.

To make it easier for you to try out some of these steps on your own data, we have created the Spectral Tools add-in, which you can learn more about and download here in the JMP Community. This is a simple add-in that streamlines some of the data import and visualization methods discussed in this blog post. It is only a prototype, and only a first step toward providing some convenient functions for spectral data. You can give us feedback in the comments of this blog post or on the add-in page.

In this post, we are analyzing a near-infrared data set from Martens et al. "Light Scattering and light absorbance separated by extended multiplicative signal correction. Application to near-infrared transmission analysis of powder mixtures" Analytical Chemistry 2003 Feb1;75(3):394-404. We will show how pre-processing improves the overall quality of the data and allows you to see, in this case, grouping based on proportions of constituents much more clearly.

Importing spectra

When importing spectra, the Spectral Tools add-in assumes that each spectra is in a separate file in a single directory. The spectra can be in delimited text files (.csv or .tsv, for example) or JCAMP-DX files. Only JCAMP-DX without compression is supported. The stacked format is the default data table format in Spectra Tools. You can also import files in wide format. In this blog post, we most frequently use the stacked format, but a few of the pre-processing steps require the wide format. You can convert between the stacked and wide formats using the Stack and Split commands in the Tables menu in JMP.

Data visualization

As with any new data, once you get it into your analytics software of choice, we highly recommend visualizing the state of the data, and regularly comparing to this first look after any pre-processing is done. In JMP, we use the Graph Builder platform to help with the visualization.

Figure 1 shows a line plot of the spectra in Graph Builder. With your data in the stacked format, drag the X and Y variables to their respective axis, and use the spectra ID as the overlay variable. Alternatively, you can use the Launch Graph Builder command in Spectra Tools. For the color variable, we use the response variable of interest, which is gluten content.

Figure 1. Spectra line plot in Graph Builder with spectra colored by gluten content.

To zoom into regions of interest (ROIs), click and drag the X axis. You can create a Local Data Filter in the red triangle menu for a few other useful controls. This is provided by default in Spectral Tools. These are shown on the left of Figure 1. The wavelength local data filter can be used to focus on a region of interest (ROI). To create a new data table for just the ROI data, go to the local data filter red triangle and select Show Subset. Also, to only show a subset of the spectra on the graph, select the spectra in the spectra ID Local Data Filter. You can also use the Animation Controls to cycle through one spectra at a time.

For more complicated selections like multiple ROIs, you can plot the spectra as points and select the points of interest. The Subset command in the Table menu can be used to create a data table with just the ROI data.

One important thing to remember about Graph Builder is that once you have finished setting up the graph the way you want it, you can save the script to the data table, so that the same plots can be easily recreated with new data. For example, you may wish to recreate your graph when comparing spectra before and after pre-processing. Another option for comparing spectra before and after pre-processing is to use the Launch Graph Builder command in Spectral Tools.

Also, new in JMP 16 in Graph Builder, Savitzky-Golay (SG) smoothing can be applied as an initial pre-processing step (Figure 2). Once you select your tuning parameters, you can output the smoothed data. Go to the smoother red triangle and select Save Formula. Note that this will require your wavelength to be a continuous variable (you can modify the variable type if you need using the Column Properties).

Figure 2. Savitzky-Golay smoothed spectra in Graph Builder.

In Graph Builder, there are alternate smoothing methods – such as smoothing splines and LOESS – that you may also want to try out. There are also alternate weight functions in the smoother red triangle menu. Many of these options were newly added in JMP 16.

Other summary measures that will quantify the quality of spectra are accessible in Graph Builder. To create these, remove the overlay variable so that the mean spectra is shown. Then, selecting the bar plot and box plot options will generate the plots in Figure 3. Variation that is constant across wavelengths often indicates noise – and a need for further pre-processing – whereas large variation that is specific to certain wavelengths may indicate real chemical effects.

Figure 3. Visualization of spectra variation in Graph Builder. (Top) Mean absorbance barplots at each wavelength with standard error. (Bottom) Absorbance boxplots at each wavelength.

One other type of plot that is useful if your data is in wide format is the parallel plot (Figure 4). This is particularly useful if you are analyzing the spectra in other platforms that require the wide format such as PCA, PLS or other predictive modeling platforms. Selecting spectra in the parallel plot will highlight the corresponding rows in other platforms:

Figure 4. Graph Builder parallel plot of spectra colored by gluten content.

Pre-processing

One of the major stumbling blocks with analyzing spectral data is the need for complex data pre-processing. We will show how using an add-in for Savitsky-Golay derivative filtering and a formula for standard normal variate (SNV) scatter correction can “clean up” your data, making it more easily interpretable and ultimately allowing you to build more reliable predictive models.

For the first step, we will remove the apparent baseline shifts in the spectra. For this, we use a SG data filtering add-in developed by Ian Cox (see the add-in attached to this post below). This will only accept data in a wide format. Choose the “Data Filtering” add-in from the list of installed add-ins in JMP. Select the columns to be manipulated and select OK. Three output graphs will pop up for review.

The SG smoother user interface is where the polynomial order of fit, 2 – 8) is defined, and the length of the right and left edges of the smoothing window are also defined. Changing any of these values automatically updates the graphs, since the derivatives are performed on the SG smoothed spectra. Since derivative filters amplify noise, a certain degree of smoothing is often required. Figure 5 shows the output graphs.

Figure 5. (Top) Savitzky-Golay smoother, (Middle) first derivative filter, (Bottom) second derivative filter.

The filtered data can be saved to a new data table, which can then be transferred back to the original data table for further pre-processing if desired. In this case, the first derivative data was selected and saved back to the original data table.

This appeared to effectively remove the baseline shifts. However, there is still some scatter effects remaining in the spectra. One way to visually separate out the scatter effects and chemical effects is to use a grouping variable in Graph Builder. Move the gluten content variable to the one of the group axes. Graph Builder will find a sensible binning and create separate panels for each bin. In these data, we only have 5 values of gluten content. Scatter effects remain because the first derivative removed the additive baseline shift, but did not remove the multiplicative scatter effect. We will show how to illustrate multiplicative scatter effects with a scatter effects plot in the next blog post.

Figure 6. Spectra line plots with spectra separated by gluten content. Variation due to scatter effects can be seen in these plots.

To remove the remaining scatter effect, we perform a SNV standardization. SNV requires the data to be in a stacked format. We create a new column and add the formula shown in Figure 7.

Figure 7. Formula used to perform the SNV.

The raw spectra and final pre-processed spectra are shown in Figure 8. To easily launch a plot like this, use the Launch Graph Builder command in Spectra Tools.

These simple steps have already resulted in a much easier to interpret spectra, with baseline shifts and multiplicative scatter effects removed. In their original paper, Martens et al. demonstrated how the more advanced extended multiplicative signal correction can further improve on this pre-processing workflow. We will cover this in the next post.

Figure 8. Spectra before and after the pre-processing steps applied in this blog. (Top) Before pre-processing, (Middle) after Savitzky-Golay first derivative filter, (Bottom) after standard normal variate.

Future posts

We plan two more posts in this series. The next will show some more advanced pre-processing methods like multiplicative scatter correction and baseline subtraction.

We will also demonstrate how dynamic time warping in Functional Data Explorer can be used to align chromatograms. The final post will apply some of the predictive modeling methods in JMP, such as PCA, PLS, and Generalized Regression, to spectra after pre-processing.

Try some of these approaches out, and let us know how they work with your data!

Discovery Summit Europe 2021 starts now!

Diana_Levey — Fri, 26 Feb 2021 17:38:13 GMT

Technically, the conference happens the week of 8 March. And as you can see from the agenda, there are plenty of reasons to block off time on your calendar during that week to attend.

So why do we say it starts now? Because some of the most important content of the conference is available right now: the papers!

What’s in a paper?

You can check out Discovery Summit Europe papers right now.Simply put, stories. In Discovery Summit's myriad papers, JMP users tell their stories of data discovery: the problems that had them stumped, the statistical solutions they tried and the results that saved them time, money, even lives.

You’ll also find papers presented by members of the JMP R&D team. These presentations often take a different story arc, offering a behind-the-scenes look at new features and functionality in the upcoming release. No matter the presenter, no matter the topic, each paper has been judged and selected by the Steering Committee.

In this new world of online meetings and virtual conferences, we learned quickly that hours and hours of watching videos gets old really fast. So, we are making Discovery paper presentations available now for you to take in at your preferred pace: binge watch papers like your favorite TV shows, or space them out, savor them and give them time to sink in. Discovery Summit papers are ready when you are. Check them out in the Discovery Summit section of community.jmp.com.

The best is yet to come

As awesome as learning from recorded papers can be, we have something even better planned for the week of 8 March.

When we’re together online and in real time, all of the sessions will be interactive.

We have built in opportunities to ask questions of the JMP development staff, of our keynote presenters, and of other attendees. We’ll even run select papers as part of the conference in what we call the simu-live format. That means the paper presenters will be watching with you and other attendees, and available to answer questions and add commentary. Check the agenda for exact dates and times of these presentations.

Tell a friend: Registration is free!

Even though the content continues to be unequaled, the Discovery dynamic is different online. One way to help drive home the lessons learned, we find, is to make sure co-workers attend, too. Together you can share best practices you picked up in conference conversations, rewatch the papers that are most relevant to your work, discuss a statistical strategy one of you caught but the other missed. You get the idea. There’s power in numbers. And there’s no better time than when the conference is free of charge and free from travel. Recruit a friend to register now!

In the meantime, start your Discovery Summit Europe experience now by visiting the library of recorded paper and poster presentations.

What is a covariate in design of experiments?

Ryan_Lekivetz — Fri, 26 Feb 2021 20:19:05 GMT

Throughout my time at JMP, I have had many design problems that needed the use of covariates, both for my own problems and to help customers find a solution to their design problem. What I have noticed is that many customers, even very experienced ones, are not aware of the covariate option in the Custom Design platform.

To start off, I want to point out it has lived right in the Add Factor dropdown in Custom Design since JMP 10, and you may not have even noticed until now:

What is a covariate?

Part of the confusion stems from the use of the term covariate. In some contexts, you will see it used as an effect to control for, but not of primary interest (ANCOVA), and you may even see it loosely used for any independent factor in defining a model.

In a designed experiment, a covariate is an input variable that we want to account for in our experiment but we cannot control it to be any value in the way we can for other types of factors. However, if we can measure the values of such inputs ahead of time, we can account for them when designing the experiment.

When would I use a covariate?

If you think of a covariate from the standpoint of “uncontrolled, but observable ahead of time,” there are a few different use cases that come up. You may often hear the idea of a candidate set. In the Custom Designer, specifying the covariate factors produces a “candidate set” of runs for the Custom Designer to use. In Custom Design, the candidate set is specified from a data table. Very often, we have additional controllable factors (that can take on any value in the range), that we can allow the Custom Designer to pick as it sees fit (the way we usually define factors).

Broadly speaking, I tend to break the use of covariates into two cases:

Using a subset of the rows.
Using all the rows.

Using a subset of rows

When using the Custom Designer to select a subset of the covariates set, the idea is to use the values of the covariates that we can measure ahead of time to pick the best runs from the candidate set according to the experimental goal. This ends up being much more efficient than simply taking a random sample.

For an excellent example, I like to point people to Chapter 9 of Optimal Design of Experiments: A Case Study Approach. If you do not have a copy of the book, you can read an outline of the idea in an earlier blog post from Bradley Jones.

Another common use case is to provide a candidate set that enforces some constraint on the design space. While this can be done using factor constraints from the Custom Designer, the candidate set approach is useful when the region is quite complex, or if you want the runs of an experiment to have a certain structure. For example, you might want continuous variables to take on only five distinct values or restrict the number of non-zero factors in any given experimental run.

Using all the rows

Like the subset case described above, this can occur when all our experimental units are chosen, and we can measure some uncontrollable values before designing the experiment.

A common use case that may be less obvious is to force a desired structure for a subset of the factors.

For instance, say if you were designing a 12-run experiment with one two-level categorical factor, X1 with levels A and B, and four continuous factors, X2-X5, with an added restriction that for the categorical factor, 1/3 of the runs need to be at level A and 2/3 at level B.

All we need to do is create a data table for the categorical factor with 12 runs, and a column labeled with that factor name. Put four rows as A, and eight rows as B.

In Custom Design, choose Add Factor->Covariate, and then add the remaining four continuous factors. If we keep the number of runs at 12 before clicking Make Design, like this,

the resulting design will force the runs for X1 as specified by the candidate set (with the desired ratio) and design for the optimal settings for the continuous factors.

Other examples using this idea from previous blog posts include the creation of an experiment robust to a linear trend in the response over time or ensuring a definitive screening design structure for a subset of the factors, such as an experiment for hard-boiled eggs.

Anything else?

We have made some improvements to covariate handling in JMP 16. I've highlighted those new pieces in another post.

Analytics for social innovation

J_Marquardt — Fri, 26 Feb 2021 17:08:17 GMT

Ayana Littlejohn is committed to social innovation projects and outreach that inspires young Brown and Black girls to pursue education and careers in technology.Ayana Littlejohn provides impactful, data-driven insights by building predictive models for Fortune 500 brands, focusing on new product forecasting and customer intelligence for SAS. In addition to her core responsibilities, Ayana also serves as a Black Initiatives Group Leadership member and provides analytical expertise to the SAS Social Innovation team investigating inequalities in policies affecting marginalized and vulnerable populations. She spoke at a recent event on ways to find meaning with purpose-driven analytics, which you can now watch on demand.

Here’s an excerpt from the panel discussion:

“SAS hosted a social innovation project last year in which employees could create teams and explore and use analytics for topics like vulnerability, social justice, health care and environmental issues. I wanted to be a part of every single project, but ultimately, I decided to focus on [health and] vulnerable populations because of all these headlines I was seeing. I really wanted to see, is there an explanation? Is this a problem with Black people and our genetics, or is this a problem with how we don’t have the same access to what others have access to? Which then, in turn, leads to a problem with our overall health?

“We used public data from New York from all of the five boroughs. We profiled the five boroughs to see which boroughs had larger minority populations.…Now we see who is in each of the boroughs. From there, we said, ‘Let’s look at some health metrics.’

“How do we measure accessibility to health and other metrics that would be indicative of how often and what type of healthy food people in those boroughs are consuming? We were able to get our hands on some really rich data sets that included things like the average consumption of fruit and vegetables, the number of bodegas in a neighborhood….We also had access to the number of fresh markets and local markets – stores like Publix, Harris Teeter and Whole Foods.

“We decided to use these as a ratio. What’s the ratio of these bodegas that traditionally carry a lot of unhealthy food (that we have seen through studies are linked to hypertension and high cholesterol)? And what we found was that, in the neighborhoods with higher proportions of minority constituents, there was a much larger bodega-to-fresh-food-market [ratio]. And there was a much smaller rate for fruit and vegetable consumption.

“Then we said, ‘Is this correlated to health factors that we’ve seen in these areas?’ We looked at things like kidney failure, hypertension, high cholesterol, mortality rates – a litany of health issues.

“Instead of comparing what we found to each of these individual health variables – because we had about three weeks to do this – we combined all these health risks into an overall health index.”

Preview her comments here, and hear more about social innovation and analytics in this on-demand webcast.

Purpose, passion, and curiosity

anne_milley — Thu, 18 Feb 2021 19:44:16 GMT

Vic Strecher – longtime JMP user, author of Life on Purpose, professor and behavioral scientist at the University of Michigan, and CEO and founder of Kumanu – shares some of his latest research on finding meaning with purpose-driven analytics in January’s episode of Statistically Speaking. He has become a fan of structural equation modeling (SEM), as is evident from his fascinating plenary.

Here, Vic discusses his latest research using SEM to learn more about social determinants of health.

His plenary set the stage for a purpose-driven panel discussion. Joining Vic on the panel are Tim Gardner, CEO and founder of Riffyn; Ayana Littlejohn, analytical consultant at SAS; and Anderson Mayfield, marine biologist at NOAA and the University of Miami. What do they have in common? Curiosity and a passion for changing the world with analytics.

Tim founded Riffyn to enable better data, better decisions, better science, and a better world. In addition to building predictive models for her day job, Ayana dedicates her time and analytical talents to social innovation in health, equity, education and climate change issues. Anderson is passionate about understanding coral reefs to implement effective conservation efforts globally.

While the panelists come from different backgrounds and have diverse purposes, they share a common passion and curiosity that are inspiring. If you are in need of a little purpose-driven analytics inspiration and some cross-pollinated ideas, 2021’s first episode of Statistically Speaking is just the ticket.

An analyst’s new year’s resolution: Setting up for success

wendytseng — Wed, 10 Feb 2021 15:59:12 GMT

One of the best pieces of advice I got from a professor was to think about the time investment in a project as 90% preparation, 10% execution. You can argue about the percentages, but I think the point is clear: If you spend time preparing, actually doing the project will be easier.

This blog post will show you how you can set your JMP Preferences so that you always see the model outputs that you want to see. Think of this as “meal planning” to set yourself up for healthy eating the rest of the week.

While I will be focusing specifically on Standard Least Squares outputs, this advice extends to all analyses in JMP. Take time to set your Preferences to ensure you are looking at the outputs that are the most consistently helpful. This also results in time savings by minimizing the number of mouse clicks.

The goal of this five-part blog series has been to help analysts who are new to modeling get started or gain momentum while using it to advance their work. In working with scientists and engineers, I have often heard them say, “I want to use modeling because I know it will help us solve bigger problems faster, but I am not sure which model outputs to consider and how to interpret them.”

A Framework

With the goal of simplifying and providing clear guidance, I have devised a framework that is shown in Figure 1. This “menu” is a tool I hope analysts will find useful for focusing on the key outputs based on the situation, that is, the data and the analyst’s appetite. Some modeling problems do not require a lot of extra research and troubleshooting while others do. And some analysts want or need fewer outputs to make decisions, whereas others love getting into the details or need more looks to gain confidence in their conclusions and decisions.
Figure 1: Key outputs in Standard Least Squares. Choose your meal based on your situation and appetite!

JMP’s developers set up default reports in Standard Least Squares based on the data. Think of the Emphasis that is chosen as the template of outputs shown. The outputs that are shown and expanded are a great place to start, but if you want to use the menu framework I have proposed in this series, you will need to do some customization of your Preferences to ensure the “menu” outputs always show up as a default.

To set your model preferences, go to File > Preferences and navigate to the Preference Group “Platforms” and select “Fit Least Squares.”

Here, you can check off your favorite graphs/tables/statistics so that they will always pop up (Figure 2). This is “meal planning” for the motivated analyst who has a new year’s resolution to conquer! :flexed_biceps:

Figure 2: Fit Least Squares output preferences. The non-default outputs are highlighted here with colored boxes and correspond to the outputs in the menu framework in Figure 1.

Summary

Taking a few minutes to set your Preferences in JMP can set you up for success in both your modeling efforts and any of your data analyses work. Ensuring that the core outputs that help you make decisions are always front and center will make your job easier and allow you to work more efficiently.

If you are interested in the earlier blog posts in this series, below are links to allow for easy access. The first blog post references a data set that you can use for practice and provides links for accessing JMP software if you don’t already have it.

Finally, there is no substitute for the foundational knowledge you can get from the Statistical Thinking for Industrial Problem Solving Course. It is free and incredibly engaging. The format is comprised of short videos (approximately 2-10 minutes in length) followed by exercises. I recommend the Correlation and Regression, Design of Experiments and the Predictive Modeling modules as complementary learning to this blog series.

Best wishes to you this New Year! I hope are able to make great strides toward your goals (whether they focus on modeling or not) this year.

Quick links to individual posts in this series

Register now for the best seat in the house

Diana_Levey — Wed, 03 Feb 2021 14:18:08 GMT

Registration for Discovery Summit Europe is now open!Our European Discovery Summit may not take you to a beautiful destination in March, but it will take you to new heights in how you use statistics to make better decisions – for free!

Specifically, at the entirely online Discovery Summit Europe, you will find:

JMP R&D staff – including JMP creator John Sall – showing off their favorite parts of JMP 16, JMP Pro and JMP Live.
Scientists, engineers and other data explorers presenting their statistical solutions to real-world problems via papers and posters.
JMP experts – authors, teachers, developers and users – excited to discuss your data, your business challenges and your use of statistical techniques.
Thought-provoking plenary sessions, with the opportunity to ask questions.
JMP enthusiasts from across industries ready to share their best practices in interactive data exploration.
JMP systems engineers and customer care staff on hand to make sure you get the most out of the event.

All of the exciting content will be available through easy-to-navigate web pages in a comfortable, easy-to-consume schedule:

On Monday, 22 February, papers and posters will be available to watch at your leisure.
On Monday, 8 March, attend the JMP Scripting Forum or the Predictive Modeling Forum. Each forum lasts three hours.
On Tuesday, 9 March through Friday, 12 March, spend your afternoons in a mix of interactive sessions, scheduled demos and enlightening talks. Each afternoon starts with a wellness exercise; each afternoon ends with a social event.

You cannot beat the convenience. And the content is priceless...literally. No planes or trains needed. No hotel reservations to make. Not even a registration fee. But you do need to register. And the sooner you do so, the sooner you can count on having the best seat in the house (OK, the best seat in your house).

An analyst’s new year’s resolution: Whole grains for the whole picture

wendytseng — Fri, 05 Feb 2021 22:00:48 GMT

It’s common knowledge that a balanced diet of plants, good fats and whole grains is what our bodies need to function at their best. If we want to be good analysts, we need to fuel our gray matter properly, right? Well, our modeling projects need a complete set of nutrients, too.

This is part four of a five-part series where I am proposing a framework to help analysts who want to apply modeling to their work. This framework is aimed at addressing the common pain points I hear articulated as, “Which model outputs should I look at? There are so many!” and “What do they mean? What statements can I make from the numbers I see in the outputs?”

This framework is in the form of a menu (from a restaurant your mom would approve!) where you, the analyst/diner, selects outputs based on the complexity of your data and your disposition as an analyst (refer to Figure 1 below). Some analyses are straightforward and don’t require that you consider many modeling outputs, while some others need you to look at more options. Equally, some analysts need to move quickly with their projects or don’t enjoy getting into the statistical weeds, while others enjoy diving in head first and setting up camp :smiling_face_with_smiling_eyes: to look at their problem from many different angles.

When your analysis is straightforward, you may choose to just order the big salad or “Veggie Plate.” When the model needs a little more attention, you can add some salmon (“Healthy Fat”) to your salad. Finally, if you’re modeling historical data where a lot of the predictors may be correlated or if you want to look at additional data to tease out signals and get a deeper understanding, you can add a side of brown rice to your meal.

Figure 1: Key outputs in Standard Least Squares. Choose your meal based on your appetite!

Check out post #2 and post #3 for details on the outputs in the Veggie Plate and Healthy Fat options. In this post, I am focusing on the Whole Grain outputs. We will start with the output that helps you figure out if your model is healthy when you are working with historical data. Next, we will look at the outputs that can further increase your understanding of the process and effectively communicate the findings with collaborators.

For each output, I will describe: 1) what the output is, 2) what you should be looking for and 3) the actions can you take, given the information. My goal has been to simplify but that, of course, comes at a risk of potentially oversimplifying. If you want to get a deeper understanding, I recommend the Statistical Thinking for Industrial Problem Solving course (see the end of the blog post for a recommendation on the specific modules).

Oh – and if you want to follow along and generate these outputs yourself in JMP, see the first blog post in this series where I describe the data set and how you can get access to JMP if you don’t have it.

Is my model healthy?

VIFs (variance inflation factors)

What is Multicollinearity? Why is it a problem?

Multivariate Scatterplot Example

Penalized Regression

Principal Components Analysis

Partial Least Squares

How do I interpret the results?

Parameter estimates

Interaction Plots

Summary

The Whole Grains outputs are the ones you go to when you are analyzing historical data or when you want to get additional views into understanding the process you are studying…just as whole grains are what you should add to your diet when you need the most fuel for that hard workout or busy day.

When you are analyzing historical data (versus data from a designed experiment), your predictors may be highly correlated, which can result in model bias – and ultimately drawing incorrect conclusions from your data. VIFs are a good way to diagnose if this is an issue you may be facing.

When it comes to interpreting and using your model, the Parameter Estimates can help you describe the degree to which a particular factor individually impacts the response; they also map directly to the model equation for those who want to see what’s behind the Prediction Profiler. Finally, the Interaction Plots can be a helpful way to picture how factors interact with each other visually in a single static snapshot.

Next week, in the final post, I will describe how you can set your Preferences in JMP so that certain model outputs always show up – regardless of the report Emphasis that’s selected in the Fit Model dialog.

I would be remiss if I didn’t remind you, one last time, to check out the Correlation and Regression and Design of Experiments courses that are part of the Statistical Thinking for Industrial Problem Solving course. If this blog series leaves you more curious about the underlying statistical concepts, this course will satisfy that curiosity. It has been designed specifically for scientists and engineers who want to better use data and statistics to advance their work.

An analyst’s new year’s resolution: Add some good fat to your diet when necessary

wendytseng — Wed, 03 Feb 2021 21:14:05 GMT

Am I reading a statistics blog post or a nutrition and/or self-help guide? If you are a bit thrown off by the title of this blog, you are likely new to this blog series. This post is part three of a five-part series where I address questions I have gotten a lot from scientists and engineers over the years: Which model outputs should I look at? What do they mean? What do I do with them?

With the goal of simplifying and providing clear guidance (not to make you hungry or make you feel guilty for eating processed foods), I have devised a framework that is shown in Figure 1. This “menu” is one you can use to focus in on the key outputs based on your situation (your data and disposition).

Figure 1: Key outputs in Standard Least Squares. Choose your meal based on your appetite!

In this post, I am going to focus on the outputs that you can look at in addition to the Veggie Plate outputs if you need some first aid for model health issues, or if you are a data nerd and want to nerd it up (you’re in good company, so hold your head high!). First, let’s review the outputs that you can use to answer the question, “Is my model healthy?” Next, let’s examine the outputs that will help you interpret the results to increase your understanding of the process and communicate them to your colleagues. For each of the outputs, I’ll try to be clear and concise about: 1) what the output is, 2) what you should look for and 3) what you can do with the information.

If you want to follow along ("taste test") with the sample data set that I use in the screenshots, see the first blog post.

Is my model healthy?

Lack of Fit

*Lack of Fit

Box-Cox Transformation

Transforming Data to Make Better Predictions

How do I interpret the results?

Variable Importance

Assess Variable Importance

Effect Tests

Summary

In this post, we discussed two outputs to help you assess and remedy, if necessary, model health issues: Lack of Fit and Box-Cox transformation. We also discussed two outputs to help you further understand the process and, ultimately, decide how to improve the process and communicate your actions to stakeholders: Variable Importance and Effect tests.

If you are working with an historical data set that may have a lot of correlated predictors, I encourage you to push forward to Part 4 of this series next week, where I’ll be discussing VIF’s and how they can help you assess model health. I’ll also be walking through Interaction Plots, which can be a helpful way to visualize how factors are influencing each other.

As mentioned in previous posts, there is no substitute for the foundational knowledge that the Statistical Thinking for Industrial Problem Solving Course can give you. Consider completing the Correlation and Regression module, specifically. Earn a badge upon completion that you can post on LinkedIn, add to your résumé and annual review. Make 2021 the year of personal and professional growth!

Outliers Episode 3: Detecting outliers using the Mahalanobis distance (and T2)

JerryFish — Tue, 26 Jan 2021 21:30:12 GMT

Welcome back to my blog series on Outliers. In the first episode, we looked at defining and visually identifying outliers. In the second episode, we used quantiles (via Box & Whisker plots) to help identify outliers in one dimension.

In this episode, we look at identifying outliers in multiple dimensions using the Mahalanobis distance. We also take a quick look at T², which is a simple extension of the Mahalanobis distance.

Multidimensional outliers

Figure 1 is a reproduction of one of the figures from Episode 1 and shows a two-dimensional plot of 1,000 data points with a clear outlier (marked with the red square):

Figure 1: 1,000 data points in two dimensions, outlier marked in red

The scatterplot in Figure 1 allows us to easily determine outliers visually in two dimensions, even when the outlier is not evident from the univariate histogram distributions. So, the Box & Whisker (quantile) techniques described in Episode 2 aren’t always reliable when analyzing multidimensional data. And visually identifying outliers (as in Episode 1) becomes difficult as the number of dimensions increases. So how can we more reliably detect these multidimensional outliers?

The Mahalanobis distance

The Mahalanobis distance (D_M) gives us a numerical method for identifying multidimensional outliers. It is named for its creator, Indian statistician Prasanta Chandra Mahalanobis (1893-1972). For more information on Dr. Mahalanobis, I direct you to this Wikipedia article.

A D_M plot for the data set shown in Figure 1 is shown below:

Figure 2: Mahalanobis Distance plot for 1,000 data points from Figure 1

Figure 2 shows all data points in row order versus D_M on the Y axis. The value of D_M for each point represents the standardized “distance” of each point from the centroid of all data points. The red data point clearly stands out from the remaining points, so it would warrant further investigation as an outlier.

So how does this calculation work?

The general method

Let’s say we have a two-dimensional distribution of 1,000 points. In each dimension, 999 of the points are drawn from a normally distributed population with mean of zero and standard deviation is 1.0. Then I added an outlier point as the last point in the sample. If we make an X-Y scatterplot, we might have something like this:

Figure 3: Scatterplot of 1,000 uncorrelated data points, with one marked outlier (red)

Note the red point of interest in Figure 3. We can easily calculate the mean and standard deviation of the 1,000-point sample in each dimension, and we can calculate the Euclidean distance from the red point to the centroid of the points using the Pythagorean Theorem. For multidimensions:

In fact, we can draw circles centered on the centroid representing equal distances from the centroid, as in Figure 4:

Figure 4: Scatterplot of 1,000 uncorrelated data points, with equidistant circles from centroid overlaid

In Figure 4, the circles represent distances of 1, 2, 3, and 4 units (from smallest to largest) to the centroid of the points. Here it is clear to see that the point in red is the furthest from the centroid of the points. Its distance is represented by “d”.

This works well if we have similarly scaled distributions (i.e., that the standard deviations are the same). But what if we have different scales, as shown in Figure 5?

Figure 5: Scatterplot of 1,000 uncorrelated data points, having different means and standard deviations in each dimension.

Here we have two problems. First, the mean of the points in the X3 dimension is non-zero. That is simple enough to fix by just subtracting the mean from all data points.

The second problem is that the standard deviation in the X3 dimension is much larger than in the X1 dimension. This renders a distance calculation moot, unless we can compensate for the differences. This is also easily done. After correcting for the mean shift from zero, we simply divide in each dimension by its standard deviation. This normalizes for any mismatches in standard deviations. The multidimensional equation to evaluate this distance becomes:

where m and s represent the mean and standard deviation in each dimension. If we carry out these computations and replot the standardized data, we get the figure below:

Figure 6: Data from Figure 5 after transforming to correct for mean and standard deviation of each dimension

So now we have a way to adjust the data for non-zero means and for unequal standard deviations. This works well as long as the data in each dimension are independent of each other (i.e., uncorrelated.) But if there is correlation like that shown in Figure 1, then what do we do?

One way to do this would be to employ principal components to establish new axes aligned with the trends in the data, and then perform the operations described above. But this requires a lot of clicks. (If interested in how this might work, see Appendix A.)

Fortunately, Dr. Mahalanobis gives us an easy and elegant way to calculate this distance!

Mahalanobis’ equation for calculating distances

Dr. Mahalanobis’ elegant equation for calculating the D_M is:

where:

is a vector of values of the N dimensions of the i^th individual observed data point.

N = the number of dimensions (or columns) of interest.

T indicates the Transpose operation.

is a vector of the means of each of the N dimensions (or columns) of interest.

S^-1 is the inverse of the NxN covariance matrix for all of the data points (available from the Multivariate platform in JMP).

Plugging appropriate values into the above equation gives the D_M from the i^th point to the centroid of all points in the data set. If you are interested in how this calculation works, Appendix B works through a small sample problem.

The Mahalanobis plot in JMP

D_M by itself is of little use to us. It only has meaning when compared to other data points. The Mahalanobis plot in JMP shows these results.

We can generate the Mahalanobis plot by going to Analysis/Multivariate Methods/Multivariate, selecting the columns, then from the red dropdown select Outlier Analysis/Mahalanobis Distance. For the example data set of Figure 1, this gives:

Figure 7: Mahalanobis distances for 1,000 data point sample (repeat of Figure 2)

Here we see that the data point in Row 1 of the data table has the largest distance (D_M), indicating it is the furthest from the centroid of all 10 data points. The blue line represents a 95% upper confidence limit (UCL) on whether we believe the D_M is significantly different from zero. The red point is clearly above the UCL, so it warrants further investigation as an outlier.

But the outliers aren’t always so obvious to detect, particularly with small data sets. In the example in Appendix B, I show sample calculations for a small data set (10 data points in three dimensions). This data set generates a D_M plot as shown below:

Figure 8: Mahalanobis distances for 10 data point sample in Appendix B

The D_M plot has identified Point 1 as an outlier. But we also see that D_M for Row 1 is not all that different from D_M for Row 7! Why is this? You’ll have to wait for the next episode for the answer to that question!

A quick note about T²

JMP also offers an option for a T² calculation under Analysis/Multivariate Methods/Multivariate, and then Outlier Analysis. T² is simply the square of D_M. Some people like to use it to force more visual separation between the outliers and the bulk of the data, but T² really contains the same information as D_M.

Next episode

In the next episode, we’ll cover the Jackknife distance, which is another way of looking at outliers and which will make the outlier from the above example data set stand out.

See all posts in this series on understanding outliers.

Appendix A: Using principal components to identify outliers

It would be nice if we had a way to rotate the axes in Figure 1 so that they lined up with the data trend, something like this:

Figure 9: Conceptual rotation of axes for two-dimensional data set

Then we could use the techniques described earlier for standardizing the data to detect outliers.

One way to do this is Principal Component Analysis (PCA). PCA finds the set of orthogonal (mutually perpendicular) axes that best fits the data. You can have as many principal components as you have original dimensions (columns) in your data set. In JMP, PCA can be found in two places: either under Analyze/Multivariate Methods/Principal Components, or under Analyze/Multivariate Methods/Multivariate, and then from the red dropdown next to Multivariate, choose Principal Components/on Correlations. The red dropdown also gives an option to save the principal components back to the data table. Once you have done this, you can use the methods to correct for means and standard deviations outlined above to determine the distance from the centroid.

Appendix B: D_M calculations for an example data set

Let’s say we have a sample data set with three columns and 10 rows:

Figure 10: Small sample data set used for DM example calculations

In JMP, we can run Analysis/Multivariate Methods/Multivariate on these three columns to produce the following scatterplot matrix:

Figure 11: Scatterplot matrix for example data

In Figure 11 we can see that Row 1 data (highlighted in red) appears to be an outlier when comparing in the X1-X2 dimensions, though it is not as clear in X1-X3 or X2-X3 plots.

We want to calculate the Mahalanobis distance for the first point. So, filling out the vectors and matrices in the above D_Mequation, we start with the point of interest:

The means of each column can be found in several different ways. Let’s use Cols/Column Viewer, then Show Summary:

Figure 12: Summary statistics for example three-dimensional 10-point data set

So the vector of means becomes:

Next, we need the covariance matrix, which is found under Analysis/Multivariate Methods. Select Covariance Matrix from the red dropdown triangle:

Figure 13: Covariance matrix for example three-dimensional 10-point data set

We also need the inverse of the covariance matrix, S^-1. JMP doesn’t give us that information from the menu system, but we can get it easily from a custom script as follows:

Figure 14: JMP JSL script to find inverse of a matrix

This results in:

S^-1=

Then we just plug in and solve the matrix equation:

The Mahalanobis distance (D_M) is 2.814 for the data shown in Row 1 of the example data set.

Figure 15: Mahalanobis plot for example data set (repeat of Figure 8)

An analyst’s new year’s resolution: Eat a hefty serving of veggies

wendytseng — Wed, 03 Feb 2021 21:15:27 GMT

If you have resolved to use models to solve more problems, this blog series is for you.

One of the biggest barriers I have seen from analysts who want to do this is that they are overwhelmed and/or confused with all the outputs they need to look at once they’ve built a model.*

In the first blog post in this series, I talked about how JMP’s default Emphasis is a good place to start when it comes to figuring out which model outputs to focus on. However, even with those default settings as a starting place, there are still many outputs that one has to prioritize and figure out how to interpret.

To help demystify and simplify model output interpretation, I have developed the menu in Figure 1 to help analysts focus on the key outputs based on how deep they want to go (or how hungry they are, so to speak) and to what extent the problem requires the extra attention (or “nutrition” :smiling_face_with_smiling_eyes:).

*We are focusing on Standard Least Squares modeling in this blog series because that is the most common type of model new users try first because of the accessibility in JMP (versus JMP Pro) and the ease of interpretation of the model equations back to the real world.

Figure 1: Key outputs in Standard Least Squares. Choose your meal based on your appetite!

In this week’s post, I am focusing on the core outputs for users who are looking to move quickly with their analysis and decision making and whose data allows for the simplicity. This set of outputs are collectively referred to as “The Veggie Plate” in the menu because it’s always best practice to eat a healthy serving of veggies, right? They are staples for good health in humans just as they are staples for healthy and useful models.

If you want to follow along with the sample data set that I use in the screenshots, see the first blog post.

Is my model healthy?

Actual by Predicted

** Did you know that if you change your cursor to a question mark (Tools menu) and click on any output, you will be taken to the specific section of JMP documentation? Try it with the Actual by Predicted Plot.RSquare and RSquare Adjusted

Residual Plots

JMP's Statistics Knowledge Portal

Transforming Data to Make Better Predictions

How do I interpret the results?

Effect Summary

Prediction Profiler

Four-Minute Video: Prediction Profiler

Scaled Estimates

Summary

I hope you found this post helpful in focusing in on the essential outputs to get you started on your modeling goals.

If the Veggie Plate outputs get you what you need to make decisions, it will probably make you a hero on your project team and you should go celebrate by eating a hefty serving of veggies! (Whomp! OK – maybe a sundae – all things in moderation, right?).

If you think your model may need more work or if you are just hungry to learn more, you can join me next week where I will describe the “Healthy Fat” outputs. This set of outputs will help you diagnose and remedy model fit issues and enable you to go a deeper in order to clearly communicate your results to others.

I invite you to stick around with me through this blog series. In the last blog post on Feb. 10, I will describe how you can customize your Preferences so that your favorite model outputs always show up.

P.S. If you are looking for foundational understanding, please consider taking the Correlation and Regression module of the Statistical Thinking for Industrial Problem Solving course (cheaper than any meal because it’s free!).

Something to look forward to: Discovery Summit Europe

Diana_Levey — Wed, 20 Jan 2021 14:11:00 GMT

The Steering Committee has selected the papers and posters for Discovery Summit Europe 2021.There’s light at the end of the tunnel, but this is a very, very long tunnel.

This time last year, we were just learning about Covid-19, unaware that it would change the way we live and work. Now, of course, we know all too well what it looks like when we have to isolate for the safety of our loved ones, our communities and ourselves.

We also know what it looks like when scientists and engineers come together to help humanity beat back a virus. It’s beautiful. That’s why we can see that light at the end of this tunnel.

It’s a privilege to know that JMP software is one of the weapons being used in the fight against Covid-19. It’s the logical choice for researchers and manufacturers that need to make better decisions faster...and with statistical precision.

Of course, these heroes are not the only people using JMP these days. Scientists and engineers in all industries are tasked with keeping business moving despite hardships, with doing more with less, and with improving their products and processes.

Come meet some of these data explorers at the entirely online Discovery Summit Europe. Find out how they’re surviving – and even thriving – in these strange times. Hear about the challenges they’ve overcome, and the statistical strategies they’re deploying to stay successful.

Papers and posters have been selected by the all-star Steering Committee. Keynotes are booked and nearly ready to be announced. And the online venue is being created for an interactive, educational and entertaining experience.

No need to buy a plane or train ticket. No need to book a hotel room. And no need to find budget for registration.

Just block your calendar for the week of 8 March, and spread the word among your statistically curious co-workers.

If you want to ensure that you receive the Discovery Summit registration announcement and other important news from JMP, simply opt-in to our distribution list at jmp.com/optin. And encourage colleagues to do so as well. Learning with peers is especially important when so many of us are living and working remotely.

An analyst’s new year’s resolution: Use models to solve more problems

wendytseng — Wed, 03 Feb 2021 21:14:46 GMT

As a new year begins (was there ever a time you wanted a fresh start more???), you may have begun formulating some new goals for the year or started working toward them already. We are a few days into 2021, so hopefully you haven’t yet abandoned the ones you made!

If you have room for another new year’s resolution (or if you need a new one), may I suggest one that I have heard a lot of analysts articulate to me: “I would like to use models (or more do more modeling) in my work.” This goal sounds more achievable than “I will lose 10 lbs.” or “I will get eight hours of sleep,” right?

Analysts are a motivated bunch, but they often get stuck because they don’t have a method to follow. The biggest barrier I have seen from analysts who want to use models to solve their problems is that they get confused and/or bogged down with all the outputs they need to look at once they’ve built a model.

“Which model outputs should I be looking at? There are so many! And how should I interpret them?”

The truth is this: There are a lot of outputs you can look at, and it can be pretty overwhelming if you don’t know which ones are the most important and what they are telling you.

Quick detour

Knowing this is a topic (and what I put forth in this series) that may spark a lot of debate and discussion, let me tell you a little bit about my background so that you have some context for my point of view. My education is in biomedical engineering, and I worked in medical device R&D for several years before transitioning to a role as an applied statistician for a consumer packaged goods company (CPG) supporting shampoo formulators in designing and analyzing experiments to better formulate shampoo. (The next time you wash your hair, pay tribute to the statistical models that were used to perfect the lather volume and texture!)

After leaving the CPG company, I spent a good chunk of my career using data and analytics in retail. The scientists, engineers, business analysts and other collaborators I worked with all had to use data, analytics and software to make quick but sound decisions. A few had some formal training in statistics, while the majority simply learned statistics and statistical software on the job and are looking for clear guidance on best practices with an emphasis on the principal “less is more.” I tried to structure this guidance with these practitioners in mind (myself included).

What you'll learn

In this five-part blog series, I am going to focus on regression model outputs, specifically, Standard Least Squares output because that is the most commonly used method used for modeling. Standard Least Squares are a great beginning, particularly for those new to modeling.

First, be comforted by the fact that the statisticians and developers at JMP set up the default reports in a way that presents the most important information to analysts.

For those already using JMP: Have you ever noticed the Emphasis in the Fit Model input dialog and wondered what it is? I know I was pretty confused about it for a while. In contrast to the Personality options right above it, Emphasis does not change the type of analysis that is done. The different options simply turn on different model outputs as a default when you click Run. JMP uses the amount of data (number of rows) you have in combination with the number of model terms you want to estimate (number of terms in the Model Effects dialogue) to determine the best set of outputs to show you by setting a default Emphasis (Figure 1). JMP is always guiding analysts toward the best path, and the Emphasis selected is no exception. JMP uses the contents of your data to set the default Emphasis. (For more specifics on how it does this, you can read the documentation.)

Figure 1: Standard Least Squares Emphasis. JMP sets a default Emphasis based on your data, but you can change it. Emphasis determines the set of outputs that are shown; it does not affect the statistical analysis.

The outputs that are shown after you click Run are the ones you should focus on first. The outputs that are the most important given your data and the model you are trying to fit are expanded and displayed.

That being said, there are still a lot of graphs and tables to look at and without some training, you will not know which outputs to prioritize and how to interpret or take action on those outputs. Sadly, there still is no pill that you can take to give you this knowledge!

Until that pill exists, I hope this blog post serves as a quick guide.

Model health and interpretation menu

Model outputs can be broken into two main categories of outputs: model health and interpretation. Model health outputs help you determine if your model is healthy or if you need to perform some “first aid.” Interpretation outputs are those that help you decide which factors are important and how they influence the response(s).

I have developed the menu in Figure 2 to help you focus on the key outputs based on how deep you want to go (or how hungry you are, so to speak). I think it’s only appropriate to use healthy foods in this menu because we are all on our best behavior (or have the best intentions) when setting our new year’s resolutions! My previous drafts of this blog post used a fast food menu board, which seemed…incongruous :smiling_face_with_smiling_eyes:. I offer you three meals:

Figure 2: Focus in on the key outputs in Standard Least Squares. Choose your meal based on your appetite!

The Veggie Plate has the core outputs only and is for the analyst who is looking to move quickly or whose analysis allows for that simplicity.

The Veggie Plate

The Veggie Plate + Healthy Fat (salmon is one of my faves!) includes the Veggie Plate outputs plus additional outputs that can help with finding remedies for poor model health or for analysts who really like getting into the data. Omega-3s have so many healing benefits!

Veggie Plate + Healthy Fat

Finally, the Full Square Meal includes outputs (“Whole Grains”) that are appropriate when you are struggling to get a good model or when you are just feeling extra hungry for more outputs to help you understand your problem.

The Full Square Meal: Veggie Plate + Healthy Fat + Whole Grains

Taste testing the menu

If you want to follow along, I’ll be using the Tiretread data set in JMP’s sample data library. If you don’t have JMP, you can download a free 30-day trial or use it in a remote desktop environment when you take the Statistical Thinking for Industrial Problem Solving course (also free; link in the P.S.).

The Tiretread data set comes from a designed experiment where the goal was to optimize four product characteristics (responses): Abrasion, Modulus, Elongation and Hardness using three ingredients (factors): Silica, Silane and Sulfur. The analysis of the experiment involved building and using four (one for each response) standard least squares models relating the factors (via model terms describing main effects, interactions and quadratics*) to each of the four responses.

*Not to beat a dead horse, but the Statistical Thinking for Industrial Problem Solving modules on Correlation and Regression and the Design of Experiments will help you understand the terminology better.

Summary

I hope you will be bold in 2021 and pursue this goal of using models in your work and learning how to interpret the outputs. Know that JMP has your back with the default settings in the Fit Model platform. JMP considers the amount of data you have and the number of effects you are trying to estimate and presents the most important outputs to you.

And I also have your back! I hope you will continue reading the rest of this blog series to get an additional boost of confidence using models.

In the next three posts, I will describe how to interpret the outputs that are in each of the three “meals” and what actions to take in response to them where appropriate. In the last blog post of this series, I will show how you can set your Preferences in JMP so that your favorite outputs always show up in your model results.

Back in a week!

P.S. To go deeper on the topic of modeling, I highly recommend the Statistical Thinking for Industrial Problem Solving course we offer that is free, specifically the Correlation and Regression module. It goes deeper into the statistical concepts and includes hands-on exercises as well (along with free access to our software).

Outliers Episode 2: Detecting outliers using quantile ranges

JerryFish — Tue, 26 Jan 2021 21:33:14 GMT

Welcome to my second blog installment on Outliers. In this episode, I’ll look at using quantile ranges to detect outliers.

In Episode 1, we looked at describing and visually identifying outliers. You may recall that there are simple one-dimensional (or univariate) cases where we need to identify outliers. These are the easiest to identify, and quantile ranges are an excellent way to look for them.

For multidimensional outlier detection, you’ll have to wait for my next posts!

What Are Quantiles?

Quantiles are very easy to understand. Let’s say we have a series of 20 numbers. We can sort the numbers from lowest to highest. We can then group these points into quantiles, which are identified by cut points in the sorted data that describe the point below which X% of data falls.

Note that quantiles are generally expressed as a fraction (from 0 to 1). They correspond exactly to percentiles, which range from 0 to 100. I will use these interchangeably throughout this post.

Figure 1 shows the 20 numbers before and after sort, and the 0.20 and 0.50 quantiles (or 20^th and 50^th percentiles).

Figure 1: Data from 20 point sample, random order and sorted

The cut point for the 50^th percentile is the median of the sample. (In this case, the median is numerically 32.5, the midpoint between 40 and 25.)

A few other interesting definitions include:

Quartiles, which divide the data into 25% groups. The first quartile represents the data points that fall in the lowest 25%, the second quartile points fall between 25% and 50%, and so forth.
Interquartile range, or IQR, which defines the range covered by 2^nd and 3^rd quartiles.

Note that there are no assumptions made on the shape of the distribution when talking about quantiles.

Where Are Quantile Calculations Found in JMP?

You can find quantiles listed in three places in JMP. The Distribution platform and Graph Builder both use the same calculations. Explore Outliers/Quantile Range Outliers offers more flexibility to the user in terms of outlier sensitivity. Let’s look at each.

Distribution platform

In Distribution, all continuous/numeric data histograms include a tabular quantiles summary. By default, JMP displays several quantiles of interest. Using the Distribution platform on the column labeled Original Data in Figure 1 gives the following output:

Several common quantiles are shown in the center table of Figure 2. You can change the quantiles that are displayed in this table by selecting the red hotspot and choosing Display Options, and then either Set Quantile Increment or Custom Quantiles.

Also displayed in Figure 2 is the Box & Whisker plot. Symbols shown in the plot are annotated in Figure 2.

The ends of the whiskers are of most interest in detecting outliers. Note that the whisker length is measured from the min and max of the interquartile range and that the whiskers end at the last data point that is inside 1.5 times that range. This is a common rule of thumb for outlier detection. Points falling outside of these whiskers may be worth further investigation.

But in the data set shown in Figures 1 and 2, this technique detects no outliers. This is not uncommon, particularly with small sample sizes.

Figure 3 shows the Distribution platform output for the 1,000 sample data from my previous blog post. These samples are drawn from a random normal distribution, with mean of 0.0 and standard deviation of 1.0. I also include the point at X1=4 which was visually identified as an outlier in my previous post:

Figure 3: Distribution platform output for 1,000 point sample from normal distribution

With the larger sample size, some of the points now appear to be outliers (at least as defined by the ends of the whiskers). In this particular case, since we know that the data come from a normally distributed population with 1,000 points, it is probably not uncommon to see at least one point with a value of 4. In a real-world case, the points beyond the whiskers might warrant additional investigation.

Making Box and Whisker Plots in Graph Builder

You can also generate box and whisker plots in Graph Builder. The entire Graph Builder setup is shown in Figure 4:

Figure 4: Graph Builder setup for Box & Whisker plot (1,000 point sample from normal distribution)

Note the icon for graph type above the graph (circled in red), which enables the Box & Whisker plot type.

As with the Distribution platform, the whisker lengths are 1.5*interquartile range, and extend down from the 25^th percentile and up from the 75^th percentile.

Explore Outliers

What if we want to choose other whisker limits?

As stated above, a whisker length of 1.5*IQR is a common practice for identifying outliers. I believe this probably comes from looking at large sample normal distributions. 1.5*IQR beyond the interquartile range can be shown to encompass 99.30% of the normal distribution (leaving 0.3488% of the data in each tail). This would identify a relatively rare event – if you have a normal distribution.

But what if you don’t have a normal distribution? What if the distribution looks something like this?

Figure 5: Skewed distribution

Figure 5 shows a skewed distribution of N=100 points, with five outliers beyond the upper whisker, per the 1.5*IQR method. But is the point at X1=3 really an outlier? Or perhaps there are other points concealed by the whisker that would warrant outlier investigation. How do we adjust this 1.5*IQR rule?

JMP allows you to do this under Analyze/Screening/Explore Outliers, and then choosing the Quantile Range Outliers option. Bringing up this option presents the input screen shown below for the 100 point sample summarized in Figure 5.

Figure 6: Quantile Range Outliers user interface and sample output

There are two main inputs for the Quantile Range Outliers panel:

Tail quantile (TQ). This fraction describes the smallest cut point in the distribution that will be used in whisker calculations. (1-Tail Quantile) describes the upper cut point in the distribution. The difference between these cut points describes the range used in subsequent calculations.
1. For the IQR calculations described previously, tail quantile would be 0.25, indicating that the whiskers would start at the 25^th and 75^th percentiles, and the range would be the interquartile range (or 75^th-25^th percentiles).
2. The default value for the tail quantile in the Quantile Range Outliers is 0.1, indicating that the whiskers would start at the 10^th and 90^th percentiles, and the length depends on the (90^th-10^th percentile) range.
Q. This is the multiplier used to determine the length of the whiskers.
1. In the previous IQR calculations this was 1.5, so the interquartile range was multiplied by 1.5 to get the whisker length.
2. The default value for Q in the Quantile Range Outliers is 3, indicating that the whisker length will be defined by the (90^th-10^th percentile) range multiplied by 3.

The output of the Quantile Range Outliers is shown at the bottom of the panel in Figure 6. Here we see that Column X1 had a 10^th percentile at -1.283, and a 90^th percentile at 1.65556. Ends of the whiskers (noted as the low and high thresholds [LT and HT, respectively] in the output) are based on the following calculations:

LT = 10^th quantile – (90^th quantile – 10^th quantile) * Q

= -1.283 – (1.65556 – -1.283) * 3

= -10.099

HT = 90^th quantile + (90^th quantile – 10^th quantile) * Q

= 1.65556 + (1.65556 – -1.283) * 3

= 10.4713

In this case, there is only one value in the column that exists beyond the high threshold, at a value of 12.0, which is displayed at the bottom of the report shown in Figure 6.

Sensitivity to tail quantile and Q

The outlier detection sensitivity is clearly governed by the values of tail quantile and Q. The traditional 1.5*IQR and the 3*(90^th-10^th quantile) methods are both acceptable, with the former being much more sensitive to detecting outliers. You can use the Quantile Range Outliers platform to adjust these values as needed for your particular case.

While the effectiveness of the choice of tail quantile and Q will depend on your particular distribution, we can gain some insight into their behavior by looking at normal distributions. Below is a table showing several combinations of tail quantile and Q, along with the percent of values in the tails of a normal distribution that would fall outside of the whiskers:

Table 1: Percent of outliers detected in large sample normal distribution for various values of tail quantile and Q

Expanding on this idea, we can create a contour map showing a more complete picture of the quantities in Table 1:

Figure 7: Percent of outliers detected in large sample normal distribution for various values of Tail Quantile and Q

Figure 7 plainly shows that the combination of high values of tail quantile and low values of Q makes the calculations more sensitive to outliers, while low values of TQ and high Q makes us less sensitive.

Next Episode

Next time we will discuss Mahalanobis distance, which is used to detect outliers in multiple dimensions.

See all posts in this series on understanding outliers.

Everything I need to know about data visualization, I learned from Thomas the Tank Engine

MikeD_Anderson — Wed, 16 Dec 2020 02:20:13 GMT

This time, our story begins about five years ago, maybe a little more. My daughter was in her train phase, so she and I would build tracks with the little wooden train sets. Sometimes, we would take over the kitchen floor.

She would have me read Thomas the Tank Engine stories to her at bedtime, and her favorite show was the Thomas & Friends stories from the BBC. Now what does that have to do with JMP? Well, at the time, not a dang thing. I didn’t even work for JMP during that phase. But since I haven’t been traveling (because the world is going bananas), I’ve been going back to my graphic design roots a little and studying data visualization. It’s been a lot of fun.

I’ve been watching video presentations by our own Xan Gregg (), going over my notes from other presentations, reading some Edward Tufte and Stephen Few. I even got to take a couple of classes with Nick Desbarats.

Somewhere in the last few months, something clicked, and I realized that all of these experts in data visualization have been running on the same track (if you pardon the preemptive pun). If you were to boil everything data visualization experts say down to one simple thought, it could come right out of a Thomas the Tank Engine book: All a visualization wants is to be useful.

Flashy trains vs. working trains

In my daughter’s (heavily copyrighted) books, the anthropomorphic trains generally all try to be useful. But they all have their own personalities: Some like to be clean, some don’t mind being a little rough, and some like to be flashy. It takes all kinds, right? But ultimately, the highest praise that could be imparted to the engines by a human is that they were “really useful.” Not that they were pretty, flashy, or rough around the edges, just that they were useful.

If we were to anthropomorphize a graph, it might have a similar feeling. So, how does a graph become useful? Well, it starts with a problem statement, message or finding that the graph is intended to communicate. Let’s look at a few examples, compare them to their stated purposes, and then look at some tweaks to make them more useful.

This graph showed up in my news feed a while back. The stated purpose was to “[show] how activities stack up in terms of coronavirus risk” based on four factors defined in the article.

So, how well did this graph meet the purpose of the visualization exercise? For me, the answer is: “They did OK but could have done a lot better.” We’ll talk about the details of why later, but the core point is that it’s difficult to tell at a glance which activities are risky and to compare risk levels between activities.

In this next example, I’m going to take some of my own medicine and use a visual I created for my 2020 Halloween article. I analyzed the text for Orson Welles’ broadcast of "The War of the Worlds." You can have a look at my analysis here. The purpose of this visualization was to summarize all the terms in the radio script and highlight important themes or terms.

How did I do? I’d say I failed miserably. Why? Well, again we’ll get into the details later, but things like word clouds drift more into the realm of data art. You see word clouds used as logos, marketing materials, T-shirts, etc., because they’re fun or visually striking. As a tool for communicating insights -- even with helpful highlighting -- they’re pretty awful.

Ah, spider plots…my old nemesis. Sorry. I have to admit a certain bias against these. But let me see if I can get you to see why they bug me so much (sorry about the pun). Here’s the caption for a spider chart from a journal article: “The spider diagram of the five sensory attributes for 14 types of beer is shown in [the graph]. Different samples had their individual sensory character, and samples 8, 10, 14, 11, 9 and 2 had the strongest sour, bitter, sweet, goaty, acerbity and ‘other tastes’ individually.” And, here’s the graph:

Yep. Pretty bad. But it’s a fairly common situation in these charts.

One downside of doing the work to learn more about data visualization is that once you’ve seen the problems, you really can’t unsee them. And the problem in our current world is that there are a lot of graphs being produced daily. Between the pandemic and the US elections, we are inundated with graphics. While these visualizations can be flashy and polished or rough around the edges, for the most part, a lot just aren’t useful. So, we have a world of sad, unfulfilled graphs. (That’s a sad thought, isn’t it?) Well, it’s the holiday season, so let’s fix that. First, we need to get to the underlying problem.

Pretty may not be useful, but useful is always pretty

Why are there so many non-useful graphs out there? (I’m not going to call them bad graphs; it’s not their fault that they aren’t useful -- they were just drawn that way). My opinion on this is that it boils down to people missing a single concept: Things can be pretty without being useful, but something that is useful is inherently pretty. And by pretty, I mean that there is an elegance of form that comes from something perfectly meeting its intended purpose.

Nick Desbarats uses the Greek term telos to describe this concept. A lot of us in this business (myself included) can get so hung up on pushing the boundaries of visualization that we forget data visualization is about communication. Once we’re visualizing for the sake of visualization and lose that communication piece, our graphs are in danger of becoming non-useful. Now, that’s not to say that those non-useful graphs aren’t beautiful, just that they aren’t optimal communication tools.

Stick to the easy track

So, what are we to do? I’ve shown you a bunch of graphs that aren’t really as useful as they could be. And because they aren’t as useful as they could be, they are sad graphs. We don’t want sad graphs, do we? No! So let’s fix things.

In those first examples, the issue is that the authors aren’t playing to the strengths of the brain’s visual processing systems. If we were to follow the line of reasoning laid down in Daniel Kahneman's book Thinking, Fast and Slow, we could say we have two systems for processing information: One operates unconsciously and is insanely efficient (System 1); the other is designed for handling complex problems but is more deliberate (System 2). Using this second system is also more energy-intensive, so we run the risk of tiring our viewers if we use that one. So, when people have to think too hard about graphs, they aren’t useful (i.e., sad graphs) and our brain gets tired (sad brain)! To fix this problem, the trick is to use that first, more instinctive processing system.

System 1 is designed for fast pattern recognition, judging linear distances and similar tasks. Think of it this way -- if it were needed thousands of years ago for split-second life-or-death decisions, it’s going to be processed by System 1. That’s why data viz experts harp on the idea that using bar charts is better. We’re better at judging linear distances (straight lines) than curves or areas. We generally process bar charts with System 1. Another way to say this is a person is hardwired to quickly read and process the information in a bar chart. The shortest distance between two points is a straight line, and the fastest path through our brains happens to be one, too!

Going back to that first graphic, the root problem is that we are creating a visual that requires us to use System 2. We have to create some arbitrary scaling, quantify each square for an activity against that scaling, and sum up the values for each square. All just to get a total risk score. And then we have to do that again if we want to compare activities. (Just writing that makes me tired!) So, why not do all that in the graph? Here are some alternatives (two that I created in JMP and one from the Texas Medical Association) that are geared toward using System 1:

See how much easier those are to read?

Let function drive form

Let’s think again about the tank engine that started this whole line of thought. Have you ever considered why a locomotive looks the way it does? My grandfather is a big fan of model trains, so I guess my musings on this topic might be genetic. Or they could be a result of the summers I spent as a child looking at his trains or going to a store to collect his latest acquisition. Either way, I’ve always found a certain elegance to the look of a steam locomotive.

Source: https://en.wikipedia.org/wiki/Steam_locomotive_components

Even when they decided to modernize the look, function drove the form:

Source: https://en.wikipedia.org/wiki/Steam_locomotive#/media/File:Number_4468_Mallard_in_York.jpg

Similarly, we should always start a graphing exercise with a purpose statement of some kind. Remember, for a graph to be useful, it must have a purpose. If we’re doing exploratory analysis, the statement might even be a question that is driving the investigation.

In the case of the word cloud I used, the goal was to show the most important themes in the script. The word cloud doesn’t do this well. As I said, it’s more “data art” than “data visualization.” Word clouds have a problem with fooling visual perception, since longer words “appear” more important in a word cloud simply by virtue of their length. In the example below I've processed the data a little and changed the visual to a parallel plot. It's better to track each character's or group of characters' association with different topics, providing a more insightful view that is less likely to mislead the viewer. (This is interactive, BTW, so feel free to click around with the Local Data Filter on the left.)

The key point here is that aesthetics are important, but the foundation of a useful visualization is a clear statement of purpose and adherence to that purpose.

Don’t get wrapped around the axle

Probably the easiest way people try to make a graph sexy (yes, I called a graph sexy; get over it) is to take a Cartesian graph (with x,y axes) and wrap it around a central axis. This results in things like pie charts, radar charts, spider charts, etc. And they are sexy...but are they as useful as they could be?

The answer to that question is generally, “no.” Humans aren’t wired to compare distances or sizes in a radial system (pie charts, when used correctly, are an exception to that rule). Things can also get cluttered really quickly. And, unfortunately, a lot of people in the data viz world fall in this trap, including the experts.

Oh! And, rather than going after the spider chart (…again and again), let’s all agree with Xan Gregg that spider charts aren’t particularly useful and look at some different examples of this problem.

The Arctic Death Spiral (yes, it’s enough of a thing to get title case) is a solid example of the problem. The name does evoke drama, and maybe that’s the ultimate persuasive goal of the visual.

The direct goal of the visual is to communicate that sea ice volumes aren’t recovering year over year and, in fact, the situation is getting worse. The problem (with the visual) is that by transforming the data into a radial coordinate system, we’re now working with System 2 cognition, meaning our brains have to work harder to get to the point. So, why not just go for a System 1 solution, like this one?

It’s pretty clear from this graph that: a) there is a downward trend, and b) we’re well below anything we’ve seen in the past 40 years. Further, by using a bar graph we are working with System 1, which means you probably reached those conclusions a lot faster than with the first graph.

This second graph is one that Alberto Cairo uses to gauge visualizations on six scales. He originally proposed it in his first book Infografia 2.0, and then discussed it again in The Functional Art.

The purpose of the graph is to communicate on the six dimensions how a graphic performs on complexity vs. intelligibility. In the context of our first example, it’s scoring a graph on (conceptually) how much it uses System 1 vs. System 2. Those dimensions are laid out as pairs of design concepts (Abstraction vs. Figuration, Functionality vs. Decoration, Density vs. Lightness, Unidimensionality vs. Multidimensionality, Originality vs. Familiarity, and Novelty vs. Redundancy). But, by fanning these dimensions around a central axis, the reader has to scan across the graph to see the score of each side of the pair. (Also connecting the scores results in something that looks uncannily like a spider plot, but we’ll let that point pass.) Why not just do something like this?

The same information is communicated, and it relies on that speedy System 1 cognition to communicate it! So, you get the information faster.

I guess the point I’m trying to make here is that, unless there’s a really good reason, avoiding radial plots is probably a better course for graphs that are really useful.

Always be truthful

This can be a touchy subject, and I’m not about to accuse anyone of purposely misleading others. The problem with truthfulness in analytics comes when we let our personal biases creep into the products of our labors. Data analysis and data visualization have an element of subjectivity built into the science. There is an element of opinion, or at least data interpretation, present in any visual you create. I’ve heard it said (a couple of times by Xan at Discovery Summit and JMP On Air) that all charts are biased, and some are useful (kind of a riff on the famous George Box quote). The important point in making a truthful visual is that the choices we make during its construction and data interpretation process are both transparent. It can be as simple as putting the important point in the graph title:

Or providing a helpful annotation:

The important point is that our graphs need to own their assumptions and conclusions and not try to hide them.

Wrapping things up

There are a lot of things I haven’t covered in this article. Truthfully, in the context of the point I’m trying to make, those things are all simply different strategies for creating useful graphs. If we stick to the goal of making really useful graphs, the rest will naturally follow.

Radial graphs aren’t generally as useful as their linear counterparts; our brains just aren’t wired that way. Failure to call out your assumptions and interpretations makes the graph less useful because your readers don’t understand your thought processes or unconscious biases. Most importantly, the lack of a problem statement or thesis for your visual will invariably make it harder to make a useful graph.

So ultimately, let’s all remember, a useful graph is a happy graph. And we should all do our best to make our graphs as useful and happy as possible. And may all your graphs (and your holidays) be happy. Now, the snow is starting to fall up here in ‘Toga Springs, so I’m off for some more cocoa and contemplation. See y’all in 2021.

Machine learning: What you really need to know

anne_milley — Thu, 10 Dec 2020 21:20:36 GMT

Procter & Gamble’s Principal Statistician Bill Myers

Machine learning, artificial intelligence and other buzzwords are thrown around a lot, but what do these terms mean other than methods to solve problems? While many see them as automated problem-solving methods (and there is a place for automation), these methods are no substitute for clarifying the problem at hand and thinking critically about the questions you hope to answer. So says Procter & Gamble’s Principal Statistician, Bill Myers, in his plenary for Southeast Asia’s first Statistically Speaking.

Bill’s extensive experience in realizing value from data is evident in the wisdom and examples he shares. He provides context for machine learning in the analysis process from data to value. He even speaks about the synergies machine learning methods can have with other analysis methods, including design of experiments. And he models the roles that mentoring and collaboration play. He was a critical component of Procter & Gamble’s participation in collaborative research with Georgia Institute of Technology, which won the 2020 SPAIG Award (Statistical Partnerships in Academe, Industry, and Government).

Miao Chen, Data Scientist at TEL Singapore

Following Bill’s plenary, Miao Chen, Data Scientist at TEL Singapore, provides great examples of where automation can speed extracting value from data, furthering my assertion that there is a place for automation. From more easily sourcing data, freeing up resources, and strategically using JMP scripting to automate analyses, these examples show how automation can facilitate the data-to-value process.

Both speakers understand how to leverage analytics to build and maintain competitive advantage. The livestream attracted many viewers from many time zones, but the on-demand version is always available at your convenience.

Outliers Episode 1: The elusive outlier described, visually identified, and judged

JerryFish — Tue, 26 Jan 2021 21:29:01 GMT

A quick quiz: What are outliers?

Odd data points that could influence modeling results and need to be addressed.
Odd data points that warrant further investigation, possibly providing new insights into your process.
A really good book by Malcolm Gladwell. (Outliers, 2008)

The answer, of course, is “All of the above.”

If you are in the business of looking at data, you have run across “outliers” from time to time, those numeric readings that don’t fall within the normal pattern of the data. As a result, our least squares fits of the data (for example) could be distorted. Hence, we need to quickly identify them and deal with them appropriately.

This is the first in a blog series that looks at outliers. Today, we will look at visually identifying outliers. In the next episode, we will examine whether outliers are generally good or bad; the last episodes will examine different means and algorithms used to detect outliers.

Examples of Outliers - Visual Identification

One of the easiest ways to identify outliers is visually. Hopefully, you are already plotting your data to look for trends, etc. Depending on how you plot your data, outliers will often be obvious. The examples below help to describe outliers using some simple plots.

Example 1: One-dimensional data

Figure 1 shows a dot plot of 1,000 samples pulled from a normally distributed population (mean=0, standard deviation=1).

Figure 1: An example with one-dimensional data.

The point at X1=4 is highlighted in red. In this case, it is 4 standard deviations from the mean. For a normal distribution (like the one shown above), this is a very unlikely occurrence. In fact, we would expect to draw a “4” from a normally distribution with mean 0 and standard deviation of 1 only 0.01% of the time. This is definitely an unusual observation.

Example 2: Two-dimensional data, independent and normally distributed variables

In the example shown below, we have two input variables: X1 and X2:

Figure 2: A 2D example, with independent and normally distributed variables.

In Figure 2, we have two normally distributed and independent variables, X1 and X2. The histograms next to each can be used to surmise normality. Note the point marked in red at (4,4). Again, we have an outlier that is visually obvious, since it is located far away from the means of either X1 or X2.

Example 3: Two-dimensional data, normal and independent variables, different means and standard deviations

Let’s go a little further with the two-variable example. This time, we have different means and standard deviations.

Figure 3: A 2D example, with different means and standard deviations.

The point at approximately (1,27) might be considered an outlier. It is inside the X1 distribution, but clearly outside the X3 distribution.

But what about the red point at (4,5)? Is it an outlier? Our eyes tell us that it might be, but how can we tell? How do we assess it algorithmically? (For the answer to this, you'll have to wait for subsequent blog posts!)

Example 4: Two-dimensional data with correlation between the variables

Below is yet another two-dimensional example, this time with correlation between the two variables:

Figure 4: A 2D example, with correlation between the two variables.

Visually, the red point at (-3,6) in Figure 4 is quite obvious to the eye. But looking at either variable independently would indicate that this point is well behaved in both the X1 dimension and the X4 dimension. (This is an excellent example of why it is important to plot your data!)

Example 5: A trickier two-dimensional outlier

Another odd outlier in two dimensions is shown in the figure below:

Figure 5: A trickier two-dimensional outlier.

Again, it is obvious to the eye, but more difficult to detect algorithmically. Still, there is a way! (See future blog posts.)

Example 6: More than two dimensions

What if we have more than two dimensions? Things become more difficult to visualize. For three dimensions, you can try a 3D scatterplot and rotate it around to identify outliers. For more than three dimensions, you can try to make a series of scatterplots to cover all pairs of variables (such as in the Analyze/Multivariate platform), but remember, the outliers may be difficult to spot.

As dimensionality increases, we start to turn to algorithms specifically developed to help identify outliers, which will be covered in future posts.

Future Blog Episodes

In upcoming posts, we will discuss the various outlier algorithms available in JMP, including how they work on the above examples (and more) and when to use them. These algorithms include:

Quantile range outliers
Robust fit outliers
Mahalanobis distances (standard and robust)
Jackknife distances
Hotelling’s T2
Multivariate k-nearest neighbor outliers

See all posts in this series on understanding outliers.

Answer crucial market research questions

anne_milley — Fri, 20 Nov 2020 20:04:44 GMT

In the first Statistically Speaking on consumer and market research, our featured guests — three amazing and accomplished PhDs — shared a number of interesting ways to gain a greater understanding of markets and customers to derive better informed policy and product decisions and strategies.

Roselinde Kessels of Maastricht University, Liz Knapp of Avon, and Laura Castro-Schilo of JMP

We had fun along the way, borrowing some poorly designed survey questions from @BadSurveyQ on Twitter and surveying JMP employees about changes to their use of skin care and cosmetic products since the onset of the pandemic.

In her plenary, Roselinde Kessels of Maastricht University shared two discrete choice experiment case studies to help viewers answer the following questions:

Who are my customers?
What drives them toward a product or service?
How can I appeal to them?

The examples she chose are timely. The first was the COVID-19 digital contact tracing application in The Netherlands, including further exploration of the opt-out choices. The second focused on determining COVID-19 vaccine prioritization in Belgium.

In addition to Roselinde, the panel featured Liz Knapp, senior chemist at Avon, and Laura Castro-Schilo, senior research statistician developer at JMP. The methods used in consumer and market research are broad, and our panelists covered a lot of ground: survey data analysis, text/sentiment analytics, exploratory data analysis/data visualization, and structural equation modeling.

We were hoping to have a sighting of Laura’s puppy, Keefe, but he was asleep in another room for the duration of the webcast. Instead, we’ll have to settle for a photo:

There were many valuable take-aways from this episode of Statistically Speaking, including:

The importance of designing experiments carefully.
The benefits of collaboration.
Why you should strive to measure and model the unobservable.
How to gain greater insights and aid long-term product planning by exploring a variety of data (including text).

You can watch it on demand.

JMP Clinical 8: Modernize with new reports and integration with JMP Live

geoffrey_mann — Fri, 20 Nov 2020 19:54:50 GMT

With modernized reports in JMP Clinical 8, clinical data scientists can collaborate, publish and share their work with others through JMP Live using a web browser. Pharmaceutical companies’ scientists are now also able to share JMP, JMP Pro and JMP Clinical data and reports with a single technology, JMP Live. Check out JMP Live in action with Dr. John Cromer’s blog post on how to Create live reports in JMP Clinical 8, as well as his earlier post, How to make JMP Live reports more interactive.

Another new feature in JMP Clinical is the periodic safety report, which enhances its already rich adverse event reporting. Interactive development safety update reports (DSUR) and periodic safety update reports (PSUR) for drug safety scientists are typically submitted semi-annually or annually, depending on the regulatory agency and the stage of the trials. Lead developer Rebecca Lyzinski’s blog post, DSUR/PSUR report in JMP Clinical: Assess safety in ongoing clinical trials, provides more details on these reports and regulatory guidances.

To support medical doctors looking for safety signals in adverse events data by running Standardized Medical Queries, JMP Clinical 8 streamlines the visualization of risk differences, as well as organizing tables by narrow terms, broad terms and system organ class. By doing so, JMP Clinical 8 complements views aimed toward statisticians and biometrics users, as seen in the screenshot below. To understand more of the details, please check out Rebecca Lyzinski’s blog post on this new Medical Query Risk Report.

Another priority in JMP Clinical 8 is improving the standardization of reporting with study data and metadata management. This standardization in all of JMP Clinical’s reports, specifically with regard to color and layout, results in a simpler and friendlier user interface. In just one example, color can be universally specified in all graphs and tables for the treatment received; the order in which the treatments are listed can also be specified for easy comparison. Dr. John Cromer’s post, JMP Clinical 8: Controlling how graphs and tables look with Study Metadata (Color and Value order management), demonstrates how to add these options with just a few clicks.

Finally, JMP Clinical Application Programmers Interface (API) allows users to integrate their own reports into JMP Clinical more easily. For users who need to automate adding studies – as well as those who want to develop their own reports that integrate filters and selections with patient profiles, subject filters and derived variables – this API works with all reports in the dashboard simultaneously.

Follow our JMP Clinical Resources page for more blog posts and videos on the latest releases.

JMP Clinical 8: Analyzing safety signals with the Medical Query Risk Report

rlyzinski — Fri, 20 Nov 2020 15:29:40 GMT

In clinical trials, medical queries are used to group adverse events into medical conditions. An analysis of the risk of the medical queries can unearth potential safety signals. Forest plots displaying the risk measurement for each medical query along with a confidence interval help visually detect these safety signals.

JMP Clinical 8 includes a new Medical Query Risk Report to analyze medical queries with a risk plot that includes up to four tables of counts and percentages. The report adds to the existing SMQ Distribution and Incidence Screen reports.

General Options

A new, more flexible option for selecting the treatment or comparison variable is used on the options dialog for the Medical Query Risk Report. Similar to other JMP Clinical reports, Planned, Actual, or Specified Below is chosen as the Treatment or Comparison Variable to Use. However, if Planned or Actual is chosen, then the user can choose which planned or actual treatment variable to use, such as Description of Planned Arm. The values from the chosen variable are then used to populate the Placebo or Comparator Value option. Placebo is the default for this option if it exists as a treatment group.

Other options in the general section allow the user to set parameters to determine which adverse events to include and to set the location of the folder where the medical query files live. The most commonly used medical query files are for standardized MedDRA queries (SMQs). More information is available on these at the Medical Dictionary for Regulatory Activities (MedDRA) website.

Risk Plot

The first section of the report, the risk plot, allows the user to visually analyze the incidence of medical queries. The medical queries are shown along the y-axis and are grouped along the x-axis by scope (narrow or broad). The left panel of each scope displays the percent occurrence of each medical query and is colored by treatment group. The right panel of each scope displays the risk measurement along with the 95% confidence interval.

To the side of the risk plot is a medical query report filter that subsets the plot and tables to the medical queries selected.

On the options dialog, the user can select Broad to run the report on both broad and narrow medical queries or Narrow to run the report only on narrow medical queries. Narrow queries are for events highly likely to represent the condition, while broad queries are for all possible cases that could be related to the condition. The user can also pick whether to display risk difference, relative risk, or odds ratio as the risk measurement on the plot by using the risk measurement drop-down menu.

The plot can be sorted using the options under “Sort plot and tables by.” If Risk Measurement is selected, then the “Treatment Value for Sort by” option is enabled. For studies with more than two treatment groups, this option allows the user to sort the plot by one of the treatment groups. If no value is picked for this option, then the sum of the risk measurements is used for sorting. If Count is selected as the sorting variable, then the plot and tables are sorted by the total count of events across treatment groups. If Alphabetical is selected, the plot and tables are sorted alphabetically by the medical query name, dictionary-derived term, or system organ class.

Medical Query Tables

The report displays up to four tables, Narrow Medical Queries and Terms, Broad Medical Queries and Terms, Medical Queries, and System Organ Class.

The Narrow Medical Queries and Terms table presents the narrow medical query name in italics with the dictionary-derived terms contributing to the query indented underneath. Counts and percentages are displayed for each query and term by treatment group.

The Broad Medical Queries and Terms table similarly shows the broad medical query name with the contributing dictionary-derived terms. This table appears when Broad search is selected as the scope on the options dialog.

The Medical Queries table displays another view of the counts and percentages for all the medical queries by scope.

If “Run Report with System Organ Class” is checked on the options dialog, then the System Organ Class table appears. Each body system or organ class is shown on a row and the medical queries contributing to the system organ class are indented underneath in italics. The medical queries are ordered by scope under each system organ class.

If “Use System Organ Class from SMQ Files” is checked, then the coded terms for system organ class will be taken from the medical query files rather than from the adverse events data set.

Lastly, if the “Show Medical Queries/Dictionary-Derived Terms in separate columns” option is checked, then the Narrow Medical Queries and Terms table and the Broad Medical Queries and Terms table will display the medical queries in the first column of the table and the dictionary-derived term in the second column. This option also splits the system organ class and medical query into two columns in the System Organ Class table.

By using the plot and tables from the Medical Query Risk Report, the user gains a better understanding of the safety data, especially when combined with the other SMQ and adverse event reports presented in JMP Clinical 8.

What makes for a strong paper or poster abstract?

arati_mejdal — Wed, 18 Nov 2020 18:25:16 GMT

We have a call for content open for Discovery Summit Europe 2021. You may be thinking of sending in an abstract showing how you used JMP to solve a problem. And you may be wondering what factors the steering committee considers in evaluating paper and poster abstracts. Here are some tips.

For papers

Show how to use JMP interactively.
Instigate conversations and open dialogue about how analytics can be spread across organizations, not stuck in isolated pockets.
Describe real-world problems solved by the strategic implementation of analytic methods and statistical techniques.
Showcase an interesting application you’ve built with JMP Scripting Language.
Describe an “aha!” moment of discovery and how you got there.
Advance and shape the application of statistical methods in a multidisciplinary setting.

For posters

Perhaps your application or research is better suited for a smaller, niche audience. In that case, consider submitting a poster abstract. Posters can depict a class assignment, a research project or a business application. Academic, corporate and individual users of JMP may submit entries. Posters are judged based on their originality, innovative application and/or the use of visualization to express the data.

We are seeking posters that:

Exhibit substantial evidence of the nature of a designated problem.
Present the methods used to reach a viable solution.
Provide a solid explanation of that solution.

Deadline for submissions for Discovery Summit Europe 2021 is Nov. 30. Submit your abstract here.

DSUR/PSUR report in JMP Clinical: Assess safety in ongoing clinical trials

rlyzinski — Fri, 13 Nov 2020 16:24:32 GMT

When a clinical trial is ongoing, periodic safety updates to regulatory agencies are required. The Development Safety Update Report (DSUR) is used for drugs still under development to assess risk to the subjects enrolled in the study, while the Periodic Safety Update Report (PSUR) is used for drugs already on the market to assess long-term safety. Both reports evaluate adverse events and adverse reactions that occur during the trial, since tracking adverse events is crucial in determining if any serious reactions are related to the study treatment. The reports also include the demographics of the study subjects, whether subjects complete or discontinue the study, why subjects discontinue the study, and exposure to treatment information.

JMP Clinical 8 includes a new DSUR/PSUR report that produces up to five tables and three listings that can be used in these regulatory submissions.

General Report Options

Like other JMP Clinical reports, the DSUR/PSUR report has an options dialog with some general options at the top. The general options allow the user to pick the treatment or comparison variable to use, the analysis population, a saved subject filter, and whether to merge any supplemental data sets. In addition, there is an option to use a data cutoff date for ongoing subjects. In ongoing clinical trials, it is common for exposure end dates to be missing for some subjects who have not completed the treatment regimen. To calculate exposure duration for these subjects, a data cutoff date can be entered in lieu of the exposure end date. The data cutoff date is typically the date the data is pulled from the Electronic Data Capture (EDC) system for the analysis being performed.

Under the data cutoff date, users have a set of checkboxes to designate the tables and listings to include in the results. A total of five tables and three listings can be selected.

At the bottom of the dialog, users can opt for either an RTF or PDF version of the report, set the number of decimal places to use for summary statistics, and decide whether to include rows on the tables for missing values.

Demographics and Baseline Characteristics

Once the report is run, up to five sections of interactive tables will appear. The first section is for demographics and baseline characteristics, which includes tables for age, age group, race, sex, ethnicity and country. Summary statistics are provided for age, including a missing row if the “Show Counts for Missing Values” option is checked. The other tables display counts and percentages for each treatment or comparison group, as well as a total count and percentage.

The ranges displayed in the age group table can be changed using the option under Demographics for Set Age Groups. Up to five age groups can be specified. If this option is not selected, then the age groups default to “Age 39 or younger,” “Age between 40 and 64,” and “Age 65 and older.”

Disposition of Subjects

The next interactive table is for disposition of subjects, which summarizes the enrolled and randomized populations when available, the subject status, and the reason for discontinuation.

Under the Disposition options, the “Count randomized subjects with no disposition event as ongoing” option adds a row to the disposition table under status for ongoing subjects in addition to those completed and discontinued. If the “Split into treatment and study status based on EPOCH” option is checked, then the disposition table is split into two sections: end of treatment and end of study. When this option is checked, the user can choose which value of EPOCH from the DS data set to use for treatment status and study status. With these options enabled, the table will summarize treatment status, reason for discontinuation from treatment, study status, and reason for discontinuation from study.

Exposure

Part of a DSUR or PSUR submission includes information about exposure to the study drug. Summary statistics are displayed for duration of exposure in days calculated from the first day that the study drug is administered to the last day the study drug is administered for each subject. Counts and percentages are also presented for the route of administration of the study drug.

On the options dialog, there is one option for exposure to indicate that a 0 dose for placebo or vehicle represents a dose interruption.

Adverse Events and Serious Adverse Events

A major part of any DSUR or PSUR report includes an analysis of the adverse events that occur during the clinical trial. Two tables for adverse events are included in the report, one for all adverse events and one for serious adverse events. Both tables display the counts and percentages based on the group and term levels selected in the options.

The term level and group level to be used in the adverse event tables can be chosen using the drop-down menus under the Adverse Events options. The variables chosen in these options also affect which values are displayed in the adverse event listings. Users may also choose to run the adverse event output solely on serious adverse events or on an event type. If treatment emergent is selected as the event type, then users are given the option to ignore available treatment emergent flags. If on treatment or off treatment follow up is selected as the event type, then the offset for end of dosing and treatment end date is equivalent to the start date options become available.

Review Subject Filter and Report Drill Downs

All the interactive tables described above respond to the global subject filter, which can be expanded by clicking on the gray bar on the left-hand side of the review builder. From here, the tables can be subset based on unique subject identifier, study identifier, age, sex, race, treatment group, study site and country. Additional filters can be added by clicking on the AND and OR buttons at the bottom, allowing you to pick other variables from the ADSL or DM data set.

Drilldowns – such as profile subjects, show subjects, adverse events narrative generation and create subject filter – can be run on any selection made within the tables. These drilldowns are available along the top of the report when a selection is made.

RTF and PDF Reports

At the top of the report, a link is displayed to open the report as either an RTF or a PDF.

These reports will display all the tables and listings selected in the options. The tables in the RTF and PDF reports display the same information as the interactive tables. The listings include one for subjects who died, one for adverse events leading to discontinuation of study treatment, and one for serious adverse events related to study treatment.

The first page of the RTF report is a table of contents, making it easier to navigate the document. To enable the table of contents, simply right-click on the first page and select Update Field. The table of contents appears and can be navigated to any of the tables or listings with Ctrl + Click.

With the new JMP Clinical 8 DSUR/PSUR report, the output needed for periodic safety updates can be generated quickly and efficiently, with the added bonus of interactive tables that are not normally available for this type of analysis.

Sleeping on the job

Peter_Hersh — Wed, 11 Nov 2020 19:16:02 GMT

Sleeping has always been one of my family’s favorite activities. My wife, Kelly, and I have different philosophies on sleeping. I tend to be in the camp with Benjamin Franklin, who said, “Early to bed and early to rise makes a man healthy, wealthy, and wise.” Kelly prefers Mindy Kaling’s viewpoint: “There is no sunrise so beautiful that it is worth waking me up to see it.” Either way, we both enjoy our sleep.

Our mutual love of sleep inspired me to write this post. I wanted to look at the impacts of some of our behaviors on sleep, and Matt Walker’s TED talk gave me the perfect test focus. In the talk, Matt shares how caffeine and alcohol affect sleep. He mentions that caffeine is a stimulant that stays in the blood for much longer than we think. Caffeine has a half-life of 5-6 hours – meaning if you drink a cup of coffee in the afternoon, half of that caffeine is still in your system at bedtime. Matt also mentions that alcohol is a sedative, so it affects the quality of sleep.

Measuring sleep

To measure my own sleep, I used two devices: a Fitbit Charge HR and an Apple Watch Series 3 with the AutoSleep application. I found that the two different trackers did not have the exact same measures of sleep, but the amount of total sleep, quality sleep, and awake time were in good agreement.

Sleep Study

For the study, I needed to recruit some willing victims, umm, I mean participants. Thank you to Eric Hill (), Holly Connors (), Hadley Myers (), Arati Mejdal (), Brian Watts () and Juliette Plager () for your help with this study. All these participants had a Fitbit or Apple Watch to measure their sleep. I asked each of these participants to keep their bedtime routine as consistent as possible for the week of the study, with the only change being if they consumed caffeine or alcohol that evening.

I was interested in different people who had different impacts with the two substances. Kelly and I have different tolerances to caffeine; while I am very susceptible to it, Kelly seems to have a much higher tolerance. Would it be the same for this group, or are different people impacted in different ways?

I made a DOE using the Custom Design platform in JMP, with the person as a blocking factor. Blocking factors let you group homogeneous factors (person in this case) to isolate extraneous noise associated with the factors. This means you expect less variation in sleep from a single person than you would across a group of people. In this setup, I added my block as a fixed effect as I wanted to estimate the difference between each person. To learn more about blocking factors, watch this webcast by my colleague on the topic – it is a great resource.

The design gave me good power to predict the impact of caffeine, alcohol, and the impact of different participants.

Results

As you can imagine, different people sleep for different amounts of time on average, and this group was no different. Sleep time for this group ranged from under six hours to more than nine hours a night.

This person to person variation adds a layer of complexity when analyzing the data. To properly analyze this data, we need to take that added variation into account. For example, if you looked at the impact of caffeine on total time asleep without taking a person into account, there is too much variability to see the statistical significance. Luckily when you create a DOE in JMP and have a blocking factor, JMP knows how to analyze the data.

As you can see from the Profiler, drinking caffeine in the afternoon led to a reduction in total time asleep and quality sleep. Drinking alcohol at night had very little impact on the overall amount of sleep, but it did show a drop in quality sleep, leading to lighter sleep.

Only one participant in the study got more sleep when caffeine was consumed in the afternoon. However, this participant did also have a slight drop in quality sleep as a result. As for alcohol, only one of the participants had an increase in quality sleep after consuming alcohol.

Conclusion

Sleep is important! If you are trying to improve the amount of sleep and quality of sleep, consider cutting out afternoon caffeine or after dinner alcohol.

Winning papers and posters from Discovery Summit Americas announced

Diana_Levey — Tue, 03 Nov 2020 16:14:04 GMT

Attendees at Discovery Summit Americas rated papers and posters presented at the conference. We tallied the votes, and the winners were clear! Because of the hybrid nature of the conference, we gave prizes for both prerecorded and live papers. Congratulations to all!

Best Contributed On-Demand Paper
At the Corner of Lean Street and Statistics Road
by Stephen Czupryna, Objective Experiments
Best Invited On-Demand Paper
Automating the Data Curation Workflow
by Jordan Hiller and Mia Stephens, both of JMP
Best Contributed Live Paper
Towards Predicting the Fate of Reef Corals
by Anderson Mayfield, University of Miami and NOAA
Best Invited Live Paper
ABCs of Structural Equations Models
by James Koepfler and Laura Castro-Schilo, both of JMP
Best Poster
DOE Gumbo: How Hybrid and Augmenting Designs Can Lead to More Effective Design Choices
by Heath Rushing, Adsurgo
Best Student Poster
Predict Customer Churn in Telecom Company
by Kamal Kannan Krishnan, Ayush Kumar, Namita Singh and Jimmy Joseph, all of the University of Connecticut

We hope you'll check out these award-winning presentations.

What's the difference between contributed and invited papers? Contributed papers are those presented by JMP users and customers. Invited papers are by SAS employees. Poster prizes are awarded to JMP users/customers only.

Change your world

Whether you’re focused on the specific analytical task in front of you, or you’re looking at the bigger picture, there are a lot of reasons to keep on learning and growing. JMP offers these resources to expand your knowledge and skills.

The JMP Community

Explore more than 50 Discovery Summit breakout sessions by topic or by type. Even if you attended the conference, there’s a good chance you missed something valuable.
Watch, rewatch and/or share Discovery Summit keynotes that are now posted in the JMP Community, including the Fireside Chat with Shankar Vedantam of Hidden Brain.

JMP Training and Certification

Take any Live Web JMP class through the end of 2020 and get a 25% discount by using the code JDS25 at checkout.
Get a 25% discount off of any JMP certification exam by using the code JMPS20 between now and March 2021.

Statistical Thinking for Industrial Problem Solving

Take one module or all seven. The free, self-paced learning is ready whenever you are, covering statistical thinking, exploratory data analysis, quality methods, decision making with data, correlation and regression, design of experiments, as well as predictive modeling and text mining.

Discovery Summit Europe Call for Content

This is your chance to be in the spotlight for our European event. Because everything is online, there’s no need to update passports or buy airplane tickets. All you need is for the Steering Committee to accept your paper or poster abstract.

JMP Blog articles

We’re doing things a little differently with Discovery Summit Europe

Impulse sus esfuerzos de mejora continua con JMP

JMP para optimización, fabricación y calidad de fármacos: Parte 3

Data visualization: Facilitating the analytic workflow for scientists and engineers

Cringeworthy statistics statement #2 explained: Statistical significance vs practical importance

My take on Cringeworthy #2

Mark your calendar for The Effective Statistician’s 200th episode

JSL自動化分析，4步驟讓你的報告自動「跑」起來！

使用JSL 任務計畫程式

步驟1：啟動任務計畫程式

步驟2：創建任務並命名

步驟3：配置任務啟動時間

步驟4：瀏覽並點擊完成

Retirement income: What you can expect yearly and monthly

雙十一數據解密：電商銷售額與哪些因素有關？數據分析告訴你

Step.1 資料獲取

原始資料來源

資料背景介紹

Step.2 視覺化探索性分析

跳出率高低，關係到網路行銷的成功與否

使用者數值型特徵的多元探索

使用者上網方式對銷售的影響

新老客戶和工作日/週末對銷售的影響

多樣分析結果的集中展示

Step.3 資料預測建模

通過決策樹，篩選影響銷售的關鍵特徵

優中選優，更多資料採擷方法的嘗試

JMP para optimización, fabricación y calidad de fármacos: Parte 2

Tomorrow, and tomorrow, and tomorrow … finding significance and meaning in repeated measures data

A cringeworthy statistics statement: First in a series

Introduction

A simple example: Average height of men vs. women

Is this a "big deal?"

What can we do about it?

Conclusion

画期的なコロナ飲み薬、重症化を89%減少させるってどのような根拠から？

運用JMP淨化離群值，以半導體分析WAT資料為例

什麼是離群值？

善用JMP分析WAT資料，解決半導體製程問題

移除離群值，建構更準確的預測模型

報名11/26半導體研討會

JMP para optimización, fabricación y calidad de fármacos: Parte 1

Celebrating Dyslexia Awareness Month — and neurodiversity in general

Piecewise Nonlinear Solutions Part 3: Using JMP's Formula Editor to solve for unknown parameters

Introduction

Setting up Parameters in JMP's Formula Editor

Entering Parameters for the Piecewise Functions

Entering the Piecewise Formulas - including a Boolean Trick!

Next in the series

Exploring 19th century data with a 21st century lens: The Broad Street pump

Statistically Speaking: Answers to audience questions on design of experiments

Q: Would you please relate sample size with possibilities of having a reliable innovation?

Q: Do you have recommendations on resources to tackle the most challenging part of DOE: choosing factors and defining ranges?

Q: Would you be willing to elaborate on the common mistakes that new DOE users should be aware of?

Q: What is the most important factor to keep in mind when selecting a DOE?

Q: How can artificial intelligence complement DOE?

秩和檢定與兩兩比較的思考脈絡與分析方法

秩和檢定的基本概念

獨立樣本秩和檢定

單樣本秩和檢定

配對樣本的秩和檢定

結論

評估SMT供應商品質有困難？使用JMP讓分析變簡單

01：敘述性統計分析比較

02：假設檢定

03：ANOVA分析

04：管制圖

結論

最新線上研討會 // 資料科學於高科技的6大實例應用

Top papers and posters at Discovery Summit Americas 2021

實驗設計 (DOE)入門：經典篩選設計與全因子設計

什麼是實驗設計(DOE)

實驗設計的三大推手

傳統實驗方法：試誤法、一次一因子和傳統設計

實驗設計指導原則

進行實驗設計目的

使用 JMP 進行實驗六步驟

經典篩選設計VS全因子設計

(1) 經典篩選設計 Screening Design