What you can (and should) do with all that text data

[#1]

We were very excited to feature Manya Mayes, a natural language processing (NLP) expert and Head of Data Science at 1440 Consulting, as the plenary speaker in our first Statistically Speaking on text analytics. We were also delighted to host her as a panelist with two other savvy text analytics practitioners: Weida Tong, Bioinformatics & Biostatistics Director at the Food and Drug Administration; and Jeff Swartzel, Scientist at Procter & Gamble.

The practical expertise these statistically minded experts convey and the interesting examples they share are inspiring. We had so many questions for the panel during the livestream that we couldn’t get to them all. Our featured guests have kindly agreed to answer them via this blog post.

How do you measure and demonstrate ROI for text analytics? 

Manya: As you have anticipated, measuring ROI for text analytics is not always easy. When identifying early warning of a product issue, it may never be possible to identify the true ROI for a problem that is fixed before it becomes a Ford Explorer/Firestone Tire issue (which had an estimated cost of $7.5B) or the Toyota sudden acceleration issue (which incurred $1.2B in penalties from the Department of Justice and $2B in recall costs). Had both companies taken action on text analytics results (which were easy to find using the National Highway Traffic Safety Administration [NHTSA] data and good text analytics capabilities), they may never have truly known how much they saved. The use of text analytics in a labeled and supervised prediction model provides a way to validate the ROI, but this is not always possible. In addition, for the analysis of data that informs future company business decisions and strategy, it is possible to measure the return per dollar invested via a driver/sensitivity analysis.

Jeff: This is tricky, and I often struggle with this question because the impact can be inconsistent. Sometimes very quick and easy work gives valuable insights, whereas other times, a high-effort analysis can yield results that are less impactful. The things I typically focus on as the value from text analysis are: 1) time saved, 2) the ability to intervene ahead of problems, 3) the uniqueness of insights that might not have been possible without the technique, and 4) increased ability to focus on topics of importance.

How was the plot on the screen (whompy wheel vs. time) generated from the Text Explorer window?

Manya: Using Text Explorer on the sample of complaints about electric vehicles, I removed a set of low information terms by adding them to the stop list, then I removed terms that appeared once or twice, and followed that with a latent class analysis. The results of the latent class analysis (in JMP Pro) include a Cluster Probabilities by Row window. In that window, it shows the most likely cluster for each document. It is possible to right-click on Make Into Data Table. The next window contains the results as a data table that can then be joined back into the original sample of vehicle complaints by selecting Tables > Join and then joining these results with the original data, matching by row number. I selected just the rows for the whompy wheel cluster. The original data has a faildate field that can then be plotted using Graph > Graph Builder. In Graph Builder, I used a histogram, with faildate as my X variable, and the Response Scale set to Count. The remainder was mostly cosmetics, where I changed the color by clicking on the faildate variable in the legend (see attached image for more details). To get the information by model, I dragged and dropped the vehicle model variable on the Group X portion of the graph and used the vehicle model as a color variable.

Why are scores for negative sentiments lower than positive sentiments (98 v. 100)? 

Manya: The sentiment scores range between -100 and 100. I just happened to pick negative terms with scores of -98 and positive terms with scores of 100.

These images show that, on the whole, there were more positive documents than negative ones. The scores show the sentiment calculated across the whole document (review), so a document could have an overall positive score, even though it has parts that are negative and parts that are positive.

Is the text analysis for the FDA review process used more in targeting review efforts, or is it supplementary to reading documents? 

Weida: Text analytics has been used in a broad way in the FDA, including both research and review. Among them, information retrieval and document classification have a strong presence in the review process. In addition, we also investigated text summarization, named entity recognition, and sentiment analysis, most of which is a regulatory science-centric endeavor.   

What synergies do you gain with text analytics and other analysis methods?

Manya: The combination of text-based information into other analysis methods is common. The text is essentially given a structured representation. The incorporation of the now structured text can help with increasing model accuracy for machine learning supervised models when applied to a whole host of use cases (customer acquisition, retention, fraud, risk, etc.). It can also help with model explanation, and it can inform the right language to use for branding, messaging, etc. It can also help with unsupervised classification/clustering, where the terms, in conjunction with structured data such as transactional and demographic information, can help describe the clusters.

Jeff: JMP is great for directly connecting the output of text analysis to other relevant variables (time, customer rating, meaningful categories, etc.) for modeling (term selection and discriminant analysis). Relevant keywords and topics can also inform more qualitative analyses elsewhere.

What do you think about the bag-of-words approach vs. more advanced NLP? 

Manya: The bag of words approach does not take context or directionality into account, although phrases can be included, making the analysis simpler and quicker, while still providing value. The more advanced NLP does provide more context, although it still requires pretrained models and the context can be much more localized (more likely to be within a sentence, for example). The more advanced NLP provides greater automation, but potentially misses the nuances involved when a data scientist gets to know the data in detail. Essentially both techniques have their pros and cons.

Jeff: I sound like a broken record, but I always say, “Connect the analysis technique to the question.” Frequently, I’m able to get fast and useful insights with basic methods (especially when the question is clearly defined). People will come to me and ask something along the lines of “We don’t know what we’re looking for, but we want to know everything about everything. Can we NLP our way to omniscience?”  I’ll either help them focus on question definition or start exploring the text with latent class analysis, latent semantic analysis, or sentiment analysis. Also, the bag-of-words approach has the benefit of being very easily conceptualized and interpretable, and it’s a great starting point. In my limited experience, methods like BERT are great for classification questions with well-trained machine learning models.

What does it mean to move toward “advanced regulatory science”? 

Weida: The FDA defines regulatory science as “the science of developing new tools, standards and approaches to assess the safety, efficacy, quality and performance of FDA-regulated products.” In a sense, advanced regulatory science means taking emerging technologies and methodologies to improve the FDA’s operation.

How does JMP facilitate your collaboration with others and with text analytics in particular?

Manya: The JMP Community is an excellent platform for collaborating with others. Text analytics is part of the JMP Community, as well as SKP (the JMP Statistics Knowledge Portal), which is about to become more active with text analytics content and collaboration.

Jeff: At P&G, we have a very large JMP user base, and many people are familiar with the basics. It’s easy to share the output from a JMP analysis and have a working session with someone using Text Explorer. Sharing scripts and journals makes it very easy.

Do you think we have just scratched the surface of what can be done with all the textual data we are collecting? 

Manya: Recently, text analytics and NLP have made some huge jumps in capabilities, but there is still more to come. There is a good scratch in the surface, but with additional capabilities, text won’t be considered any differently than structured data. For now, the human element is still very valuable.

Jeff: Yes, and I think that building a culture of people who are familiar with the fundamentals of this kind of work is the best starting point for building an understanding of analyzing text data

What else do you share with the people with whom you analyze text data? 

Jeff: Sometimes I will make it a point to remind them that it’s not smarter than we are, meaning the analysis techniques aren’t going to be able to look at a sentence like “I stopped using your product because it smelled funny” and understand it on a human level. The funny smell might be an indication of a big problem, or something people typically say about this product, or even part of an intentional change that was understood to be received this way, or even something sarcastic and ironic. Text analysis and NLP always need to be paired with domain knowledge and curiosity around the questions that we want to answer. We can’t (yet) push a button and know all what we want to know (and even what we don’t know that we might want to know).

If you missed the live event, you can watch it on demand for more insights. In addition to the above questions, we had many other excellent questions from our viewing audience. Learn more about “whompy” wheels, funny smells, and other insights you could gain from some of the text data you have collected.