The problem with analytical rituals
Created:
It’s settling into the deep part of winter in Saratoga Springs, New York. This is the time of year I sit with a hot cocoa, look out at the snow, and think. The new year is a contemplative time for many cultures – a time to assess our shortcomings, to make resolutions to improve or change. Saying that I’m introspective this time of year is putting it mildly. I don’t know what it is about winter, but it puts me into a deeply contemplative state of mind. I get a lot of ideas during this time of year for projects and writing. This little musing has been crystallizing all weekend – a dusty older voice in my head, like Jean Shepherd reminiscing about his time on Cleveland Street, except mine is talking about data, analytics, and Mongolian Hot Pot...
Mongolian Hot Pot
During the first week in January, I was in a planning meeting with the other members of the NY JMP Team in NYC. We decided to do a little team building activity around a trip to a Mongolian Hot Pot restaurant in China Town. Here’s are a picture:
Now, before you think this is a travel log, hold up a sec. I do have a point. First, the restaurant is excellent – you should go if you’re in the area. Just Google it. Second, during the meal, we were discussing how they provided instructions for how to “do” Mongolian Hot Pot – the order in which different things were added to the soup to cook and how long they should stay in the broth. 
It got me thinking about how many of my favorite food experiences all had a ritual component. Hot Pot, Ethiopian, Sushi, Thanksgiving Dinner, even making pizza for my family on Friday nights – we have a habit of creating rituals around consuming food. They give the experience added meaning and significance. Food rituals can bring people together and cement relationships. And, in my current contemplative state of mind, it got my wheels turning.
Analytical Rituals
Now, what does this have to do with JMP or analytics in general? Well, I was thinking later that night about how rituals crop up in other parts of the human experience – and I’m not necessarily talking about religious observance here. Over the years, I’ve seen many cases where analytics have become ritualized. And it turns out that this can be a big problem.
Rituals all have a common attribute. Their goal is to teach or remind through actions. They also generally have some kind of framework (cultural, societal, etc.) to reinforce the meaning of the rituals. Analytical rituals start similarly – with an attempt to codify a best practice or something that has worked well for someone so that it won’t be forgotten. These analytical rituals have many names – Procedures, SOP’s, Macros, Templates, Success Stories, etc. And, truthfully, that’s fine, and probably a good thing. The ritualizing of analytical practice does what any ritual does. It adds significance to the practice. By ritualizing analytics, we elevate its importance in an organization – particularly when leadership adopt or endorse the rituals. As commonly is the case, analytical rituals generally become part of an organization's culture – become embedded in its DNA. And that’s where the problems start – when was the last time you had a good think about your DNA?
The Problem with Analytical Rituals
The problem that comes with analytical rituals is that they, like all rituals, can move over time from conscious observance to rote observance, that is, just going through the motions. And when that happens, they lose their meaning. This is particularly easy for analytical rituals because they don’t necessarily have the cultural or societal support structure to bind the meaning to the actions. Analytics, on the organizational scales we're seeing today, are fairly new and don't always come with the inherent tribal knowledge or caution that come with other sciences. And that's a problem.
For analytical rituals, the loss of meaning is particularly dangerous. Since the meaning contains the reason for the analysis, the assumptions that are being made, and the appropriate use cases – the Why of the ritual – the loss of meaning can lead to misapplication of the analysis. Ultimately, analytical rituals stripped of their meaning are dead and dangerous. Over my career, I’ve seen many cases where it’s apparent that a valuable analytical ritual has lost its meaning and devotees are just going through the motions – and maybe not the correct motions at that. Let me give you some examples.
The War on the P-Value
A little while ago, I was reading an article about how some scientific journals were starting to discourage reporting of p-values or of the p < 0.05 criteria for significance. The argument was that people were blindly using the criteria, or in some cases tweaking results so that they could report a favorable result vis-à-vis the p-value. The actual situation is a bit more complex.
At some point in the past, researchers were educated in statistics and began reporting p-values as a convenient shorthand of the significance of findings or strength of trends in their data. This practice was incorporated into an analytical ritual by the journals with a guideline of p < 0.05 being a benchmark for significance. This is a good thing – the p-value is a fine metric for reporting values, provided the context is maintained along with the reasons for choosing particular p-value criteria. Further, a clip level of p < 0.05 for hypothesis testing is widely used and is generally good enough for many applications – provided the practitioner understands that it’s a guideline that has to be interpreted critically, and not by rote.
The problem here came when the meaning of the ritual, along with its context, was lost. Hollow rituals lead down bad roads – as has been seen in disciplines where these p-value guidelines have turned into p-value rules. The p-value rules have in turn promoted or enabled undesirable behavior.
The Case of the Change Point Worksheet
Speaking of bad roads, let's talk about the venerable (or infamous, if you like) analysis template. Many organizations, in an effort to simplify the work of their technical staff or compensate for a gap in institutional knowledge, develop analytical templates in spreadsheets or standard analyses in online analytical hubs. One might argue that this codification of analyses is a good thing, and I would agree, to a point. The organization has put resources towards making it easier to follow a sound analytical practice. As was the issue with the p-values, the problem comes when the meaning of the analytical ritual gets lost. One example in particular sticks out in my mind.
I saw an organization that validated changes to its process based on a spreadsheet template. The template had a colored box and a box and whisker plot. If the box was green, the change was considered acceptable. When I saw this template, I asked the company when it was acceptable to use the template and when it wasn’t. They couldn’t give me an answer other than it was the tool used to validate all process changes. The ritual had lost its meaning. Digging deeper (and reverse-engineering the template), I was able to determine that the analysis was a simple ANOVA – which comes with a certain set of assumptions and conditions under which it can be used. Someone in the past had developed the template to standardize one specific type of analysis and, because of the loss of the analytical ritual’s meaning, it was being wildly misused.
The Problem Solving Methodology
I’m going to tread lightly here, but this next area of analytical rituals might tweak some noses. There are many analytical rituals that come packaged in “Problem Solving Methodologies” or “Continuous Improvement Methods.” I’m not going to name names – you can probably think of two or three on your own depending on your industry. Let me be clear: There is nothing wrong with these methodologies or the analytical rituals that they teach. They are good starting points to help people along their analytical journeys.
The problem comes when the adherents to these methodologies do not have the underlying meaning for the analytical rituals, which is something that I see a bit more often than I would like. This may be caused by many things – because the adherent forgot, because they weren’t properly instructed, or even because the instructor was hobbled by organizational constraints. In any case, the result is the same: systems of analytical rituals devoid of the underlying meaning are recipes for disaster. They lead to inappropriate application of tools or misinterpretation of the results. These, in turn, can lead to mistrust in statistical analysis in general, which is a hard thing to fix.
The Solution
So, what is one to do about hollow analytical rituals? In theory, the fix is easy: Instill the meaning back into the ritual; remind ourselves of why we are doing a particular analysis. In practice, that might be a tall order. People questioning any ritual are met with resistance. For some reason, analytical rituals tend to hit a nerve for some and are particularly difficult to address. I don’t know why, though I have my theories. The truth is that without constant reinforcement of the concepts, any analytical ritual can eventually lose its meaning. Like with Shelley’s Ozymandias, time will always win if we are not vigilant. And, that brings me to the point for this little stream of thought: The only true way to fix a hollow analytical ritual is by fixing it for ourselves.
A New Year’s Resolution
In many cultures, the new year is a time for resolutions. It’s a time to look at our weaknesses and resolve to improve – not to be perfect, just better than we were last year. So, here is my proposal – a challenge or call to arms, if you like: Examine your analytical rituals. Find one that you think you might be a little weak on, and get better at understanding it.
There are some excellent free resources available from JMP to help. The first is a website we provide that has information on common statistical topics. It's called the Statistics Knowledge Portal, and it's updated fairly regularly with new content. The other is a complete course of study in statistics provided by JMP and SAS Education called Statistical Thinking for Industrial Problem Solving (if that's a bit of a mouthful for you, we just call it STIPS). There's also the message board on the JMP User Community. Last, if you need help getting started, reach out. I don’t know everything (ask my wife), but I know a lot of people who are a lot smarter than I am and they love to help. We'll get you pointed in the right direction.
Thanks for sticking through this little article (vanity piece, maybe?). I honestly wrote it down just so that dusty older voice would go away long enough for me to work on some other writing (one of my resolutions).
I do like cocoa and contemplating, though...
Happy new year! Good luck and success with all your resolutions – analytical or otherwise.
Comments