In my previous post on Lord’s Paradox (1967), I showed how two models, even when fitted to the same data, can lead to opposite statistical inferences. These models are the analysis of covariance (ANCOVA) and the t-test on a difference score computed from two repeated measures of data. In this post, I elaborate on that previous example by discussing a third model: the Linear Mixed Effects (LME) Model.
The ANCOVA model suggests there’s a significant effect of having used a competitor’s product on the second occasion’s customer satisfaction, while controlling for customer satisfaction at time 1 (some might call this an effect of using the competitor’s product on the “residualized” customer satisfaction change). The t-test, on the other hand, suggests there are no differences in how customer satisfaction changed over time between those who did and didn’t use your competitor’s product.
Knowing that different models make different assumptions about the data, you might wonder what would happen if you had fit a LME model to your data. Indeed, many fields recognize LME models as the cutting-edge approach for modeling data with a hierarchical or nested structure (e.g., repeated measures nested within individuals). Here, the LME model can be used to estimate an intercept and slope (i.e., change) for customer satisfaction over time and the effect of your key predictor on them. With luck – remember you only have two occasions of data! – you might even be able to estimate random effects for the intercept and slope too. That is, estimates of how customers deviate from the average trajectory.
The original data require some manipulation so they’re ready for an LME model (scroll down to download the data file from this post, which includes scripts for all LME models below). Let’s take a quick look:
The data now have one Customer Satisfaction variable and a Time variable which indicates the time of the assessment (0 = time 1 and 1 = time 2). You can use the Fit Model platform in JMP with Mixed Model personality in JMP to fit an “unconditional” model for customer satisfaction (Hint: turn off the option to center polynomials because 0 already represents a meaningful value of our predictors, and get indicator parameterization by selecting this option from the red triangle menu). This is a model focused solely on characterizing the customer satisfaction trajectories over time:
Notice there’s no standard error associated with the Residual. This isn’t surprising because we’re asking too much from the data! Our data don’t have enough information to estimate this many random effects. One solution to properly identify the model is to eliminate the covariance between Intercept and Time, which in this case is zero:
This unconditional model gives us useful information about the process under consideration: because Time is coded 0/1, the significant fixed effect of the Intercept suggests customers have an average satisfaction score of 4.04 on the first occasion of measurement that is significantly different from zero (see Figure 5). Importantly, customers deviate significantly from this average value given the significant random effect of Customer ID (this is the variance or random effect for the intercept). Moreover, the non-significant fixed and random effects of Time suggest the average customer satisfaction score doesn’t change on the second occasion of assessment and this is true for all customers. Because this model is just about customer satisfaction, we’re essentially modeling what’s depicted here:
With a better understanding of the customer satisfaction trajectory, we can go back to the original question posed by your manager: Is there an effect of having used a competitor’s product in how your customers’ satisfaction changed over time? Well, we already know customer satisfaction DIDN’T change over time, so you might opt to stop here, but we might discover something new if we explore the results of a “conditional” LME model. That is, one in which our key predictor of interest is entered as a predictor in the model:
As before, we excluded the covariance between Intercept and Time because only two occasions of measurement don’t provide enough information to estimate such effect – this can have important implications that I’ll elaborate upon in the Key Points below.
Our conditional LME model results shed light on Lord’s Paradox and help us understand the big picture.
First, none of the random effects are significant, suggesting that fixed effects’ estimates characterize our whole population of customers. Second, the Intercept and key grouping predictor are significant. Because Time is coded 0/1, the Intercept, 3.54, represents the average customer satisfaction score at time 1 for those who used the competitor’s product (this is in line with the red line in the spaghetti plot of Figure 1). There’s no effect of Time, so the average 3.54 customer satisfaction score didn’t change from time 1 to time 2 among those who used the competitor’s product. The significant effect of the key predictor (the grouping variable) points to a significant difference of 1 unit in customer satisfaction at time 1 between those who did and didn’t use the competitor’s product. That is, those who didn’t use your competitor’s product had an average customer satisfaction score of about 4.54. Finally, the lack of a significant interaction of the grouping variable with Time indicates no detectable difference across time in customer satisfaction for those who didn’t use the competitor’s product.
The LME models provide us with a bigger picture of the patterns in our data, but this comes at the cost of increased complexity in modeling and interpretation of the models – certainly when compared to the ANCOVA and t-test! So I’ll leave you with some key points on Lord’s Paradox and the three models we fit to these data.
Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68, 304-305.
van Breukelen, G. J. P. (2013). ANCOVA versus CHANGE from baseline in nonrandomized studies: The differences. Multivariate Behavioral Research, 48, 895-922.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.