Golf is Hard, but Designing Experiments is Easy

Hello, everybody. This is Don Lifke, statistician with Sandia National Labs. It's nice to be here. I'm going to give you a little bit of a presentation that is off of the normal work I do, and when we get into it, you'll understand why. But welcome, and pull up a chair, and let's get going.

I'm going to give you a little bit of background. I'm going to give you a little bit of PowerPoint, but we're not going to do death by PowerPoint. We're going to exit out of this lovely software and go right into JMP and do some demo live.

At Sandia, the program that I typically work in is the nuclear deterrence. We have five other areas. As a statistician, I really work in all of these areas, but more typically, I'm in the nuclear deterrence group. I use JMP in all of these programs in just about everything that I do. Again, I'm working as a statistician, but I'm actually an electrical engineer by trade. By education, I should say.

Nuclear deterrence, some of the programs that we work on are just… It's a cam slide from our corporate presentation, so I want to show you just a few of the things that I've used JMP on. We've used it on the B61 program, the W80-4. A lot of the components that go into weapons. Like they say, our weapons are used every day. They are a deterrent and a very effective one at that.

These are several of the programs that I've used JMP on. Actually, on this screen, all except one, I've worked with the engineers on these programs and have used JMP pretty extensively.

Last year I was at the JMP conference, we were in Indian Wells, California. Beautiful place. Amazing. I would have never imagined having a hotel room on a golf course. It was really cool. Great place. I got an idea while I was at this conference. We had a night called Shots in the Night, and it was a fun evening event. If this is your first JMP conference, of course, you're finding out how much fun these conferences are and how they keep you busy. You do not have any spare time. Your evenings are pretty busy.

This event was really cool. It was at the driving range at the hotel where the conference was at the Indian Wells Golf Resort. Let me see if I can get this video to play. We're going to give it a shot here, it's about [inaudible 00:02:31] seconds long. I don't know if this would show on your screen.

All right, there you have it. I just thought that was a cool little video. The neat thing is that these golf balls glow in the dark. They set them in a little box that has lights on it, and you can see where you hit the balls. Here's some pictures from last year's conference I hope to be sharing here if I can get my presentation to roll again.

This is actually the driving range that the JMP conference took over. It was really a huge, huge event. Everybody was there. The nice thing is they brought in little food trucks. You'll see Jeff here on the right. You probably recognize him as your master of ceremony for many of the JMP conferences. I think it's kind of funny that Jeff's got a little halo above him. I think that's coincidence more than anything. Maybe not. Maybe there's something going on there, Jeff.

These little screens behind you, I'm going to show you a close-up of one of these. These hitting bays are pretty awesome because what they do is they show you great statistics as you hit your golf ball. It's showing you the total distance that your ball goes. It was showing how far the ball carries, how much was flat, the ball speed, etc. All of these variables, 10 of them total.

Of course, the one I really care about is this total distance. There's some other things in here that you might be concerned about as well, and it captures that, and so I captured that as well. But at that time I thought, "This is really cool. This would be a great time to do an experiment on my golf game and see if I can improve my trajectory a little bit, the distance that I hit a golf ball." And I thought, "This is great data."'

I've actually done a similar experiment many, many years ago at a JMP conference, but I had to use GPS tracking and go out and do some calculations, and it was kind of a pain. But this was great because it tells me the total carry-up in the upper left-hand corner of the screen, and much easier to collect the data.

With that, let's do an experiment. What motivated me was that the fact that I can't really share a lot of the experiments that I do because they're not unlimited unclassified release. I need teaching examples that I can share publicly. This is a really good opportunity to do that because I think most people can understand at least the basics of what's going on with the golf swing and the variables that I'm going to show you.

This is just a quote as I was working on this presentation, I was sitting there watching the Golf Channel, and I saw this pop up, and I thought, oh, "That's neat. I'm going to take a picture of that." It really is about that in golf, to try and minimize the uncertainty, which they say, maximize the certainty.

The variables that I considered in my experiment, I'm going to show you visually, and then we'll go in, and we'll design the experiment and do the analysis. But just so you understand the grip, I, considered three different types of grips, an interlocked grip shown on the left where the pinky and the finger are overlapping on the grip, the split grip, and then I put a variable in between that where instead of the split grip, the hands are actually touching, so I call that a touching grip.

Again, three different types of grips. I looked at the club head and I looked at three different club heads here. I looked at the very closed club head on the left, a neutral club head on the right, and somewhere in between that which I called a closed club head. I found that in my golf game, it seemed that I get a little bit more distance if I close the club head a little, but I'm not sure because I've never really measured it, so this is a great variable to include in the experiment.

Then the right hand, there are just a couple of right hand settings here that I used. I used a neutral and a strong right hand. The neutral is just your typical golf grip. A strong right hand is where you rotate, and I'm right-handed golfer by the way, you rotate the right hand just a little bit more away from the middle of the club. It gives it a little bit of a stronger swing, I think. It tends to pull the club, pull the ball a little bit more to the left, and cause a little less predictability. But I thought let's include that as a variable too. Why not? It's free. Nice thing about DOE is you can throw in a lot of variables and not really increase your runs very much.

The right foot, I've noticed also when I'm golfing that if I back up my right foot a little, sometimes it cures my slice. I don't think there is a cure for my slice, but it tends to help a little bit. I thought, "Well, let's throw that in there too." Pulling the right foot back just a little bit.

Lastly, the position of my ball in my stance. In this experiment, I'm hitting a nine iron, so I'm not hitting a driver. A nine iron is more typically positioned back in your stance or maybe in the middle, but not so much forward in your stance. But what I've noticed when I'm playing golf is that if I do put the ball forward in my stance, I tend to hit it a little further. It seems to contradict what the pros tell you, which with the higher numbered irons like a nine iron, pitching wedge, etc., put the ball further back in your stance. That doesn't seem to work for me, so let's throw in the experiment and find out.

The experiment was pretty easy to design. In fact, I did this right after the Shots in the Dark event, and just went back to the hotel room, and designed this experiment in JMP. I thought, "Okay, I'm going to run this on Thursday when the conference ends." The conference ended right after lunch, and I thought, "I'll just go out to the driving range and run this experiment, and I'll put the data in, and I'll be able to analyze it real-time."

This is the experiment, I'll go into JMP and show you this here, but I want to show you on the screenshot here how I put in all of the factors. I've got some responses I threw in there too that'll show up, but the main response I care about is just how far I hit the ball. But with the… One, two, three, I've got five different factors. I've got three times three times two times two times three, so I got quite a bit of possible combinations here.

But as you know, with Design of Experiments, we take care of that by not considering really high order interactions like third and fourth and fifth order interactions. In fact, with this experiment, I'm only looking at the main effects. I'm not considering Interactions. That's something down the road where I can go back and maybe augment this design and start looking at some two-factor interactions. But for now, I just want to know which of these things matter with the main effects. The JMP tells me I should use 18 runs and that's just about right. That works pretty well.

This is the script that was created. I'm just including this as a screenshot in the presentation, so that if any of you want to duplicate this same exact experiment and tinker with it, you can set that random seed to the same that I did not set it to, but I was given randomly, so this is the script.

Then the design experiment looked like this. Again, I'm going to jump in to JMP here, and we'll go into that. But you can see how it laid out my 18 runs—I'm calling this a run—and within each of those runs, I have my settings of my variables, so my first run would actually be an interlocked grip, with a neutral club head, with a neutral right hand, moving my right foot back and putting the ball forward in my stance. I get on the driving range, set that up, and that was my first hit.

A quick look at what Design of Experiments does for you. Notice that the main effects here in this color map on correlations are almost not correlated whatsoever. There is a slight correlation between the right hand and the right foot. It's 1/9.111. Very small correlation here. But it's nice that I can do an experiment with five different variables, some of them with three settings, and have no correlation between my main effects. Very cool. That's the power of DOE.

I hopped out to the driving range after lunch on Thursday. Conference was over. Grab some lunch first in that beautiful restaurant overlooking the Indian Wells golf course, then headed down to the hitting bay. Here's a picture of the clubs that they gave me. Interestingly, this is exactly the same driver I use. I believe these are very similar to the irons I use at home as well. I set this experiment up, based on the pictures on the time stamp on my photos, which I'll show you. My first shot and last shot were only 20 minutes, so it took me five minutes to design this experiment and 20 minutes to run it.

Here again is my settings for my first shot. Run one, these were the settings when I got up and hit the ball, and this is what the screen looked like. I would capture this data just by snapping a picture of it because I don't want to be hesitating too much between hits. I captured all of these photos and that's my data. Then I sat down and put this data into JMP. I had my laptop, and it was pretty easy to just put the data in and analyze the experiment. Let's go to the data.

Here is my data. When I was at the hitting bay, the only thing I entered at that time was the total. That's really what I care about. I've since gone back and filled in all these other variables that are on that screen, so that I can model those as well. When you do design an experiment and JMP, if you use DOE and Custom Design, this is what I use. This is where I added my factors. I'm not going to bore you with watching me type, but I basically added my two-level discrete numeric factor, my three-level discrete… I'm sorry, that is not correct. Remove not discrete numeric.

My categorical factor, I had two levels, I had some three levels and those were five variables. Then for my response, I basically included… I think when I designed this, I thought I would have three responses. It turns out I really only cared about one, and I actually had eight responses. Because if you don't like the responses you put in here when you design your experiment, you can do your analysis to whatever responses you want.

But setting up this experiment was pretty easy in this Custom Design. You can see how I'm just adding variables, I go over here and type my names in, then I would continue in JMP to design the experiment for me. Down here you can see the number of runs, but I'm not typing in the same exact information because I just don't want to bore you.

From this I just hit Make Design and JMP sets up the design for me. My five variables would have been here, and my Color Map on Correlations would have been down here. Let's go ahead and run the DOE Dialog to show you what that looks like.

As I set up this design, I went through this computing of the design, and this is what the software set up for me. My factors that I typed in are up here. This is how I typed it in. Pretty simple. The responses I typed in, I actually thought that I would be getting slice, and ball speed, and distance. I am getting distance, I am getting ball speed. I'm not sure about this slice variable in here [inaudible 00:15:28] curve, but turned out to… I didn't care about that at the time. It's something we can go back and look at later in a future experiment, but for now all I really care about is distance. Ball speed and distance were so correlated that I really didn't need to look at all three of those.

But setting up this experiment, I could type this probably in three minutes. Set up the experiment, tell JMP to make the table. But let me show you this color map on correlations again. The older versions of JMP Color Map on Correlations was actually a color map. Now it defaults to black and white. I usually change it to a color map, but this is that same color I showed you on the previous PowerPoint slides.

Then with JMP you just make the table and it creates the table for you. Now all I have to do is hit these 18 shots using these variable setups and enter my distance here, and then I'm ready to analyze the experiment. I've already got the data in. I'm not going to save that. I'm not going to worry about saving this for you. I'll go back to this Custom Design because I want to show you a little bit on power analysis as well. Let's run the model and see what this shows us.

When you click model, the script is saved to your data table as you entered it when you created the custom experiment. JMP says, "I'm guessing that you want to use these as your effects because that's what you typed in when you created the experiment, so I don't have to redo anything there." My response is I have three of them I showed you in my design. I really don't care about carry or flat carry. For now, I'm just going to remove those, and I'm just going to run. Look at the total. I just removed everything else.

Now we're in DOE Dialog again. Sorry. I'm just lazy. I don't want to type stuff. I'm going to remove this, and I'm going to remove this. I had accidentally selected my effects and removed them as well. Here is my model that I want to look at. When I run this, what I see is a lot of noise, because guess what? Don's golf game is very noisy. No surprise there, right?

I've colored these points by distance just in the data table. This is a JMP demo, let's show you how to do that. Rows, color, or mark by column. You can say I want to color them by total distance. That's what I've done. All of my rows are colored by that total distance. If I had another screen open, where did it go? Now my data table is on top of everything. Remember down here in JMP whatever you had open is going to show up down there.

Looking at this model what I'm looking at is P values. I am also looking at my residuals. This is actual minus predicted. Essentially, it's the distance from any one of these points to the line. If I were to take this line and lay it out horizontally, this red fit line and lay it out horizontally, that's what you would get.

How far are your points when you look at the actual versus predicted? How far are they from that? How far are the actuals from the predicted? I have a pretty noisy game. You can see that my model is giving me somewhere in the plus or minus five yards. That's how much of it is still not explained. I'm only talking about half the variation of distance with these variables.

My root mean square error is actually six. If I look at these residuals of [inaudible 00:19:35] six root mean square error on that stuff. Let's see what else can we look at. I think I want to look at… Let's just go right down. Look at the profiler first and then start messing with the models. If I look at the profiler, what I find is a visual of my effects, and let me make this bigger for you.

When you're doing this profiler, be careful. If you make one graph bigger, it makes them all bigger. You can quickly use up all of your screen. But if I look at my profiler, I can see that just roughly looking without going through what I'm going to go through in a minute, here I'll simplify the model. It looks like my touching grip is best, a neutral club head is best, a neutral right hand is best. A right foot, even with the front seems best and a ball and stance forward seems best. I did that manually.

If you click on your prediction profiler and look at your optimization, I've already got a desirability function in this because when I set up this experiment, JMP says for your response, which I just told [inaudible 00:20:58] about, what do you want to do with that? And I said maximize. When you set up your experiment, you tell it, do you want to maximize it? Do you want to minimize it? Do you want to meet some target? I told them maximize, so it already has this function in here for me.

Let's just let JMP maximize desirability for me. I had done it manually, so it didn't move anything, but let's pretend like these were somewhere else. Then I tell JMP maximize desirability, and it'll say, "Okay, to maximize your total distance, use these settings." This is your predicted total distance with the uncertainty [inaudible 00:21:35] prediction, so 117. I think my furthest hit was about 114.

After I set up this experiment, I actually did some combination runs with these settings. I had a couple of balls and I believe I hit—I've got it when we go back to PowerPoint—115 and 113. That was a nice confirmation that these settings actually did give me the best distance. Now I only had a couple just in the interest of time. It was getting hot, and I was a little hungry, but I did just want to confirm that I'm getting pretty good hits if I do use those settings.

[inaudible 00:22:22] simplify things. Now, remember, all models are wrong, we're trying just to create models that are useful. I didn't say that. You know who said that? If you don't, you can Google it and figure it out. We want to take out things that don't matter because why? They don't really add to the predictability, so let's just simplify the model. You can do that manually here in JMP just by clicking on your effect and removing it. [inaudible 00:22:48].

What's the P value? It's the probability of seeing that large of an effect just due to randomness. I'll spare you the statistical details, but from an electrical engineering term, a signal-to-noise ratio. Cub head didn't matter, so I'm just going to take it out of the model. Right foot doesn't seem to matter, so I'm going to take it out of the model. Now I'm left [inaudible 00:23:11] marginal here.

Statisticians like to use .05 on really good models. I'm going to be a little more generous on this because I have a lot of noise in my game. I'm going to take out grip, and now I have things that are down below .25. If you do, sometimes when you do a stepwise regression, the default P value to put in variables and take them out is .25. I'm just going to leave ball and stance in there. [inaudible 00:23:45] chance that that's just randomness happening. That's good enough for me to leave it in my model because I know I got a bunch of noise in my swing.

Right hand and ball in stance seem to be really the only two things that had a noticeable effect and here's what they look like. For my golf game, it looks like [inaudible 00:24:05] right hand as opposed to a strong right hand. [inaudible 00:24:08]. I've been trying this strong right hand to get rid of the slice, but it seems that my neutral right hand is actually giving more distance. If I just want to look at using the profiler, I can see that a neutral right hand gives me about 5 yards more than a strong right hand.

Keeping the ball forward in my stance, I already had that intuition that that was good for my particular swing, that that was better. Let's close that out. Let's just do a quick fit y by x. I can just quickly look at each variable versus the total distance. [inaudible 00:24:49] total distance, and I'm just looking at each variable [inaudible 00:24:52]. By looking at it over there.

I can look at this statistically and really bore you and drive you nuts. But let's just look at the total. It looks like that touching grip was best. I don't see a lot of difference on the closed versus neutral [inaudible 00:25:15]. A little bit more distance on this neutral [inaudible 00:25:20], but I just want to show you a quick fit y by x channel. It seems like putting [inaudible 00:25:28] distance here, the ball in the center and ball in the back didn't seem to matter as much, didn't seem to be different as much as putting it forward in my stance.

I fit the model. A quick Scatterplot Matrix of all of the variables. I always like to look at this stuff when I'm setting up experiments, and then after I analyze them just to get a feel for what I'm looking for is [inaudible 00:25:57] speed here versus that particular variable. I can see that I covered the design space pretty well.

These variables [inaudible 00:26:10] cover the corners and all of the combinations of these two variables versus each other, and that's a pretty good coverage of all the two variables versus each other. If we look at this and unjitter the points, you can see a little more clearly. But each point here has more than one data point behind it now. But covering all the bases of these variables versus each other, that's what's really cool about designing an experiment properly.

I can also see a lot of strong correlations. This is my flat carry versus my total carry. The variable I care about is total distance. Let's see, total distance. It correlates to flat carry and ball speed and all of that stuff as well as you would expect. Scatterplot Matrix is always a fun little thing to look at.

One last thing I want to talk about is this power analysis. When I designed this experiment, I only did 18 runs. If we look at the power analysis when we set up this experiment, it seems pretty good power here. This is the power to detect a signal if one does exist. Sorry, statisticians, if I didn't say that exactly right. But remember, I'm an engineer acting as a statistician. It looks like a good power here.

But in reality, remember, my root mean square error was more like six. When I do that, I really don't have a lot of power to detect a variable here. But I still detected a couple of things even with this very low power. But with that low of a power, what I might want to do is maybe hit more than 18 balls. I know I can hit… I'll tell you my score. I know I can hit 90 because that's how many I hit in a round of golf, so 18 really shouldn't be my limitation. I should be able to hit that 18. I could go back and augment this experiment and add some more runs to it, maybe get a little bit more power to detect the effect of some of these variables.

Let's go back to PowerPoint. We did our live demo and JMP. Let's wrap it up. In conclusion, what we saw is that my golf swing has a lot of noise in it. That signal-to-noise ratio, it's hard to pick off a signal when you have so much noise. Not surprising. I'm not the world's best golfer. My handicap right now is slightly less than the 15, which puts me at about an average bogey golfer.

On a par 72, I typically shoot one over par, which is add 18 to that, and you got a 90. I'm a bogey golfer, which it's kind of my goal. I want to be a bogey golfer. If I could knock off a stroke a hole or a stroke and a half a hole, I could make $10 million a year doing this. But it is not easy knocking off a stroke a hole, for those of you who golf already know.

What I found was that some factors might matter, but really only the neutral right hand and the ball forward in my stance seemed promising to me. I've implemented that into my game, and I now do more of a neutral right hand than a strong right hand. I moved the ball forward in my stance, especially on my shorter irons, like a nine iron, and I'm getting a lot more distance. It's not all about more distance, but I'm also getting more predictability on how far I'm going to hit the ball.

I talked about my confirmation runs that I hit on the driving range. I hit a couple of balls with those optimal settings and got around 13 on 15 yards. My max, I think, in my experiment was 114. Pretty good confirmation. This is just a picture of one of my scorecards that I was very excited about. I typically would shoot a 45 on this particular day. I shot two over par. I circle my pars, pros they circle their birdies. Amateurs like me, we get excited on pars and we circle those, and I'll double circle a birdie.

I had a pretty good round. I shot two over par. I won't say it was all strictly because of my design experiment and my changes to my swing, but you always show the best data when you're trying to excite people about your project that you're working on.

With that, I would like to dedicate my presentation to my brother. My brother Joe, we lost him to an aggressive brain cancer back in 2020 and late 2020, and Joe taught me the love of golf. He got me interested in this game. It's his fault. And he used to come to the JMP conferences with me, and he was also a former JMP user.

He was an engineer at Sandia as well, working in the satellite division. And so Joe and I used to hang out and go to lunch together every day. I miss him and want to dedicate this to him. Interesting story, down here on the left, this is Joe and me at The Boulders after he found out he had brain cancer. We were in Phoenix because he was being treated at the Mayo Clinic. We went out and got a late start because of a frost delay in Phoenix. Go figure. We got through nine holes, and we thought, "You're going to miss your appointment. We got to leave."

We left in the middle of our round, went to his appointment at the Mayo Clinic, came back and finished this round of golf. Joe was a tough cookie. This ball here on the green is actually a ball where Joe had hit it too far and hit this boulder behind him and bounced back onto the green. I just thought that was a hilarious event and took a picture of that.

This is Joe and Walter White, Bryan Cranston at the golf course where I play, and this is Joe and I out at Paco Ridge Golf Course in Albuquerque. Thank you very much. I hope that this inspires you to use Design of Experiments in all of your work and also some of your play. Thank you.