Fly helicopters to elevate your effectiveness when experimenting

[#1]

Sometimes it is hard convincing my kids that I have a proper job. I work from home a lot, and recently they’ve seen me building and flying these paper helicopters:

How is this work? At JMP, we are always trying to find ways to get more scientists and engineers engaged with data analytics. A big passion for me is design of experiments (DoE) because I know how it helps scientists and engineers to quickly find the best solutions to difficult problems.
One of my colleagues came up with the idea of running a competition that would give people hands-on experience of DoE through a real-world experiment. The paper helicopter experiment is a classic example where you can experience the challenge of solving a multi-factor engineering problem.

The engineering challenge and how not to solve it

The aim is simple: Find the design that gives the longest flight time.
Take a minute to think about how you might do that. I’m sure that you can think of plenty of “factors” that you would expect to affect the flight time:

• the type of paper
• how long the wings are
• how wide the wings are
• and so on…

Now, with all these things that you could change, how are you going to work out what is the best design? How will you decide the best combination of factor choices?
I suspect most of you would dive right in and make a helicopter that you think will work and fly it. Then you might build a second one with a change to the design – maybe you make the wings a bit longer – because you reckon that will improve the flight time. You try that one and find it flies for a bit longer. Then for the next build, you might change a different factor.
You keep going like this until eventually you run out of time or energy. Out of all the things you have tried, you will have found one that works best. Job done!

But you are left with some troubling doubts:

• Have I really explored all the possibilities?
• Do I understand the effect of each factor?
• Do I really know why this design worked well?
• Could there be a better solution that I didn’t try?
• Could I have got to a solution in less time and with less hassle?

I speak from personal experience because this is pretty much how I used to work in the lab when I was developing manufacturing processes in the chemical industry – that is, until I learned about DoE.

A better way with DoE

For the paper helicopter competition, entrants were guided to use the DoE strategy to find the best design. They were provided with a template for four different helicopters and instructions for how to build them.

They were told to create another set of the same four helicopters but on a different weight of paper (or to create them on the same paper and add a paper clip) to give eight helicopters in total. By building and testing all eight helicopters, they would have tried all possible combinations of short/long wings, thin/fat wings and heavy/light paper types. Varying all the factors at the same time probably goes against what you have been told. There are some big benefits when you experiment in this way.

What we learn with DoE

A big part of my job is visiting customers for seminars and workshops to raise awareness of what they can do with the software. We ran this same experiment with attendees at one of these visits recently. These are the results from one of the groups. They measured the flight time of each helicopter three times and took the average:

You can see that the best flight time (1.98 seconds) is for helicopter #5, which was made of paper with long, thin wings. Although the same but with thick wings (#6) is almost as good (1.83 seconds). The card helicopter with short thin wings (#3) is worst (0.51 seconds). We can use these results to estimate the separate effects of each factor.

The average flight time for card is 0.78 seconds. The average flight time for paper is 1.52 seconds. Now we can work out that the effect on flying time of changing from card to paper is 1.52 - 0.78 = 0.74 seconds. We can do the same for the other factors.
Better still, we can use software like JMP to fit a model that gives all this information together. This Prediction Profiler is a visualisation of a model fit to the same data using JMP:

Click and drag the broken red lines in the plot to see what happens when you change factor settings. This interactive visual was published on JMP Public along with some more related to this blog post. You can see the effect of each factor on flight time. Paper type has the biggest effect. Wing width is not very important. DoE gives you a rich understanding of your system or process.
With this balanced data set, we can also look to understand more complex behaviours. Is there an interaction between the effect of paper type and wing length? That is, is the effect of paper type affected by wing length? This simple visual of the data suggests that there is not much of an interaction:

The difference in flight times between card and paper is very similar with both long and short wings. In other cases, these interaction effects are very important. If you vary only one factor at a time, you will not have the data that you need to find important interactions.
This is a very simple example to demonstrate how you can learn more and faster with DoE. In the real world of industrial science and engineering, the experiments can get more complicated. For example, you will often have more factors, so testing all possibilities is not realistic. You should know that the DoE method can be adapted to handle practical constraints like this. Whatever the experimental challenge, you can learn more and faster with DoE.

And the winner is…

I will announce the winner tomorrow in my next blog post and share some excerpts from their report. The competition was promoted through Chemistry World, the magazine of the Royal Society of Chemistry. We included the template as an insert in the magazine and the team there created this great video:

Build the best helicopter challenge from Chemistry World on Vimeo.

We asked people entering the competition to send us a short report of their work for their chance to win a DJI Mavic 2 drone. We got a lot of great entries. As a judge, I was looking for work that demonstrated clear thought, effort, learning, and a sense of fun. So,come back tomorrow to find out who won.

For now, you might want to find out more about DoE, so here are some more resources:

• The competition has closed, but you can still get the template and instructions to try the helicopter challenge for yourself.
• Pilar Gomez-Jimenez, principal scientist at Johnson Matthey, recently talked about how to start with DoE and develop the mindset in this webinar with Chemistry World.
• Last year, I set out to write a series of blog posts to demystify DoE with simple explanations of some of the terms and concepts that can be confusing for people when they are starting out. You can find the complete series here.