One-factor-at-a-time (OFAT) Testing

[#1]

Previously, Joseph Morgan has discussed the idea of unit testing in JMP in detail. Some excellent reads:

•  Low Code/No Code JSL Unit Test Development: A guide to developing JSL unit tests with very little (o...
• JSL-Unit: Automated JSL Testing

To keep things simple in this blog, I use terminology from Whittaker (2000) to define unit testing.

Unit testing: Input domain

The term “unit testing” means testing individual software components or a collection of components. Testers define the input domain for the units in question and ignore the rest of the system.

Let us focus on the second piece, where testers define the input domain. What if we have a function we want to test, but want to treat the individual arguments as units?

For example, take the Beta Distribution function from JSL (screenshot from the Scripting Index). The Beta Distribution function is used to model random variables that are bounded between 0 and 1, often used in Bayesian statistics.

We might fix alpha and beta to something like 0.5, and test a number of different values of q.

alpha = 0.5;
beta = 0.5;
Beta Distribution( 0.5, alpha, beta );
Beta Distribution( -1, alpha, beta );
Beta Distribution( 0, alpha, beta );

Now, determining what we expect for each function call to the Beta Distribution function is a blog post on its own (known as the oracle problem). However, assuming we know what we expect (in this case, 0.5, a missing value, and 0), we can test q. Each of the inputs can be similarly checked individually in this way, such as alpha, beta, and even theta and sigma.

One-factor-at-a-time (OFAT) testing

What does it mean if we focus on one argument at a time? By setting the other arguments to a fixed (such as the default) value and changing the value of one argument sounds like testing via OFAT.

When introduced to (typical) designed experiments, we discuss the pitfalls of OFAT. In particular, OFAT is discouraged due to inefficiencies with resources and missing interactions. We can see that this approach to testing has the same issues. However, if we’re testing software, we can reuse these test cases later, so creating these tests and determining the expected results are not necessarily wastes of testing resources. If we find a problem (i.e., the result is not what’s expected), we know which input was changed that caused the problem. This is especially useful if done as part of an application development process, where these tests can be run on a regular basis (again, a topic for a future blog).

Where do we go from here?

If you read this blog and wondered why we wouldn’t change more than one input at a time, rest assured, we will. In fact, future blogs will discuss treating this testing challenge as a design of experiments problem. Stay tuned!

Whittaker, J.A., 2000. What is software testing? And why is it so hard?. IEEE software, 17(1), pp.70-79.