Publications

Latest Posts

MaLT: Machine-Learning-Guided Test Case Design and Fault Localization of Complex Software Systems

Authors Yi Ji (1), Simon Mak (2), Ryan Lekivetz (1) and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC (2) Duke University, Durham, NC Journal 2024 22nd ACM-IEEE International Symposium on Formal Methods and Models for System Design (MEMOCODE) Date Published 2024 Abstract Software testing is essential for the reliable and robust development of complex software systems. This is pa...
Ryan_Lekivetz

Publications |
Jan 31, 2026 7:02 PM

17 views | 0 replies
Power Considerations in Designed Experiments

Authors Luyao Lin (1), Derek Bingham (1) and Ryan Lekivetz (2) Affiliations (1) Simon Fraser University, Burnaby, BC (2) JMP Statistical Discovery LLC Journal Journal of Statistical Theory and Practice Date Published 2020 Abstract Designed experiments for ANOVA studies are ubiquitous across all areas of scientific endeavor. An important decision facing experimenter is that of the experiment run si...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:53 PM

13 views | 0 replies
Assessing the Usability of Statistical Software Using a Discrete Choice Experiment

Authors Jacob Rhyne (1), Mark Bailey (1), Joseph Morgan (1), and Ryan Lekivetz (1) Affiliations (1) JMP Statistical Discovery LLC Journal International Conference on Human-Computer Interaction Date Published 2024 Abstract The usability [2, 3] of modern statistical software is increasingly important as users with limited statistical training become more reliant on such tools to help them address c...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:49 PM

15 views | 0 replies
Applying design of experiments in testing and validation of statistical software

Authors Caleb King (1), Ryan Lekivetz (1), and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal 2024 Annual Reliability and Maintainability Symposium (RAMS) Date Published 2024 Abstract Validating statistical software involves a variety of challenges. Of these, the most difficult is the selection of an effective set of test cases, sometimes referred to as the “test case se...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:45 PM

17 views | 0 replies
Development of a non-invasive method for species and sex identification of rare forest carnivores using footprint identification technology

Authors Jody M. Tucker (1, 2), Caleb King (3), Ryan Lekivetz (3), Remi Murdoch (1), Zoe C. Jewell (3,4) and Sky K. Alibhai (3,4) Affiliations (1) USDA Forest Service, Rocky Mountain Research Station (2) USDA Forest Service, Pacific Southwest Region (3) JMP Statistical Discovery LLC (4) Nicholas School of the Environment, Duke University Journal Ecological Informatics Date Published 2024 Abstract M...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:41 PM

13 views | 0 replies
BayesFLo: Bayesian fault localization for software testing

Authors Yi Ji (1), Ryan Lekivetz (2), Simon Mak (1) and Joseph Morgan (1) Affiliations (1) Duke University, Durham, NC (2) JMP Statistical Discovery LLC Journal 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C) Date Published 2023 Abstract Fault localization is a software testing activity that is critical when software failures occur. We propo...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:36 PM

11 views | 0 replies
Assessing the usability of statistical software using designed experiments

Authors Jacob Rhyne (1), Mark Bailey (1), Joseph Morgan (1) and Ryan Lekivetz (1) Affiliations (1) JMP Statistical Discovery LLC Journal International Conference on Human-Computer Interaction Date Published 2023 Abstract Modern statistical software is increasingly used by users with limited statistical training to address complex real-world problems. Yet, little work has been done on assessing the...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:32 PM

11 views | 0 replies
A family of orthogonal main effects screening designs for mixed-level factors

Authors Bradley Jones (1), Ryan Lekivetz (1) and Christopher Nachtsheim (2) Affiliations (1) JMP Statistical Discovery LLC (2) Carlson School of Management, University of Minnesota, Minneapolis, MN Journal Journal of Quality Technology Date Published 2023 Abstract There is limited literature on screening when some factors are at three levels and others are at two levels. This topic has seen renewe...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:27 PM

10 views | 0 replies
Testing the prediction profiler with disallowed combinations—A statistical engineering case study

Authors Jeremy Ash (1), Caleb King (1), Laura Lancaster (1), Ryan Lekivetz (1), Joseph Morgan (1) and Yeng Saanchi (1) Affiliations (1) JMP Statistical Discovery LLC Journal Quality Engineering Date Published 2022 Abstract The prediction profiler is an interactive display in JMP statistical software that allows a user to explore the relationships between multiple factors and responses. A common us...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:23 PM

11 views | 0 replies
On the testing of statistical software

Authors Ryan Lekivetz (1) and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal Journal of Statistical Theory and Practice Date Published 2021 Abstract Testing statistical software is an extremely difficult task. What is more, for many statistical packages, the developer and test engineer are one and the same, may not have formal training in software testing techniques, and ...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:20 PM

10 views | 0 replies
Combinatorial testing: using blocking to assign test cases for validating complex software systems

Authors Ryan Lekivetz (1) and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal Statistical Theory and Related Fields Date Published 2021 Abstract Testing complex software systems is an extraordinarily difficult task. Test engineers are faced with the challenging prospect of ensuring that a software system satisfies its requirements while working within a strict budget. Choo...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:16 PM

13 views | 0 replies
Direct construction of globally D‐optimal designs for factors at two levels and main effects models

Authors Caleb King (1), Bradley Jones (1), Joseph Morgan (1) and Ryan Lekivetz (1) Affiliations (1) JMP Statistical Discovery LLC Journal Quality and Reliability Engineering International Date Published 2020 Abstract In developing screening experiments for two-level factors, practitioners typically are familiar with regular fractional factorial designs, which are orthogonal, glob-ally D-optimal (...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:13 PM

14 views | 0 replies
Covering arrays: Using prior information for construction, evaluation and to facilitate fault localization

Authors Ryan Lekivetz (1) and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal Journal of Statistical Theory and Practice Date Published 2020 Abstract Covering arrays are increasingly being used by test engineers to derive test cases to test complex engineered systems. This approach to testing is known as combinatorial testing and has proven to be a cost-efficient way to de...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:08 PM

11 views | 0 replies
Construction, properties, and analysis of group-orthogonal supersaturated designs

Authors Bradley Jones (1), Ryan Lekivetz (1), Dibyen Majumdar (2), Christopher Nachtsheim (3) and Jonathan Stallrich (4) Affiliations (1) JMP Statistical Discovery LLC (2) Department of Mathematics, Statistics, and Computer Science, University of Illinois, Chicago, Chicago, IL (3) Carlson School of Management, University of Minnesota, Minneapolis, MN (4) North Carolina State University, Raleigh, N...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:05 PM

13 views | 0 replies
Design Fractals: A Graphical Method for Evaluating Binary Covering Arrays

Authors Caleb King (1), Joseph Morgan (1) and Ryan Lekivetz (1) Affiliations (1) JMP Statistical Discovery LLC Journal 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C) Date Published 2019 Abstract Test engineers are often faced with the challenge of selecting test cases that maximize the chance of discovering faults while working within a limi...
Ryan_Lekivetz

Publications |
Jan 31, 2026 6:01 PM

14 views | 0 replies
Fast flexible space‐filling designs with nominal factors for nonrectangular regions

Authors Ryan Lekivetz (1), and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal Quality and Reliability Engineering International Date Published 2019 Abstract Space‐filling designs allow for exploration of responses when each continuous input factor is set to many different values over its range. While typical space‐filling designs treat all the input factors as continuous,...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:58 PM

15 views | 0 replies
Evaluation and construction of covering arrays utilizing prior information

Authors Ryan Lekivetz (1) and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) Date Published 2018 Abstract Covering arrays are being increasingly used as a tool to determine test cases for testing complex engineered systems. The primary appeal of using covering arrays for this purpos...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:56 PM

13 views | 0 replies
Fault Localization: Analyzing Covering Arrays Given Prior Information

Authors Ryan Lekivetz (1) and Joseph Morgan (1) Affiliations (1) JMP Statistical Discovery LLC Journal 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) Date Published 2018 Abstract Covering arrays are being increasingly used as a tool to determine test cases for testing complex engineered systems. The primary appeal of using covering arrays for thi...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:54 PM

11 views | 0 replies
Covering arrays: Evaluating coverage and diversity in the presence of disallowed combinations

Authors Joseph Morgan (1), Ryan Lekivetz(1) and Tom Donnelly (1) Affiliations (1) JMP Statistical Discovery LLC Journal 2017 IEEE 28th Annual Software Technology Conference (STC) Date Published 2017 Abstract Test engineers are often faced with the challenge of selecting test cases that maximize the chance of discovering faults while working with a limited budget. Combinatorial testing is an effec...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:51 PM

11 views | 0 replies
Designs of variable resolution robust to non-negligible two-factor interactions

Authors Ryan Lekivetz (1) and C. Devon Lin (2) Affiliations (1) JMP Statistical Discovery LLC (2) Queen's University Journal Statistica Sinica Date Published 2016 Abstract In some experimental situations, there is a natural grouping of factors. Prior knowledge may indicate that active two-factor interactions only occur within groups of factors. Designs of variable resolution (Lin (2012)) were intr...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:46 PM

13 views | 0 replies
Fast Flexible Space-Filling Designs for Nonrectangular Regions

Authors Ryan Lekivetz (1) and Bradley Jones (1,2) Affiliations (1) JMP Statistical Discovery LLC (2) Universiteit Antwerpen, Antwerpen, Belgium Journal Quality and Reliability Engineering International Date Published 2015 Abstract Space‐filling designs allow for exploration of responses with many different settings for each input factor. While much research has been done using rectangular design s...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:43 PM

14 views | 0 replies
Combinatorial Testing of Engineered Systems With Hard-to-Change Factors

Authors William Fisher (1), Ryan Lekivetz (1), and Joseph Morgan (1). Affiliations (1) JMP Statistical Discovery LLC Journal Quality and Reliability Engineering International Date Published 2025 Abstract The testing of engineered systems is a complex task, which often requires a test engineer to construct a suite of test cases in order to investigate system behavior, with the primary purpose of th...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:39 PM

21 views | 0 replies
On Algorithms for Obtaining Orthogonal and Near-Orthogonal Arrays for Main-Effects Screening

Authors R. Lekivetz (1), R. Sitter (2), D. Bingham (2), M. S. Hamada (3), L. M. Moore (3) & J. R. Wendelberger (3) Affiliations (1) JMP Statistical Discovery LLC (2) Simon Fraser University, Burnaby, BC (3) Los Alamos National Laboratory, Los Alamos, NM Journal Journal of quality technology Date Published 2015 Abstract Screening designs for identifying active main effects among a group of factors ...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:38 PM

11 views | 0 replies
Design Fractals: A New Visual Aid for Assessing Design Coverage and Augmentation

Authors Caleb King (1), Ryan Lekivetz (1), William Fisher (1), and Joseph Morgan (1). Affiliations (1) JMP Statistical Discovery LLC Journal Quality and Reliability Engineering International Date Published 2026 Abstract Experimental designs are powerful combinatorial objects that allow users to investigate challenging problems with limited resources. Assessing an experimental design often involves...
Ryan_Lekivetz

Publications |
Jan 31, 2026 5:35 PM

20 views | 0 replies
Learning User Interface Preferences via Contextual Discrete Choice Experimentation

Authors Fisher, William (1), Jacob Rhyne (2), Mark Bailey (2), Joseph Morgan (2), and Ryan Lekivetz (2). Affiliations (1) Clemson University, Clemson, SC (2) JMP Statistical Discovery LLC Journal Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization Date Published 2025 Abstract Designing effective user interfaces (UIs) is a complex decision-making process that oft...
Ryan_Lekivetz

Publications |
Jan 29, 2026 3:19 PM

13 views | 0 replies