As we develop analytical tools in JMP, we inevitably must make decisions about how to prioritize:   Should we make the product more powerful by adding more muscle? Should we make it sexier, more exciting? Should we focus on pain relief, making it less frustrating, less burdensome? In the language of long A-words, should we go for anabolic, aphrodisiac or analgesic?   John Sall thinks the answer is analgesic. Pain relief should be the central motivating force for development. Of course, the three aren’t mutually exclusive. Adding an exciting power feature could also relieve pain. But pain relief is central, because pain is the condition that can really freeze us, demotivate us, make us stop at a less-than-full perspective of what our data can tell us.     ... View more

  Our Earth Day episode of JMP on Air kicks off with SAS Co-Founder and JMP Chief Architect John Sall. In his talk, Supply and Demand with Carbon Pricing, John combines his expertise in JMP and a passion for global conservation and environmental issues to compare a carbon tax to cap-and-trade-- using JMP for scripting and creating graphs. After providing some background on the issues around carbon pricing, John shows us how supply and demand slopes are affected based on the tax, as well as the effects low and boom demands would have on price.   Formulae are used to create transformed or derived variables using built-in functions, constants and/or existing variables. ... View more

  John visually explores mechanical metaphors for statistical mechanisms, encouraging us to think visually and intuitively about statistical fits in terms of tensions and compressions     Automatically Generated Transcript:   So you won't need the your color vision in order to understand the next topic, but it is about seeing things. And so let me share my screen. And so my screen should be coming through now. So I'm going to talk about visualization. But it's in a different sense than we usually mean it. It's not about what you see with your eyes, but more like what you see when you close your eyes.   And you use your inner eyes your insight, and how you perceive and imagine feeling the forces and energies involved in the situation. But we're still talking about statistics statistical fitting, which is a very abstract concept. So visualizing statistical fits means converting the concept Statistics into mechanical metaphors. And we refer to those metaphors. When we design the graphics. And with these metaphors, we can understand things better. And we can get intuition of how fitting really works.   So statistical fitting is really an optimization problem, we talked about maximum likelihood or minimizing the least squares of error. And so optimization problems, if you study it is mainly about balancing forces. So you reach an optimum we're moving from that position to another position takes energy.   And at the optimum, the net forces are zero, the net derivative is zero, the stored energy is at a minimum. So as we balance the forces and reach this minimum energy solution, we can think about it in terms of simple machines. And here, we're going to use springs and gas pressure cylinders, to visualize the forces and energies.   So we have two laws to consider.   First, the law of springs due to Robert Hooke who lived in the 1600s and was the curator of experiments for the Royal Society in the Age of Enlightenment. And the other Pioneer was Robert Boyle, who formulated the law gases, also in the same age of enlightenment. And he was considered the father of modern chemistry. So we have Hookes law and Boyle's law, which we're going to talk about in a few minutes. Now, I'm not the first to invoke Hookes law and springs, and it's been done before in fact, there's a book about it by rW Fairbrother on visualizing statistical models and concepts, and he refers to actual nukem more than 100 years ago. Who did the same thing, but the gas pressure ideas, perhaps something a little bit newer, but it's one of my favorites. So with continuous responses, we're going to talk about springs. And with categorical responses, we're going to talk about probabilities and gas pressure.   So Hookes law and, and Boyle's law. So let me do a little demo, we're going to open up a big class. And we have here a plot of height by weight. Now let's first suppose we're going to estimate the mean height.   Well, imagine connected each point is a spring and the other end of that spring is connected to a horizontal line. And so we can imagine we have to stretch that spring from its distance of zero to whatever the residual is going to be to connect it to that point, and we can pull that line and put it anywhere we want. And each position that we pull it to is going to store a different amount of energy. And the forces are going to be pulling one way or the other. And here you see, most of the forces of all these springs in red are pulling upward. And there's only a few pulling downward. And so we are going to release sets spraying, I'll press the Go button, until it finally finally settles down and all the forces are equalized between one side and the other. It's during the minimum amount in those springs.   So we can also allow the line of fit to not just be horizontal, and instead of fitting the mean it's going to fit the regression line. And so if I press the Go button and allowed to have a slope, then it's going to feel the forces those springs running vertical from each point to the line of fit and A place that balances all the forces both vertically and rotationally is going to be the least squares regression line, as we'll see in a few minutes. And so that's Hookes law and operation for continuous items.   So how does the spring really behave? And here we see hex Hookes law. So as you stretch a spring from its zero initial resting state to some distance, the force increases linearly with distance. That's Hookes law. And so the tension on the spring or force is the displacement x minus mew. And that displacement is then scaled by the spring constant, how strong the spring is. And we're going to call that one over sigma squared. And so how much energy does it take to stretch the spring from its zero position to some distance x.   From the resting point μ? Well, the energy is just the integral of that force integrated over the distance. And so it starts with a small amount of force and that force is increasing. And if you integrate this straight line, you get the equation for energy, which is one half, that's the triangle area of x minus mu squared times to spring constant, one over sigma squared. So that's Hookes. Law, the behavior of springs. And so notice the squared term. That's the important part.   So let's go to the next. visualization, and imagine what we did with the animation before fitting a mean. So connecting each point to The line of fit, we need to stretch the springs in order to reach that line of fit. And that line of fit, we consider to be movable. And as he moves the line up and down, that tensions on one side or the other, will vary until we reach a balance point. And at that balance point where the forces on one side and the other are equal, that's the position of minimum energy among the springs. And that least squares position is the balancing position for springs and also is the least square solution that we have for fitting a mean. So let's test a hypothesis with that. So here on the left, we have everything perfectly balanced, and we record the energy in the springs. And then we say, Well, I have a hypothesized mean it's a little less than the sample mean.   And so I'm gonna just move on That line of fit down and have to increase the energy I put in the springs in order to make that happen. And now it's not balanced anymore, I've had to add energy to the system, that energy we have to add can be expressed in terms of the sum of squares, the same sum of squares for the energy equation. And that is the sum of squares due to that hypothesis. And so the more I have to stretch and store energy in the springs, the less likely that hypothesized mean is going to be the true mean. Okay, so, visualizing hypothesis test using this spring metaphor can be useful.   Another situation we have concerns. One way analysis of variance that is testing that two or more means are equal. So we can visualize each mean estimated separately. And here we have four observations. Three different groups, it's connected by a spring, the observation connected by spring to have a line of fit, which will be the sample mean for each group. And, and it's in a state of balance or minimum energy stored in the springs, by fitting separate means. But now we hypothesized that really the means for all these groups are the same. And so we do that by aligning all these lines of fit to be one line of fit and finding the balance position for fitting just a single mean. And the amount of energy we have to add to the springs in order to force that constraint will then be the sum of squares to the to that hypothesis. And so testing in the one way layout or the two sample means test is exactly analogous to adding energy to set a springs in order to force that constraint   So what about concepts like design of experiments where we want the most powerful design to make our testimony most powerful. So, consider an unbalanced design where we have six observations in one group and two observations in the other group. Well, where we have six observations, there are a lot of springs, and it's forcing the line of fit to be very stiff. And it's going to be very difficult to move that line of fit away from, from the true mean. But where we have only two springs is can be very easy. And so it's going to be easy to move this line of fit up to the others. And we're not going to have a very powerful test of hypotheses, the sum of squares to to have hypotheses is not going to be as great as if we have a balanced design and with a balanced design, where we have an equal number of observations, then we have a more powerful test. And so that's You know unfolds into the theory of design of experiments for what the most powerful design is.   Now what about if we have different variances in each of the groups. So on the left, let's suppose that we have a greater error variance. And remember, the spring constant is one over sigma squared. So that means we have very weak spring constant, then that mean is not held very tightly, and we can move it without using very much energy. But if we have a much smaller error variance, meaning a much larger spring constant whenever sigma squared, then it's going to take a lot more energy to move the springs. And so the idea of reducing residual variance to make test hypothesis more sensitive means that we increase the strength of the springs, decreased sigma squared and if we have Two groups, for example, with different variances, the allocation of samples between the two groups should be related to the spring constant or the error variance between the groups.   Okay, so lots of metaphors work through from springs into statistical concepts. Same as true sample size. If we have a smaller sample with only a few observations, the means are not held very tightly in sec to pull them together. But if we have a lot more observations where the line of fit was held by a lot more springs connected to observations, then it's going to be a lot more difficult. And so as we increase sample size, we increase the power of the test greater sigma squares in order to test that hypothesis. Okay, so we've covered fitting means across classifications what About regression lines. And as we saw at the beginning, this time, we're balancing in two different directions up and down, and the slope or angle of the line. And so the line is where the forces governing both those is at a minimum energy solution.   If we want to test the hypothesis that x doesn't affect y, then we're constraining the line of fit to be horizontal. And if we do that, and balance out that situation, the energy stored in the springs compared with the unconstrained line is the test for the hypothesis that the slope is zero. So given that we, that leads to concepts like leverage, supposing our x's are distributed fairly closely together, in our sample, then we are fitting those springs new They're the middle of line. And we can pretty much pull that lever of ally in one way or the other. And it's gonna be pretty easy because the springs are not constraining the slope very tightly, or maybe constraining it up and down, but not to vary the slope.   But if we put our observations at the end of the lines and separate them out, then we have a lot more leverage. And therefore, our test hypothesis test will be more powerful to test that, and that's a basic principle of DLP to basically separate your, your values widely, as widely as practical in order to make a more powerful test.   So that's the concept of springs and testing hypotheses. But now we have categorical responses as well. So how do we do that? Well, we imagine things sort of like tire pumps. Remember With a tire pump, it's very relaxed at the top. And so if we push that tire pump down to compress the air in it, it gets, we feel more and more force as that tire pump then gets compressed down to towards a zero distance between the end of the piston that tire pump in the bottom of the cylinder.   And so in the same way, as we use springs, we can imagine tire pumps or some form of compression gas compression. And, we divide up our sample into a space where we have a partition between each space, which corresponds to a line of fit. And what we're doing is feeling the forces of gas pressure between all these partitions and bouncing it so the forces acting on one side and the other and on one side may include forces from many categories, many response levels, pushing one way versus the other. And as those balance out, then we're going to get the minimum gas energy in the system.   So if I pull this down and make them unrelated to the the ratio of the number of samples in each each level, then I have unequal energies. And this, this is also analogous from big class where I am estimating the age proportions, the probability that you're in a given age group in this sample of data. And so in that age group 12, I have eight observations and that corresponds to the eight observations and the lowest component. And with age 13, I have seven and that corresponds to the seven and the next, and so on. In the last two for 16 and 17, I only have three observations, the only three gas molecules or tire pumps in that category.   And now if I want to let it go in a cooler Bay, so that the line of fit is equalized across all these things, and I see that the distance between each one is proportional to the rate that those things occur in the population. And so the, the rates that I see in my population are the minimum energy or balancing solution to the equations for the gas pressure across these these partitions, where they all add up to one.   So So now, let's see how it works in theory, and we here we have Boyle's law. So with Boyle's law, we start out we'll start out normalized deposition of one where the pressure inside the cylinder and is basically equivalent to the atmospheric pressure. And then as we pressurize the cylinder, then we're, we feel greater and greater force as we're pressurizing the cylinder to move farther down and compress the gas. And so it turns out that the pressure is one over P, the position of the cylinder. And of course, it can get infinite if, in an ideal situation, the pressure gets infinite as we try to pressure towards zero. So now, we feel this pressure inversely related to the P or the probability rate we're attributing to the statistical solution. And so we need an equation for energy stored in the system. So the energy is just the energy From one down to whatever position p we have, and it turns out the energy one is normalized, the log of one is zero.   And so as we go from there, down to P, the integration of that there's separate goals, one over P is minus the log of p. Okay? So the energy restore and the gas pressure is minus log of the probability. So this turns out to be the negative log likelihood of a statistical system, doing pretty much the same thing, but in terms of log likelihood, or negative log likelihood, rather than gas pressure.   So let's see how we can visualize situation. And here's the situation where we have the car pole example where, let's say we have 13 A sample of 13. And six people chose American brands, two people chose the European brands of cars. And five people chose Asian brands of cars.   And so if we load up pressure cylinders and D to these compartments, where we can move the line between the compartments one way or the other, and then each cylinder is identical. And what we want to do is balance the net pressure in each partition, so that it balances out to the minimum energy stored in all these gas cylinders. And it turns out, you do that by just estimating probabilities or the log likely, the maximum likelihood solution to storing the those those energies turns out to be the rates in the population, or five out of 13, two out of 13 and six out of 13 And that minimizes the total energy in those gas cylinders. Or you can also represent those gas cylinders as just units of gas pushing in these compartments. And think about the kinetic theory of gases doing pretty much the same thing as the pressure cylinders do. So, how do you do across groups? Well, let's say that you divide your carpool examples.   So you have a different different groups corresponding to young young people, middle aged people and older people, and they have different preferences and they're gonna have different response counts for each of the car preferences for American, European and Asian brands. And so, if we then solve the same system by minimizing the energy And balancing the the pressure of these partitions, then it's going to estimate different rates in different different ones of these groups. And so now, our next thing is to think of how I'm going to test the hypothesis for this. And to test the hypothesis that the rates are the same across these groups means moving these partitions to be the same across groups, balancing it with that constraint to the minimum energy solution subject to that constraint. And the amount of energy I've had to add to the system, in order to do that will be the negative log likelihood.   For to calculate a chi square test, same idea is sum of squares, but it's using gas pressure instead of Pat and the same thing can be done with a continuous responses where I could think of of Tire pump or gas pressure cylinder at each position of the axes. And they have to fit in these partitions where the line of partitions are constrained to be of a logistic or profit curve. And then I balance all those positions. I see here that this tire pump has had to be compressed a lot. So this is like an outlier in a categorical sense. It's a pretty large x corresponding to what x would be expected, and so it's an unlikely value, but it corresponds to fitting the situation. So now, I think about these gas pressure cylinders, in terms of strengths of relationships, so, I have a near perfect fit and the logistic regression. I have complete separation between you know some value of x separates That one response category from another response category. If I have a strong relationship, then instead of complete separation, there's a partial separation. But there's a lot more going on in in one versus another. And if I have a weak relationship, then these gas pressure cylinders are fairly equally distributed across that relationship. And so the line of fit is close to a straight line. And I have a weak relationship in the energy in order to constrain it to a perfectly straight line will be very small compared to what I'd have to add to a system where we have near perfect fit.   So all this goes into designing the graphs so that when we see a graph, we think about forces energy. So let's think about the continuous case. And we think about fitting a mean. And we think about this like a vertical scatterplot. And we think of springs attached to that. And this is the balance point that balances all those springs for a univariate continuous response. For a one way situation, we think about fitting a different mean to each of these and springs connected each point. And then if I constrain it, and fit it to the straight line, that how much energy I would have to add the system goes to the sum of squares to that hypothesis, the same similar with continuous I think about springs detached from each point to the line of fit, and how strong I would have to or how much energy I would have to add in order to stretch that line of fit to be horizontal. And similarly, in categorical I think about pressure gas pressure when I do these divided bar charts now Divided bar chart is not very good for estimating rates compared to just a single bars. But a divided bar chart is good for visualizing the gas pressure illusion of the solution being to equalize the pressure in each compartment of these chambers.   And then when I do several groups instead of just one, then I get to the mosaic plot where these divided bar charts are side by side and proportional to that. And I can think of how much energy I have to add the system to make these compartments align across each of that and that's the likelihood ratio test for that hypotheses. This is one reason why I like Likelihood Ratio Test better than Pearson tests, even though both are similarly powerful and most categorical situations because I can Think of the gas pressure when I do this. And the same thing as when I think of compartments in a continuous logistic regression. In this case, I think of about maybe these points are our air molecules bouncing back and forth, vertically. And as with only a few points out here, the line of fit is dominated more by, by the other things. And I can think of logistic regression that way. So that's the way I think of visualization, seeing the forces and energies and graphs, which you can't see directly. It's not something that's visual. It's something you sense as you use these mechanical metaphors, Hookes law and Boyle's law, in order to drive the statistical concepts.   ... View more

In magical tales like the Harry Potter series, wizards can just   point at something with their wand, speak an incantation, and   magic happens. We do something very similar when we point   the cursor at something on our screens, and without even   an incantation or a click, magic happens. That is our world of   tooltips or hover windows .   Identify points Graphs of points in JMP are incredibly valuable in seeing   the distribution of points over two dimensions. But when   you see a point that is far apart from the rest of the data  –   an outlier  –  you want to identify that point. By hovering the   cursor over the point, a hover window appears showing   which point that is, plus you can click to select the point in   the table to investigate further.   From the time JMP was first released, hovering your cursor   over a point in a graph has brought up a small text window   identifying a point. In the first version of JMP, it just showed   the row number. But we ’ ve continued to add information   and functionality with subsequent versions: the row number   and any column values given the  “ Label ”  attribute; the   values corresponding to the coordinates; the ability to  “ pin ”   the hover label to the graph persistently; label columns that   allow images in the hover label window; an option to use   the images as markers. But with  JMP 15  there is much more.   What’s new JMP 15 introduces far more powerful hover windows  –   graphlets. They are built in to  a number of  platforms   where points represent many values and can be used in   a customization setting for almost any graph, providing   drill-down features.   Process Screening In  Process Screening , you can see the health of many   processes in one Process Performance Plot. In Figure 1,   one process is healthy, the others are incapable, and   Thickness 4 is unstable as well. To find out more about   the instability, hover over that point and a hover window   appears, showing a small graph of the process. Hovering   is a lot easier than selecting the point and requesting a   quick graph or control chart of that process.     Figure 1: Process Screening Functional Data Explorer The  Functional Data Explorer  characterizes many   trajectories using functional principal components.   Figure 2 is a score plot of the temperature functions of   16 US weather stations. To determine what ’ s going on   with the score space, points in the four areas have their   hover windows pinned. The first component on the X   axis is basically how extreme the winter is, while the   second component, with less than 5% of the variation,   involves the shape of the seasons.     Figure 2: Functional Data Explorer Multivariate Control Chart In the new  Model Driven Multivariate Control Chart   (Figure 3), the fact that a point is far out doesn ’ t explain   which process is involved in it being extreme. By   hovering over the point, the individual differences   are shown  –  in this case, Process 6 is implicated. Figure 3: Model Driven Multivariate Control Chart Custom graphlets Figure 4 shows the mean violent crime rate for each state   over years, but it doesn ’ t show the trajectory. To see that, I   made a separate Graph Builder showing the violent rate   by year. Next I copied script to clipboard, then went to   the original graph and right-clicked on Paste  Graphlet .   Now when I hover over a point, it makes the graph of   that state by year, in this case detailing Virginia. Figure 4: Custom graphlets Further customization Once you make a graphlet, you can customize its   features in a Hover Label Editor. The Graphlet panel   seen in Figure 5 shows the script that is run on the   subset represented in the point.  Gridlet  provides   control over which elements are shown in the hover   window.  Textlet  allows you to customize the text,   including the markups to stylize the text.     Figure 5: Graphlet Hover Label Editor Conclusion Good software can make things so easy that it seems like all   you  have to  do is point a wand (or cursor) at something,   and magic happens.   If you want to see  JMP 15  and JMP Pro 15 in action, the JMP R&D leadership team and I gave this  presentation  at Discovery Summit Tucson 2019. ... View more

We  created JMP 30 years ago  because we saw a clear need. Scientists and engineers had  data – but  not  the statistics background or coding skills necessary to  explore and analyze that data.     They  wanted something  interactive and  easy to learn on a limited budget.  While they  were in t echnical disciplines,  they  didn’t want to be technical  from a  programming point of view.   The statistical initiatives of the time included Tukey’s Exploratory Data Analysis and the emphasis on graphics, punctuated notably by the  Anscombe Quartet  published in 1973. It became important to look at your data in a graph.   When JMP was first released in October 1989,  its  original goals were:   Find the best ways to exploit the emergence of the point-and-click  GUI  of the Macintosh.   Find the best statistical graphics to go with each statistical method.   Provide a new product that was relatively inexpensive and easy to use, and would serve as an entry-level step for SAS and for those who didn’t need a product as big as SAS.   As a result, JMP offered linked selection, context clicking everywhere and a graph for every statistical test.   JMP has grown from a limited product with a limited audience to a mature product with a wide audience.   It’s amazing to see  how analytics helps our customers solve problems and innovate.     This  month ,  we celebrate the origins of JMP. We also celebrate the future.  You can learn more about our new versions  –   JMP 15  and  JMP Pro 15 ,  as well as   our new product,  JMP Live . JMP  continues to be a desktop product, but now it can publish live reports, accessible to everyone in the private community. With JMP Live, publishing, sharing and community become new strengths.   That’s why we’ve been around for 30 years :  We like empowering you to do more with your data.   Continue to read the JMP Blog this month for celebratory posts from longtime JMP users and thought leader perspectives on important topics in analytics.   ... View more

With multivariate methods, you have to do things differently when you have wide (many thousands of columns) data. The Achilles heel for the traditional approaches for wide data is that they start with a covariance matrix, which grows with the square of the number of columns. Genomics research, for example, tends to have wide data: In one genomics study, there are 10,787 gene expression columns, though there are only 230 rows. That covariance matrix has more than 58 million elements and takes 465MB of memory. Another study we work with has 49,041 genes on 249 individuals, which results in a covariance matrix that uses 9.6GB of memory. Yet another study has 1,824,184 gene columns, yielding a covariance matrix of 13.3TB. Computers just don’t have that much memory – in fact, they don’t have that much disk space. Even if you have the memory for these wide problems, it would take a very long time to calculate the covariance matrix, on the order of n*p 2 , where n is the number of rows, and p is the number of columns. Furthermore, it gets even worse when you have to invert the covariance matrix, which would cost on the order of p 3 . In addition, the covariance matrix is very singular, since there are far fewer rows than columns in the data. So you would think that multivariate methods are impractical or impossible for these situations. Fortunately, there is another way, one that makes wide problems not only practical, but reasonably fast while yielding the same answers that you would get with a huge covariance matrix. The number of effective dimensions in the multivariate space is on the order of the number of rows, since that is far smaller than the number of columns. All the variation in that wide matrix can be represented by the principal components, those associated with positive variance, i.e., non-zero eigenvalues. There is another way to get the principal components using the singular value decomposition of the (standardized) data. Now that you are in principal component space, the dimension of these principal components is small; in fact, all the calculations simplify because the covariance matrix of principal components is diagonal, which is trivial to invert. For example, let’s look at that genomics data set with 1,824,184 columns across 249 rows. I was able to do the principal components of that data using JMP 12 on my 6-core Mac Pro in only 94 minutes. Doing it the other way would clearly have been impossible. With discriminant analysis, the main task is to calculate the distance from each point to the (multivariate) mean of each group. However, that distance is not the simple Euclidean distance, but a multivariate form of the distance, one called the Mahalanobis distance, which uses the inverse covariance matrix. But think about the standardized data – the principal components are the same data, but rotated through various angles. The difference is that the original data may have 10,000 columns, but the principal components only have a few hundred, in the wide data case. So if we want to measure distances between points or between a point and a multivariate mean (centroid), then we can do it in far fewer dimensions. Not only that, but the principal coordinates are uncorrelated, which means that the matrix in the middle of the Mahalanobis distance calculations is diagonal, so the distance is just a sum of squares of individual coordinate distances. We take the principal components of the mean-centered data by group. Then we add back the mean to this new reduced-dimension data. The transformed mean is just the principal components transformation of the original group means. Example There is a set of breast cancer genome expression data that is a benchmark used by the Microarray Quality Consortium with 230 rows and 10,787 gene expressions. A discriminant analysis using the wide linear method takes only 5.6 seconds (on Mac Pro) and produces a cross-validation misclassification rate of only 13.7% on the hardest-to-classify category, ER_status. This misclassification rate is close to the best achieved for this problem, given that the other methods are expensive. Note: This is part of a Big Statistics series of blog posts by John Sall. Read all of his Big Statistics posts. ... View more

In an earlier blog post, we looked at cleaning up dirty data in categories. This time, we look at cleaning dirty data in the form of outliers for continuous columns.   In industry, it’s not unusual to have most of your values in a narrow range (for example between .1 and .7) with a few values well outside that range (such as 99, which is the missing value code). Some of the most common examples of outliers include: • Missing values codes, often a string of 9s. • Obviously miscoded, manually entered values, especially when the decimal point is misplaced. • Instrument failures, which result in an instrument failure code inserted where the data should be. • Unusual events, such as an electrical short causing a current surge or a voltage dropout. • Long-tailed distributions, such as time-to-failure distributions, where large values may be rare, but real. • Contamination outliers, in which a few items have crept into a collection of items that was presumed to have been clean.   If we had only a few columns to explore and clean up, it would be reasonable to do it by hand. However, with dozens, or thousands of columns, you need some help. The “Explore Outliers” facility in Cols->Modeling Utilities is the new tool to use.   The most straightforward approach is to get some tail quantiles and then measure some scale of the interquantile range beyond the tail quantiles. The ANSUR Anthopometry data provides a good illustration; it measures 131 body measurements across 3,982 individuals. Before analyzing this data, you need to first check for and be aware of outliers.   For each selected variable, Explore Outliers first calculates the 10% and 90% quantiles to determine a good estimate for the range of most of the data. If there is a huge number of ties, meaning that the 10% and 90% quantiles are the same, the utility reaches further into the tail to obtain a value. It then multiplies the interquantile range by 3 (the default ‘Q’ scaling factor) and looks for any values that are further out from the 10% and 90% quantiles than three times the interquantile range. This is a pretty extreme default outlier selection criterion, but you are probably most interested in the most remote outliers, at least at first. You can adjust the quantiles and multiplier later to be more sensitive.   The default scan of the ANSUR data revealed two columns with a value of -999, which is more than three interquantile ranges past the 10% and 90% tails. In this case, it is obvious that they are missing-value codes, not real data.   Since the other 129 variables did not show outliers, we can more easily focus on the columns that did by checking “Show only columns with outliers”. Selecting these columns in the display will select the columns in the data table.   Next, we need to determine which action to take to address these outliers. There are six action buttons. You can select the rows, and use the Rows command “Next Selected” to advance the data table view to each selected row. You can color the rows to emphasize that the row had an outlier. You can color the cells to make them stand out as you browse the data table, shown below. You can also exclude the rows to hold them back from an analysis.     In this case, since the outliers are really representing missing data, the two remaining actions present the obvious choices: Either click “Add to missing value codes”, which installs a “missing value code” column property for that column, or click “Change to missing”, which changes it to a hardware (NAN) missing value that JMP always recognizes as missing. You might want to use the former button if there are multiple missing value codes that have distinct meanings.     We also check for high-nines, since these are often used as missing value codes, but all the high-nines are not far from the upper tail of the data, and are thus not likely to represent missing values.   Sometimes outliers are outlying in a multivariate sense, meaning that they may be nicely in the quantile range of each column, but the data is in clusters, and some points are far away from other points. Using Fisher’s Iris data as an example, we see that there are really three clusters, representing the three species. Because the species are widely separated, there is plenty of near-empty space where an outlier can be distant from all the other points, even though not a univariate outlier. We have the k-nearest-neighbor facility to find these outliers. For each point, it finds the distance of that point to the nearest point, the second-nearest point, the third-nearest point, and so on. Here, we identify a point that is far away from the nearest point.     If we look at this point in a Scatterplot Matrix, we see that it is not unusual in any of the single coordinates, but it is far away from other points. It is far away in Sepal width when considering its location with respect to Petal length and Petal width. We happen to know the identities of these clusters, and the unusual feature is that it has a very low Sepal width compared to others in its species.     To continue, we might have multiple outliers, i.e., those points that are not far from their nearest neighbors but are still far from the data. Here is the distance of each point from its third near-neighbor.     If you look at the Scatterplot Matrix of the two blue k=3 outliers, you see that they are near each other, but separate from other points, focusing on the Sepal width by Sepal length scatterplot.     Thus, multivariate outliers can be explored and considered to discern what is happening. Outliers in the k-nearest sense can happen in contamination situations, for example, if an iris from a totally separate species was mixed in and misidentified in the Fisher Iris data.   What to do about outliers depends upon the situation, though some actions are clear. If you have missing value codes, you should add a missing value code property so that the analysis platforms will treat it as missing. If you have a misplaced decimal point, you should correct it if it is a clear case, or change it to a missing value if it is not clear.   Sometimes finding outliers can lead to important discoveries. They shouldn’t automatically be treated like dirt that doesn’t fit.   This topic was also addressed in a previous blog post.   Note: This is part of a Big Statistics series of blog posts by John Sall. Read all of his Big Statistics posts. ... View more

Many JMP users get their data from databases. A few releases ago, we introduced an interactive wizard import dialog to make it easier to import from text files. In a subsequent release, we created a feature that lets you import Web page tables into JMP data tables. In JMP 11, we introduced an interactive wizard to simplify importing data from Excel spreadsheets, covering such functionality as going across tabs and stacking groups of columns. But there was one area that we had yet to improve: making it easier to import data from databases – until now. So what are the problems to solve? Data from databases usually means an abundance of data from many different tables. To get at the data you want, you need to join and filter, something that requires too much knowledge about the data and too much expertise with the SQL language. The amazing thing about how a database stores data is how regularized it is, so that the same data is not duplicated across multiple tables. There are three huge advantages to this: 1. Since data is not duplicated, there is less wasted storage space. 2. Updating a field is a matter of updating one place instead of many. 3. Data is naturally more consistent. Databases are organized for efficient storage and efficient transactions, but not usually for efficient analysis access. Often the data you need to analyze is scattered across multiple tables. You have to join them to obtain what you need, which can make for significant work. You could just import all the data from each table, and then join them inside JMP, but there are two reasons why this isn’t the best method. First, you usually do not need all the data from each table, so you waste time and space when you import more than you need. Second, it is a burden to specify how to join the tables together. Alternatively, you could use SQL to specify a join and then select the rows and columns to import. While this makes for more efficient use of the hardware, it isn’t always easy since you have to learn SQL, become familiar with the details of each table, and write and debug the SQL code to send to the database. But JMP can help you do all of these things with ease. Databases are organized to help with access. Each table usually has a primary key to uniquely identify each row of the table. The table also has “foreign keys,” which are columns made to match primary keys in other tables — this relates that table to other tables. The typical join is then an “outer join,” matching the foreign key in one table to the primary key in another table. This is a one-to-many (outer) join, since a foreign key may not be unique but a primary key is unique to its table. In JMP, the new Query Builder dialog is organized to make this process easy. First, you specify the primary table, the one whose rows make up the unit of analysis. This primary table will usually have columns designated as foreign keys. Simply specify some secondary tables, and suddenly, the join becomes obvious and automatic. Next, you need to select the variables and rows to keep. Selecting the variables is easy in Query Builder — select them from a list that shows the table from which each variable comes, along with the data type. Selecting rows involves specifying filters, but this is easy, too. Query Builder retrieves all the categories for categorical variables and the ranges for continuous variables, and then presents controls so you can pick just the categories and ranges you want. But it gets even better, since Query Builder goes several steps further. You can preview any table or prospective join output right in the dialog window. You can also examine the resulting SQL code, plus you can save the query to a file so you can use it again to get the data. When you save the query, you can specify that some of the filtering conditions be set to prompt you, so that the next time you run the query, you can change the categories or ranges you specify to filter the data. You can read about Query Builder in this blog post from Eric Hill, the lead developer of the facility. We live in the age of big data, and most of this data is stored in databases. Making it easy to get this data into an analysis helps turn “big data” into “big statistics.” Note: This is part of a Big Statistics series of blog posts by John Sall. Read all of his Big Statistics posts. ... View more

Data entered manually is usually not clean and consistent. Even when data is entered by multiple-choice fields rather than by text-entry fields, it might need additional work when it is combined with data that may not use the same categories across sources. Sometimes the same categories are spelled differently, abbreviated differently, or capitalized differently, or just miskeyed, resulting in many more categories and an invalid analysis when they are used in a model. For example, consider the customer field in the “Cylinder Bands” data that can be downloaded from the UCI Machine Learning laboratory (originally from Bob Evans, RR Donnelly & Sons). When you import this data, you will find that there are 83 unique values of "Customer". There are a lot fewer actual customers than that — it is just that there are multiple codings of the same customer. For example, the customer Abbey Press is coded in three ways: “ABBEY”, “ABBEYPRESS”, and “ABBYPRESS”. In the early versions of JMP, we used tricks to recode the values. For example, we might copy “ABBEYPRESS” into the clipboard, then use a histogram to select rows corresponding to “ABBEY”, select the column “Customer” and then paste — which would change each “ABBEY” into “ABBEYPRESS”. Once the Recode command became available, and we could just copy “ABBEYPRESS” and paste it into the other recode Values for “ABBEY” and “ABBYPRESS” in the Recode dialog. All this works if you don’t have too many categories, but what if you have hundreds of categories? It can be very laborious to perform all those copies and pastes. There are two very important ways that recoding is much better in the new Recode facility in JMP 12. First, you can do a whole group just by selecting all the recode values and right-clicking to pick the chosen category. Now they appear as a group, all together, even if the original values were separated. No more copy and paste; it’s simply select and choose, with the results forming a visible group. But the really amazing feature in the new Recode is the one that allows you to automatically find groups of similar values. In this example, the feature automatically formed 16 groups. You can then check to see if the resulting groups have all the categories you want, but not too many. How does it determine which values to combine? It looks at each pair of category labels and determines the edit distance between the two character strings. Also called the Levenshtein distance, it is the minimum number of edit operations to convert from one string to the other. When you run the “Group Similar Values” command, it brings up a dialog to choose which kind of changes to consider when calculating this distance and the criterion to use to call it a match. The default choices for this dialog usually work well. Some of the groups were done perfectly, such as for “ECKERD”: Because “ABBEY” is so different from “ABBEYPRESS”, it doesn’t satisfy the criterion to combine them, so you must select and right-click "Group" to do that. Similarly, it didn’t catch the abbreviation of “CAS” for “CASUAL” in these two categories: And it only got five of the six “HANOVERHOUSES”: While you will need to check the results, the automated feature gets most of them. After recoding, we have reduced the number of categories from 83 to 56. Closer inspection reveals that other recodings are needed; in fact, the entries after row 503 seem to have switched from uppercase to lowercase. By control-clicking to the menu item “Convert to lowercase”, we can fix them all. As you move into larger data tables that need cleaning up, it can be very helpful to have some automated features like “Group Similar Values”, and an improved user interface flow, such as selecting and grouping instead of copying and doing multiple pastes. (The new Recode features, introduced in an earlier blog post, were implemented by James Preiss.) Note: This is part of a Big Statistics series of blog posts by John Sall. Read all of his Big Statistics posts. ... View more

Demonstrate how the Hough transform accumulates hotspots at given angles and positions. You click in points in an x-y space and it accumulates the Hough transform for those points. Click points in a line, and it creates a hot spot. ... View more

The script demonstrates the fixed points of a curve that is the Hough transform for points you click on. Cllick and drag around an x-y space, and it shows the Hough function. As you drag in staight lines, you understand that the function is fixed at a certain angle and position. ... View more

The Heatmap Summary command creates a new table with columns with images of heatmaps with respect to two heatmap category variables and intensity variables. This is particularly useful for semiconductor wafer maps from die data identified by x and y coordinates. ... View more

Long lists of improvements go into each new version of our software, and usually there are one or two themes that characterize the release. JMP 12 launches this week, and the themes of this new version are flow and frontier. By flow, I mean workflow, the way we can smooth out the steps you need to take to get work done. I also would refer to the word flow as a state of consciousness, described by Mihaly Csikszentmihalyi as an uninterrupted, engaged state with deep enjoyment and creativity. By frontier, I mean that we become able to push the edges of what's possible and also push into new feature space. Workflow improvements include: Making database access involving multiple tables easy and fast, with Query Builder. Making cleaning up data much easier, with a Recode feature that automatically finds categories that should be combined, and an outlier facility that makes it easy to locate and deal with outliers. Making it easy to find what effects are important in fitted models and to easily remove the effects that are not important. Making it easy to publish results as PowerPoint or HTML. These workflow improvements can save you a lot of steps and make your path much easier, so that you spend most of your time on analyzing the data, not in overcoming the obstacles. Enhancements that push the frontier include: Enabling data to have any expression type, such as images. Allowing multivariate platforms to handle very wide cases, where there are many thousands of variables. This is happening much more in modern situations, such as in gene expression data and multisensor data. Implementing Covering Arrays to construct test designs that cover high-degree interactions of features. Generalizing model fitting with more distributions, more selection and shrinkage features that fit faster. Enabling fast fitting of complex mixed models when they have a huge number of levels in random effects. Pushing current features to cover more situations: extending process capability to understand short-run behavior, extending correspondence analysis with multiple correspondence analysis, extending PLS with PLS-DA for categorical responses, extending space-filling designs with categorical factors and extending definitive screening designs with blocking factors. Pushing these frontiers can have a major impact in uncovering discoveries. With these improvements, we cover more ground, and provide smoother paths to the ground we already prepared. It all adds up to a more productive experience in a wider variety of application areas. When I think of how smooth the JMP workflow is, I remember a remarkable video Julian Parris made, called “Speed Stats.” Julian is from our academic team, and he wanted to show to students and faculty just how easy it could be to answer a long set of questions, live, in just 10 minutes with tools that put you in the groove. It is a remarkable illustration of flow. You may not quite achieve Julian’s smoothness, but once you get enough experience, you can come close. [iframe title="YouTube video player" width="560" height="315" src="http://www.youtube.com/embed/2kVm8rvatHs" frameborder="0" allowfullscreen]   P.S. You can see a demo of many of these enhancements in a recording of my speech from Discovery Summit Europe. ... View more

For you, today is Oct. 4. At JMP, we call it Sept. 34. We had been determined to release the first version of JMP by the end of the third quarter of 1989. But, as it turned out, we needed a few extra days to make our own deadline. So we “extended” the quarter. Today, JMP is 25 years old. This milestone led me to reflect on its startup, evolution, current state and future. Most of the design principles adopted at startup remain our design principles today, though we have added a few new ones. As JMP grew from a very basic package to one with wider capabilities, we managed that growth carefully so that the product stayed agile and the product surface didn’t become too complex. Along the way, important growth themes for JMP have included exploiting the user interface, addressing the specific needs of engineers, advancing the state of the art of fields such as experimental design, and addressing the new challenges of big data with new power in large memory and multiple cores. Central to all our work has been the graphical user interface (GUI), enabling interactive exploration, making discoveries and understanding what the data is saying. In the last 25 years, the field of statistical methods has transformed from a small specialty to a key enabler of both advancing the technological and quality frontiers and of making better business decisions, as based on informative data. Original Design Goals When JMP was first released in October 1989, the original goals of JMP were: Find the best ways to exploit the emergence of the GUI point-and-click interface of the Macintosh. Find the best statistical graphics to go with each statistical method. Provide a new product that was relatively inexpensive and easy to use, and would serve as an entry-level step for SAS and for those who didn’t need a product as big as SAS. The SAS Perspective In the late 1980s, SAS had just completed a huge project to port SAS to personal computers, which involved translating millions of lines of code from PL/I-G to C. We had to maintain complete compatibility with previous versions of SAS, which had a large installed base of SAS language programs. Even when we took tiny steps, such as trying to require quotes in title statements, we found that we had to backtrack to complete compatibility. So JMP became an experiment to go where SAS wasn’t playing. We wanted to explore the design space with a new product [Blue Ocean strategy]. JMP needed to be as different as possible from SAS and still serve a viable market. SAS had a programming language, so we made JMP not have a language. SAS stored data by rows, so JMP stored it by column. SAS handled big data, so JMP was limited to in-memory and originally 32,767 (i.e., 2 15-1 ) rows. SAS was a very wide, powerful product, so JMP became a much smaller, focused product. SAS aimed to sell to business, IT and applications development, so JMP aimed to sell to scientists and engineers. The Macintosh Perspective In 1984, the Mac appeared. By the late 1980s, Apple was starting to gain traction in the industry with the GUI, and Microsoft was investing big and embracing it with its Windows project. Everyone believed that the industry was transforming again. We had just gone through huge transitions: from mainframe MVS to mainframe CMS, from mainframe to minicomputer, from printer output to graphics output, from editing terminals to full-screen interactive terminals, and then from minicomputers to personal computers. Another upheaval with the GUI was emerging. Programming to the GUI was supposed to be different from programming for a language interface and static output. We couldn’t just port SAS to a GUI, though we could reuse the subroutine library from SAS. So what feature set would be big enough to serve the small market of Mac users at the time? We picked a set of typical analytical routines and put them in a very contextual Analyze menu. The Statistics Perspective The statistical initiatives of the time included Tukey’s Exploratory Data Analysis and the emphasis on graphics, punctuated notably by the Anscombe Quartet published in 1973. It became important to look at your data in a graph. The American Statistical Association started a Statistical Graphics section and started publishing the Journal of Computational and Graphical Statistics. This coincided with the availability first of bitmap graphics terminals and then of graphics-enabled personal computers, so graphics became easy as well as popular. I remember the excitement of reading about the Gabriel PCA Biplot, the scatterplot matrix, brushing, rotating, mosaic plots. Pioneers included John Tukey, John Hartigan, William S. Cleveland, Michael Friendly, Lee Wilkinson. The hunt was on to design a graph to go with each statistical method. This hunt led to such JMP innovations as the general hypothesis leverage plot, comparison circles, the 3D spinning biplot and many more. The Evolution of JMP So with fanfare at the SAS Users Group International (SUGI) meeting in spring of 1989, we unveiled JMP on the Macintosh. This was a very small product compared to the JMP of today, but it had a nice set of initial features all linked to interactive graphics. That was just the first of many steps. Early on, we found that engineers became an important customer segment for us. We added control charts, elementary design of experiments (DOE), and survival features. The Profilers became extremely valuable additions to the fitting platforms. This was also when we started hearing about the Six Sigma movement becoming so important. As Windows emerged as not only a competitor to the Mac but also as the dominant host platform, we ported JMP to Windows. Modern DOE came to JMP when Bradley Jones joined our team. He pioneered a long series of innovations in design of experiments, and continues to do so. We completely rewrote JMP in C++ and gave it a modern display interface and a scripting language (JSL) for the fourth version of JMP. We invested in data visualization, adding Graph Builder, dynamic Bubble Plots, Data Filters and more. We continued to improve the application development process, with a better editor, App Builder, the debugger and script profiler, JSL namespaces, and add-ins and the JMP File Exchange. We exploited interfaces to SAS. At first, this meant just providing file import and export, but later we created increasingly sophisticated direct interfaces. As our user base grew, we developed various specialty areas: reliability modeling, process quality and consumer research. We also continued our investment in design of experiments where we lead the industry. For many years, JMP was used mainly in the US and Japan. Now use of JMP has spread to Europe and China, where we expect to grow fast. We learned to adapt to big data, making JMP faster and more resilient. We added multithreading wherever it fit well. We increased our involvement in customer engagements through users groups, online communities, customer care and more. The Future JMP has grown from a limited product with a limited audience to a mature product with a wide audience. As the awareness of the value of analytics advances, JMP is well-poised to grow quickly in the next few years. JMP will continue to be a desktop product, as opposed to becoming a client-server or cloud product. But the desktop has never been more capable than it is now, and we expect the desktop to remain important for many more years. Happy birthday to JMP! JMP is 25 years old now, very healthy and hardy and still growing fast. We thank all of our users for helping us make JMP into what it is today. JMP 1.0 Crew - Young JMP 11.0 Crew - Grown Up ... View more

Create formula columns to code for missing values Informative Missing is a great feature in Fit Model in JMPPro, but to do the same without JMP Pro you need to make some formula columns. This add-in makes it easier to do. The idea is to be able to fit using columns that have missing values, and rather than imputing the missing value, you can using a coding system to make it predictive as missing. ... View more

JMP does not have a facility for fitting nonlinear mixed models, but this script can be helpful in invoking the SAS PROC NLMIXED to do the work, using the model formulas in from JMP. With degradation analysis you need to import back the estimates and plug them into a model , and invert the model to produce crossing times.  The documents include a journal, the script, and sample data sufficient to do the Device B degradation example from Meeker and Escobar. ... View more

Data with extreme outliers (frequently seen in semiconductor test data) don't lead to a good color gradient when used in Graph Builder heat maps. The extreme range results in a gradient that puts the majority of the data in a small color range. This add-in creates a "Color Gradient" column property that gives a reasonable color scaling to represent the middle 95% (or other specified) of the data. ... View more

For outliers. Clean Up Error Codes looks for strange, outlying values that are not part of the regular data. Values like these are frequently used to indicate error conditions in measurements. The add-in uses an interquantile range to identify these values, looking for values that are far away from the upper and lower ends of the range. ... View more

The Benjamini-Hochberg False Discovery Rate PValue adjustment is a very simple way to control for the rate of false positives among many tests (usually assumed to be independent or positively correlated). For lots of PValues showing in a window in similar report tables, it is fairly easy to use right-click Make Combined Data Table to get those PValues into a data table; then you can use this add-in to obtain the False Discovery Rate adjusted PValues. ... View more

This produces a new table showing all the value labels for the columns of the current table. Used for finding groups of columns with similar value labels to group together. ... View more

This converts a prediction formula into a new Treatment Difference formula. The formula column must involve a 2-level treatment effect that interacts with other effects. If it doesn't interact, the resulting uplift formula will create zeros. If the treatment effect has more than two levels, it will make the difference for the last versus the first level. This is useful for Uplift Modeling, where you want this formula to choose which potential customers to apply a treatment intervention to for marketing programs. i.e. for which covariate values is the different of the treatment on the response greatest? The same technique can be important in personalized medicine, where you want to choose groups that the treatment is effective with. ... View more

Some Categorical variables have many levels. This might be ok for id columns, but not when you analyze them and realize that they should have been continuous. This shows the columns that have many levels and allows you to either hide them or change them to continuous. ... View more

Selects Columns with fewer missing values. This can be important for fitting models in which if any of the columns is missing, the whole row is dropped. ... View more

Hides columns that have few levels across a classification. For example suppose that you are looking for variables that change in interesting ways across many years. But some of your survey questions are only asked 1 or 2 of those years, so they would not be interesting. So you want to Hide and Exclude those "weakly crossed" columns so that you can select the rest to examine. ... View more

When you import from SAS or SPSS, you have the option of using the names or the labels, but both are stored in column properties. If you later want to switch to use the short names (SASName or SPSSName) or long names (SASLabel or SPSSLabel), this addin will do it. ... View more

For long character category variables in a large file, you can save space by using a code for the category, and using a value label for the long label. You specify a character column, click Select, and it makes the new column, named after the selected column with " Code" added to it. It is typically used to save space or improve load or analysis time, which typically takes longer for character variables. 2016-06-13:  Updated Add-in to support multiple column selection. ... View more

This add-in is useful for when you want to share a file to troubleshoot problems, but you want to de-identify the data as much as possible automatically. For example JMP Tech Support may need your data to diagnose a problem, but you may be reluctant to send in the data unless you can anonymize it. It changes all the column names to X__1, X__2, etc. It changes all the nominal and ordinal columns to have new values It updates value labels in nominal and ordinal columns to update to new values It eliminates list checks It updates column names in scripts Security: You must still check your data to make sure you don't have any identifying characteristics. Continuous numeric data is unchanged -- this might include identifying information, such as social security numbers. Scripts are changed to use new column names, but still might contain other identifying information. Known issues: Scripts with keywords the same as old column names may be incorrectly substituted. The program can fooled if there are already names or values the same to the ones that are created. This is not performance-tuned to handle large problems. This does not delete most column properties, but it does delete "Notes" properties. ... View more

The desktop or laptop is now in decline, squeezed from one side by mobile platforms and from the other side by the cloud. As a developer of desktop software, I believe it is time to address the challenges to our viability. Is software for the desktop PC now the living dead, or zombieware? Myth number 1: "The desktop is dead – my tablet can do it all" It’s true. When most people use a PC, they use it for email, calendar, contacts, Web, social media and entertainment. All of these uses are great candidates to migrate to smart phones and tablets. Tablets are great for other things too, such as reading or playing games. The mobile computing market has exploded in the past 10 years. When I go on a trip, I often don't bring a laptop anymore because my tablet serves me just fine. Mobile computing has taken over. Demand for desktops is declining, replaced by mobile platforms. But does a tablet work for analytics? Yes, in several important ways. Much of analytics involves scorecard metrics on a dashboard display, and the tablet is perfect for that. Even interactive analytics delivered to a mobile device works great. SAS Visual Analytics provides that. JMP Graph Builder for iPad delivers that for local data. …And it’s false. But what if you want do deeper analytics, with models and large numbers of graphs? That doesn’t fit on a tablet. Mobile platforms are too small to hold all the data and software, and the display interface is small and not desktop-like. All of the interfaces are too simple – there is no right-click, no fine mouse-control. The file system is crude. The support environment is insufficient. Tablets are just not power tools for data analysis. While tablets have taken over for many of our personal computing needs, they have not taken over for analytics. Myth number 2: “The desktop is dead – everything is in the cloud now” It’s true. With the amazing emergence of high-speed Internet and cloud services, much that we used to do on individual computers now happens in the cloud. Services can be provided that are fast and very graphically interactive. Even through the Web, Ajax, Flash and HTML5 provide a rich interface to harness. Information systems that used to be installed with great pain and effort on a home system can now be spun up quickly in a cloud service. Salesforce.com revolutionized customer information systems. Amazon EC2 and other cloud services can provide a rich set of software and services. The cloud can also provide much deeper software implementations, making everything available without having to install anything. The cloud versions can be updated often with the latest fixes without any effort by the customer. The cloud has truly revolutionary potential. …And it’s false. You already have a desktop computer, and probably a pretty powerful one. All you need is good software that installs easily. There is no need for a cloud service, one that you will have to pay for through the usage meter, often more than serving it yourself. The cloud reduces the installation burden associated with big IT systems, but personal analytics software does not have a high installation burden. Desktop analytics software can provide a much richer and more interactive service, one that is free of usage charges. Myth number 3: “The desktop is dead – you can’t do big data on the desktop” It’s true. We live in an age of big data, when the data sets are measured in terabytes, and there are billions of rows. It is impractical to analyze data of this size on a desktop. You need a cluster of servers that can divide the work to be handled in parallel so that it takes only seconds, rather than hours or days to analyze. Analytics is also more efficient when it is done close to where the data is; so if it is analytics from a database, putting the analytics close to the data speeds things up. Analyzing big problems on a desktop may be too slow, and if the desktop depends on data being in memory, then you are limited to what fits in memory. …And it’s false. Most analytic data sets fit just fine on a laptop. They are usually far less than a terabyte, usually less than a gigabyte. We also live in the golden age of desktops, when you can get a very powerful machine that is still inexpensive. You can get a laptop with 16-32 GB of memory and 4-8 CPU cores for a couple thousand dollars. Each additional 16 GB of memory costs less than $100. Even big data can be done on a laptop if not too big. In my standard demo, I bring up a data set of all the airline flights over a 20-year period from the major US airlines. It is 123 million records. To load it takes 3-5 seconds from an SSD. To build a graph for it takes a few seconds for each drag-and-drop step. I can get a scatterplot of all the points in 10 seconds, and a regression with 65 parameters in 35 seconds. That is big data, and the PC is so capable that it is doing interactive analysis and graphics at this scale. There are opportunities to use the desktop to be much faster yet. The graphics processing unit of the PC is actually much more powerful than the main CPUs, though most software doesn’t use it. Take the lowest end of the newest Mac Pro. It has 4 cores and 2,560 GPU cores and is rated at 4 Teraflops. The high-end Mac has 12 GB of graphics memory and 4,096 stream processors capable of churning out calculations at 7 Teraflops. This is supercomputer performance in desktop machines that cost only $2,000-$4,000 dollars. If my problems change from a few gigabytes to a few terabytes, then I would be ready to give up on my laptop and switch to a server cluster. Most analytic problems are smaller than terabytes. Myth number 4: “The desktop is dead – you don’t need it for good graphics” It’s true. One of the best graphics presentations I have seen is Hans Rosling’s TED talk, and that was all done with Flash on the Web. SAS Visual Analytics proves that interactive graphics can be done easily through a Web page, and it is almost instantly interactive, even when you have billions of rows of data. …And it’s false. But real analyses are often not simple dashboards. They are large interactive documents with many graphs and many tables. With big data, scatterplots can have millions of points, and you want to be able to select points interactively. The Web services are just not up to the task. Furthermore, the Web-delivered graphics services are not fully mature. Flash is good, but it is slowly disappearing because it isn’t supported on Apple mobile devices. HTML5/JavaScript works, but it still has yet to mature into a good environment to support a rich complex graphical system. Dreams do come true We live in a world of computing where dreams come true. The mainframe bows to the minicomputer. The minicomputer bows to the personal computer. The personal computer bows to the tablet and smart phone. It seems as if these will soon bow to the smart watch or smart glasses. But at each step along the way, some applications find their best home – and other applications as well as new applications find the more convenient and smaller home better. Data analysis needs a great computing engine and a great large display. The desktop is probably going to remain the best home for rich interactive analytics. There will be other good places to do dashboard analytics, and better places to do massive problems of large complex analytic systems, close to databases. But the desktop remains a completely viable platform for analytics. So let’s keep our desktops and laptops, our PCs and Macs. They are amazingly good at what they do. Note: This is part of a Big Statistics series of blog posts by John Sall. Read all of his Big Statistics posts. ... View more

To benchmark computer performance on statistical methods with big data, we can just generate random data and measure performance on that, right? Well, it could be that simulated data may not act the same as real data. Let’s find out. Logistic Regression Suppose that we are benchmarking logistic regression. So we create some regressors as random uniform and create a categorical response that is affected by the  regressors. We use a million rows. Here is an example of a script to do that: dt = newTable("BenchmarkLogistic.jmp"); dt< dt< dt< dt< dt< If(RandomUniform()<1/(1+Exp(-(:X1-X2+X3))),"Y","N"))); Now we run Fit Model in JMP and time it. startTime = TickSeconds(); Fit Model( Y( :YCat ), Effects( :X1, :X2, :X3 ), Personality( Nominal Logistic ), Run( Likelihood Ratio Tests( 1 ), Wald Tests( 0 ) ) ); firstTime = TickSeconds()-startTime; show(firstTime); On my machine, this runs in 6.53 seconds. Not bad for a million rows. I wanted to transfer this data to another system to compare it, but it was a nuisance to have all those decimal places in the random uniform numbers. The benchmark should work just as well if I rounded them to one decimal place, so I changed my data generator script by adding a Round function. dt = newTable("BenchmarkLogistic2.jmp"); dt< dt< dt< dt< dt< If(RandomUniform()<1/(1+Exp(-(:X1-X2+X3))),"Y","N"))); Now I run the same logistic regression with everything the same except for the rounding, and it takes 1 second. Yes, it runs six times as fast by using rounded data! Does the machine have to work harder doing arithmetic with more decimal places? Of course not. It doesn’t even use decimal arithmetic. It is converted to binary, and the bits extend just as far as with the unrounded data. So what accounts for the mystery? Well, it turns out that with only 11 possible values for each X, since there are three X’s, there are only 1,331 unique combinations for the three variables. The Logistic platform actually figures out all unique combinations in the data. And when there are lots of matches, it is able to combine observations and just iterate through the unique combinations. So to logistic, most of the computing was done on 1,331 combinations instead of all 1 million rows. Thus, only a second instead of 6.53 seconds. So what is the lesson for using Logistic? Even if you have a lot of data, it is the number of unique combinations of values that counts more than the number of rows. And what is the lesson for using simulated data? If you expect a small number of levels, even if the variable is continuous, you should simulate with that small number of unique values, instead of a continuous random distribution. Now with a small number of regressors, it is easy to take advantage of the unique combinations. But as the number of regressors increases, it is less and less likely that you can collapse the data into a small number of unique combinations. Other Fitting Platforms Do the other fitting platforms do this grouping? The Least Squares personality looks for unique combinations too, but not to speed up the calculations. It needs to find them for the lack of fit test. But it turns out that if you have tens of millions of rows, it can spend seconds getting this lack of fit test, and the test is not usually interesting for large cases anyway. So then, to speed things up, when the number of rows is 20,000 or more, it changes the default for lack of fit to be off, so that it doesn’t spend all that time doing something that isn’t usually wanted. What about data in other cases, such as decision trees? Does it make a difference if the simulated data is unique or rounded? Yes, it does. When it is sorting by those X values, it takes advantage of ties by putting them off to the side so that they don’t need to be compared again -- so the fewer values you have, the faster the sort is. Clustering What about clustering? Does it matter how the data is randomly generated for measuring speed? I decided to generate four sets of simulated data to see if the details make a difference. I started with pure, independent, normal, random data: 10,000 rows by 50 columns. The time to cluster was 28.83 seconds. Now, in the real world, data is not independent. It is correlated. So the next run was the same kind of data, but transformed to be highly correlated; each column had a correlation of .95 with the others. The time to cluster that was only 4.18 seconds -- more than six times faster! Why would that be the case? It turns out that with those correlations, if you take principal components, 95% of the variation is in the first principal component. So if you were looking at distances between points -- and short-cut the distance loop when you got more than the distance you were comparing against -- then most of the time, it only has to look at the first element in order to know that some distance is not the smallest. In hierarchical cluster, we always do a singular-value decomposition (SVD) of the data before we cluster it so that the algorithm runs faster. The SVD is generally a lot cheaper than the hierarchical clustering, so this is a price usually worth paying. But the real world has data in clumps, otherwise we would not be doing clustering. Does it go faster if the data is in natural clusters rather than spread more evenly throughout space? We generated a set of uncorrelated but clumpy data, where there were 100 well-separated clumps with 100 points in each clump. The time to cluster that was only 2.83 seconds. When the data is naturally clustered, the search for near neighbors is faster, because it organizes the space better. What if you combine the clumpiness and high correlations? In that case, the time gets down to 0.98 seconds, 29 times faster than when we started, even though the data was of the same size and marginal distribution. Uncorrelated Random 28.833 Correlated Random 4.183 Uncorrelated Clumpy 2.833 Correlated Clumpy 0.983 Now what if you performed these same benchmarks in a different clustering program? The classic hierarchical clustering algorithms are slow, always taking something proportional to n^3*p time. But with JMP’s use of SVD and a very fast near-neighbor version of Ward’s method, it is much faster than the classic method -- though it depends on the data how fast, while the classic method is much more consistent and, of course, much slower. // Uncorrelated Random dtUncorr = asTable(J(10000,50,RandomNormal())); startTime = TickSeconds(); dtUncorr< timeUnCorr = TickSeconds()-startTime; // Correlated Random theCorr = .05*Identity(50)+.95*J(50,50); dtCorr = asTable(J(10000,50,RandomNormal())*Cholesky(theCorr)`); startTime = TickSeconds(); dtCorr< timeCorr = TickSeconds()-startTime; // Uncorrelated Clumped Random 100 clumps, 100 per clump dtClumped = AsTable(DirectProduct(J(100,50,20*RandomUniform()),J(100,1,1))+.1*J(10000,50,RandomNormal())); startTime = TickSeconds(); dtClumped< timeClumped = TickSeconds()-startTime; // Correlated Clumped Random 100 clumps, 100 per clump data = DirectProduct(J(100,50,20*RandomUniform()-10),J(100,1,1))+.1*J(10000,50,RandomNormal()); dtClumpedCorr = AsTable(data*Cholesky(theCorr)`); data = 0; startTime = TickSeconds(); dtClumpedCorr< timeClumpedCorr = TickSeconds()-startTime; Categorizing What about other places in JMP? Does it matter what the characteristics of the data are? Yes, it does. Generally, things go faster with categories if there are not too many of them. If there are fewer than 5,000 levels of a category, it uses associative array methods to create categories and also takes advantage of multithreading. However, if there are more than 5,000 levels, and especially if there are millions of levels, the memory allocation burden gets expensive, and it uses sorting methods instead. Conclusion If you need to benchmark the performance of a method using simulated data, it matters a lot what properties you simulate with -- it is best to try to duplicate the properties of the data that you want to apply it to eventually. In short, it depends; the details matter. JMP tries pretty hard to take advantage of typical properties of real data. Note: This is part of a Big Statistics series of blog posts by John Sall. Read all of his Big Statistics posts. ... View more