I got it to work as follows.
Problem: X & Y in original dataset have a lot discretization so there are occluded points.
Goal: display the data with jitter in X and Y but do the line-fit on the original data
Ex: Original Dataset has N=6, notice repeating values of X & Y
Y X
50 1
80 2
80 2
90 2
150 8
150 8
Procedure
1.Double the number of rows, adding new rows at the bottom of the dataset equal in number to the number of rows in the original dataset. In this case, add 6 rows.
2.Create new X' and Y' columns using formulas shown at bottom to repeat the original data but also have additional rows which are off by a random jitter from the original X using 1/20th (or some other fraction) of the sd of the various data points. Create new Group column to denote whether X' and Y' contain original or jittered results. (Instead of capturing the 1/20 jitter multiplier in the formulas you could also create a table variable so it is easier to change the multiplier on the fly.)
Y X Group X' Y'
50 1 original 1 50
80 2 original 2 80
80 2 original 2 80
90 2 original 2 90
150 8 original 8 150
150 8 original 8 150
jittered 0.8675 47.178279975
jittered 2.0640 79.698789972
jittered 2.0296 79.164367702
jittered 2.0775 92.320672901
jittered 7.9297 150.14705263
jittered 7.7895 151.34309192
3.Hide the first 6 rows where X is not empty (not sure how to do this with a formula). Exclude the second 6 rows where X is empty (same comment).
4.Plot Y' by X' and fit a line. Turn automatic recalc on so that you can play around with the formulas and see the effect on the line. Voila', you see only the jittered data but the line-fit is based only on the non-jittered data.
It seems you could script this and give the user dynamic control over the jittering:
script could take the original dataset and copy it to a new temp table, perform the function that I did manually to the table (double the rows, add X' and Y', do the hiding and excluding), fit a line, and give the user a way to control the multiplier used in the jitter width (I used 1/20 of orignal StdDev as the new jitter Std Dev) using a slider that sits beneath the graph and controls a constant that is used in the temp table dataset.
Formulas used:
Group = If( Is Missing( :X ), "jittered", "original")
X' = If( Is Missing( :X ), :X[Row() - Col Number( :X ), Empty()] + Random Normal() * (Col Std Dev( :X ) / 20), :X)
Y' = If( Is Missing( :Y ), :Y[Row() - Col Number( :Y ), Empty()] + Random Normal() * (Col Std Dev( :Y ) / 20), :Y)