Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

[#1]

Hi All,

I'm trying to write a script that will generate a random dataset (n=3 reference and test measurements) and then run a simulation with some number (1000 or whatever).  For each set I'll run an equivalence test and I'd like the output to be a report/table with the frequency of events where equivalence was demonstrated.  The goal of this tool will be to give a probability of type 1 errors given a certain equivalence margin.

I've landed on the following code but this is not working and I'm pretty much stuck.  Any direction/help that the group can provide would be much appreciated.

Thanks!

Chris

// Parameters
numSimulations = 10;
nPerGroup = 3;
refMean = 99.1;
testMean = 98.8;
sd = 0.1;
equivMargin = 0.3;
type1Errors = 0;

// Critical t-value for 90% CI with df = 2n - 2
df = 2 * nPerGroup - 2;
tCrit = StudentTQuantile(0.950, df);

// Create summary table
summaryTable = New Table("Simulation Summary",
    New Column("Simulation", Numeric),
    New Column("CI_Lower", Numeric),
    New Column("CI_Upper", Numeric),
    New Column("Equivalence_Concluded", Character),
    New Column("Mean_Diff", Numeric)
);

// Loop through simulations
For(sim = 1, sim <= numSimulations, sim++,

    // Create new data table
    dt = New Table("Sim" || Char(sim),
        Add Rows(nPerGroup * 2),
        New Column("Group", Character),
        New Column("Value", Numeric)
    );

    // Simulate Reference group
    For(i = 1, i <= nPerGroup, i++,
        dt:Group[i] = "Reference";
        dt:Value[i] = Random Normal(refMean, sd);
    );

    // Simulate Test group
    For(i = nPerGroup + 1, i <= nPerGroup * 2, i++,
        dt:Group[i] = "Test";
        dt:Value[i] = Random Normal(testMean, sd);
    );

    // Extract values
    refVals = dt << Get As Matrix( :Value )[1::nPerGroup];
    testVals = dt << Get As Matrix( :Value )[nPerGroup+1::nPerGroup*2];
    meanDiff = Mean(testVals) - Mean(refVals);

    // Sample variances
    varRef = Variance(refVals);
    varTest = Variance(testVals);

    // Pooled variance
    pooledVar = ((nPerGroup - 1) * varRef + (nPerGroup - 1) * varTest) / df;
    pooledSD = Sqrt(pooledVar);

    // Standard error of difference
    seDiff = pooledSD * Sqrt(2 / nPerGroup);

    // 90% CI
    ciLower = meanDiff - tCrit * seDiff;
    ciUpper = meanDiff + tCrit * seDiff;

    // Check equivalence
    eqResult = "No";
    If(ciLower > -equivMargin & ciUpper < equivMargin,
        type1Errors++;
        eqResult = "Yes";
    );

    // Log results
    summaryTable << Add Rows(1);
    summaryTable:Simulation[sim] = sim;
    summaryTable:CI_Lower[sim] = ciLower;
    summaryTable:CI_Upper[sim] = ciUpper;
    summaryTable:Equivalence_Concluded[sim] = eqResult;
    summaryTable:Mean_Diff[sim] = meanDiff;

    // Close the table to avoid clutter
    dt << Close Window;
);

// Show results
New Window("Type I Error Summary",
    Text Box("Total Simulations: " || Char(numSimulations)),
    Text Box("Type I Errors: " || Char(type1Errors)),
    Text Box("Estimated Type I Error Rate: " || Char(Round(type1Errors / numSimulations * 100, 2)) || "%")
);

[#2]

Which part are you stuck with?

[#3]

H Jarmo,

At the moment the code breaks trying to extract the critical T values from the equivalence report.  But there are other issues as well.  I've had it in a state where it will run the simulation and equivalence tests but not be able to get values out of the report.  

I've tried throwing some of this at AI as well but I think some of the syntax and coding conventions might be incorrect.

Chris

[#4]

I would suggest these edits:

There's no StudentTQuantile() function.  That should be t Quantile().

Variance() is not a built-in function.  Define it in your script (or just square the Std Dev() instead).

refVals = dt << Get As Matrix( :Value )[1::nPerGroup] should be more like refVals = dt:Value[Where( :Group == "Reference" )];

Same for testVals =

Set the visibility for dt to "Private" to speed things up greatly.

Variance = Function( {Values},
	Return( Std Dev( Values ) ^ 2 )
);
// Parameters
numSimulations = 1000;
nPerGroup = 3;
refMean = 99.1;
testMean = 98.8;
sd = 0.1;
equivMargin = 0.3;
type1Errors = 0;

// Critical t-value for 90% CI with df = 2n - 2
df = 2 * nPerGroup - 2;
tCrit = t Quantile( 0.950, df );

// Create summary table
summaryTable = New Table( "Simulation Summary",
	New Column( "Simulation", Numeric ),
	New Column( "CI_Lower", Numeric ),
	New Column( "CI_Upper", Numeric ),
	New Column( "Equivalence_Concluded", Character ),
	New Column( "Mean_Diff", Numeric )
);

// Loop through simulations
For( sim = 1, sim <= numSimulations, sim++, 

    // Create new data table
	dt = New Table( "Sim" || Char( sim ), "Private", Add Rows( nPerGroup * 2 ), New Column( "Group", Character ), New Column( "Value", Numeric ) );

    // Simulate Reference group
	For( i = 1, i <= nPerGroup, i++,
		dt:Group[i] = "Reference";
		dt:Value[i] = Random Normal( refMean, sd );
	);

    // Simulate Test group
	For( i = nPerGroup + 1, i <= nPerGroup * 2, i++,
		dt:Group[i] = "Test";
		dt:Value[i] = Random Normal( testMean, sd );
	);

    // Extract values
	refVals = dt:Value[Where( :Group == "Reference" )];
	testVals = dt:Value[Where( :Group == "Test" )];
	meanDiff = Mean( testVals ) - Mean( refVals );

    // Sample variances
	varRef = Variance( refVals );
	varTest = Variance( testVals );

    // Pooled variance
	pooledVar = ((nPerGroup - 1) * varRef + (nPerGroup - 1) * varTest) / df;
	pooledSD = Sqrt( pooledVar );

    // Standard error of difference
	seDiff = pooledSD * Sqrt( 2 / nPerGroup );

    // 90% CI
	ciLower = meanDiff - tCrit * seDiff;
	ciUpper = meanDiff + tCrit * seDiff;

    // Check equivalence
	eqResult = "No";
	If( ciLower > -equivMargin & ciUpper < equivMargin,
		type1Errors++;
		eqResult = "Yes";
	);

    // Log results
	summaryTable << Add Rows( 1 );
	summaryTable:Simulation[sim] = sim;
	summaryTable:CI_Lower[sim] = ciLower;
	summaryTable:CI_Upper[sim] = ciUpper;
	summaryTable:Equivalence_Concluded[sim] = eqResult;
	summaryTable:Mean_Diff[sim] = meanDiff;

    // Close the table to avoid clutter
	dt << Close Window;
);

// Show results
New Window( "Type I Error Summary",
	Text Box( "Total Simulations: " || Char( numSimulations ) ),
	Text Box( "Type I Errors: " || Char( type1Errors ) ),
	Text Box( "Estimated Type I Error Rate: " || Char( Round( type1Errors / numSimulations * 100, 2 ) ) || "%" )
);