Not getting same results in neural network run from menu vs run from script.

[#1]

I am trying to run a script that automates screening the eight different penalty modes in Neural Network and outputting the results into the current table. I have been troubleshooting why I am not getting different results from the different Fits in the script. This led me to look at the results for each individual penalty mode when the model was set up from the Neural Menu vs the script. The script was made by saving the analysis script to the Data Table and then running it. So the results should be identical.

Set up of model from Analyze>Predictive Modeling>Neural

Initial run of modelRedo Analysis #1Redo Analysis #2

If I relaunch the model and select all the same settings, I get the same result as the Initial Result above. If I run from a script, I get the same results as from Redo Analysis.

[#2]

Also, the following scripts all give the same result: It's as if neural network models run from a script only use one of the penalty methods

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("Squared"),
		Number of Tours( 3 )
	)
)

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("Absolute"),
		Number of Tours( 3 )
	)
)

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("Weight Decay"),
		Number of Tours( 3 )
	)
)

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("NoPenalty"),
		Number of Tours( 3 )
	)
)

[#3]

In your scripts you have this line:

Validation Method( "Holdback", 0.3333 )

And that's your problem. Each time you run this you're getting a different random holback set.

Since you're using JMP Pro, set up a Validation column and use that.

1. get to control which rows are in the holdback, like when you make a stratified subset.

2. use the same rows for the hold back for each version of the model. 

[#4]

My problem is that I'm not getting different results when running from a script using a fixed seed and different penalty methods.

[#5]

My problem started when I was trying to run 2 instances of JMP, each running a script with 4 NN runs. Instance 1 was running seeded models using Squared, Robust Squared, Absolute and Robust Absolute penalties. Instance 2 was running the exact same script (in a differently named file) except the 4 models were running Weight Decay, Robust Weight Decay, No Penaly and Robust No Penalty NNs.The runs take ~ 20 hours to complete and the script saves the formulas to the data table at the end. What I found was that both instances returned exactly the same results. There were 4 different results in each instance, but they were exactly the same. I tried various combinations to try to find out what was going on. I made every variable unique between the two scripts. I used a text box to set the Data Table to a variable in each instance. No matter what I did, I could not generate different results between the two instances.

If I run each model from the menu, I get eight different results. If I run in a script, I do not.

[#6]

I have uploaded a small model with script to illustrate what I mean. When you run the script, you will see that the first NN fit is unique; all others give the same results.

[#7]

There are actually two different facets of the same problem. One is that I do not get different results if I reset the seed between Fits in the script but use different penalty methods. Because of this, I can't trust that the differences I am seeing when I don't reset the seed are real; they could just be the same penalty method being used but starting from a different place. This is what I illustrated in the uploaded file.

The second (original) problem is that two different instances running the same model but with different penalty methods are giving the same result so I can't be sure which method is actually being used anywhere.

[#8]

There is a reasonable chance that I'm completely misunderstanding what you're doing and your script; however, on my end I'm able to different and reproduible results. 

Neural(
	Y( :Target ),
	X( :Name( "8-12_Mean" ), :Name( "12-16_Mean" ), :Name( "16-20_Mean" ) ),
	//Validation Method( "KFold", 5 ),
	Validation( :Validation ),
	Informative Missing( 0 ),
	//Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 1 ) ),
	Fit( NTanH( 3 ), Robust Fit( 1 ), Penalty Method( "Absolute" ) ),
	Fit( NTanH( 3 ), Robust Fit( 1 ), Penalty Method( "Weight Decay" ) ),
	Fit( NTanH( 3 ), Robust Fit( 1 ), Penalty Method( "NoPenalty" ) )
);

First, you'll have to make a validation column.  Analyze/Predictive Modeling/Make Validation column (jmp pro v13,14) Make it stratified on :Target

When I run this script, as is, I get different estimates for error between fits and between runs of the script, although the between runs are pretty close.

Uncomment the random seed line: I get different estimates between fits and between run of the scipt the results are the same.

[#9]

Having a validation column and setting the random seed does not fix the problem. I made a Random KFold(5) validation column (ValidationCol) and used it in the code below. I have also checked using a Holdback validation column and the behaviour is the same.

First, if you comment out the Fit() from top to bottom, you see that the second (Absolute), third (WeightDecay) and Fourth (NoPenalty) penalty modes give the same result as the first (Squared). Second, if you reset the Random Seed to 999 between Fits the results are the same for all the Robust Models and the same for all Non-robust models. The Set Random Seed function must be doing something (reset?) to the model parameters at runtime.

My concern is that the only way to get reproducability is by using a seed, but doing so seems to 'fix' the penalty mode.

Neural(
	Y( :Target ),
	X( :Name( "8-12_Mean" ), :Name( "12-16_Mean" ), :Name( "16-20_Mean" ) ),
	Validation( :ValidationCol ),
	Informative Missing( 0 ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 1 ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 1 ), Penalty Method( "Absolute" ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 1 ), Penalty Method( "Weight Decay" ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 1 ), Penalty Method( "NoPenalty" ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 0 ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 0 ), Penalty Method( "Absolute" ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 0 ), Penalty Method( "Weight Decay" ) ),
	Set Random Seed( 999 ),
	Fit( NTanH( 3 ), Robust Fit( 0 ), Penalty Method( "NoPenalty" ) )
);

 Robust Squared

Robust Absolute

Robust Weight Decay

Robust No Penalty

All Robust, reset seed for each fit

All, non-robust, reset seed for each fit