Nonlinear Fit - How to check for success?

[#1]

Overview:

How can I check to see a Nonlinear fit succeeded?

(Real-data doesn't always converge and may need manual review for automation/test/product issues.)

Detail:

I would like to script a response after a nonlinear fit fails to converge.  If the fit fails, I need to process another way, and/or eventually label for manual review. (I'm sampling a population, then for each part, doing a non-linear fit.  Generally if I fail convergence, there was a test issue somewhere; I don't actually want to loosen the convergence criteria.)

('Failed' message: "Failed: Cannot Decrease Objective Function")

Modified script from Numeric Derivatives Only (Help > Scripting Index > "<<Nonlinear")

(Currently running JMP 15.2.1)

Names Default To Here( 1 );
dt = Open(
	"$SAMPLE_DATA/Nonlinear Examples/US Population.jmp"
);

// Random, intentionally divergent, formula on this dataset
dt << New Column( "Random", Numeric, Formula( Random LogLogistic(5) ) );

dt << New Column( "FancyPredictionFormula",
	Numeric,
	Continuous,
	Formula(
		Parameter(
			{Rth0 = 735, n = 0.5},
			(:year / :pop) * ((1 - ((n - 1) * :pop * Rth0) / (:year
			 * (300 / :year) ^ n)) ^ (-1 / (n - 1)) - 1)
		)
	)
);

nonlinear = dt << Nonlinear(
	Y(  :Name( "Random" ) ),
	X( :Name("FancyPredictionFormula") ),
	Expand Intermediate Formulas( 1 ),
	Numeric Derivatives Only( 1 ),
	QuasiNewton SR1,
	Interation Limit(50),
	Finish,
	Plot( 0 ),
	Confidence Limits,
	Save Estimates
);

// How to check if nonlinear fit succeeded?

Ideal.

If there is a bool status variable returned by the Nonlinear fit platform somewhere indicating success or failure, that would be the ideal method to verify success, I think.

Alternative 1.

I can also tabulate or take a summary of the data to see if all the :Name("FancyPredictionFormula") values are zero.  (Values seem to be 0 if the fit fails). 

Alternative 2.

There's probably a way to check the XML and read the 'failed' message. 

Thank you for your time and consideration.

-Aubrey

[#2 · Accepted solution]

Here's an approach using Alternative 2a (scraping the words from the report). The first 2 lines do the actual work, and the third line is just a simple example of what to do with the information; you likely have something else in mind for this part.

// How to check if nonlinear fit succeeded?
r = nonlinear << report;
alltext = r[Outline Box("Control Panel")] << get text;
if (contains(alltext, "Cannot Decrease"), caption ("Failed: Cannot Decrease ..."));

[#3]

That'll do it.  Thanks Jed!

[#4]

I wonder if a better check is if the text is anything other than success:

If( Not( Contains( alltext, "Converged" ) ),
	// trap failure
	,
	// proceed after success
);

[#5]

You are closer to where I am heading with it.

I've been working with:

if (contains(alltext, "Failed"), caption ("Failed: Do additional action now ..."));

As a catch all, but I'm considering adding more of a case statement (or I guess multiple 'ifs'), to specifically handle the fail reason.

e.g. I can run again with a higher iteration count if I catch this fail-reason:

contains(alltext, "Maximum Iteration Exceeded");

The real motivation is, 'most' parts will converge with a numerical approach, but some seem to require analytical.  I'm not sure what's happening to cause this.  Convergence criteria?  But the 'fix' is to run again without numerical derivatives, something like:

nonlinear = dt << Nonlinear(
	Y(  :Name( "Random" ) ),
	X( :Name("FancyPredictionFormula") ),
	Expand Intermediate Formulas( 1 ),
	Newton,
	Finish,
	Plot( 0 ),
	Confidence Limits,
	Save Estimates
);

After catching with:

contains(alltext, "Cannot Decrease Objective Function");

[#6]

The usual culprit behind failing to converge is the starting parameter values. It is often sufficient to use domain-specific heuristics to initialize them when available.