topic Re: Least squares regression for nonlinear system of equations in Discussions

Least squares regression for nonlinear system of equations

ehchandlerjr — Sun, 10 Dec 2023 21:39:41 GMT

Hello - I have a moderately complex model for a physical process (ion adsorption for anyone who knows what that means). I have two starting variables, x1 and x2, and two measured outcomes, y1 and y2 that change with x1 and x2, and unmeasurable, or hard to measure, variables, g1,...,gn that don't change. One or both of x1 and x2 can also change for each row in a series. I have a set of nonlinear equations, s = f(x1, x2, y1,...,y2, g1,...,gn), as well as some other functions of different variables in f, depending on how complex I want the model to be. See the attachment for example equations.

The typical way we do this is in Matlab with lsqnonlin. Works well, but it's a bear to get running. I'd much rather have a (not-so-simple) column formula. However, I'm not entirely sure how to code it. The nonlinear regression thing is built for single equations, and with the three equations that all describe the same s, it seems like recursion would abound. I'm also not sure what the loss function would be. In matlab, we input guess values, calculate the least squares between the guess and the calculated value, update it if the sum of squared differences of all the guesses and equations falls outside some tolerance, and then loop. I'm assuming this is done internally by the nonlinear platform, but then is my target zero since I want the sum of squared differences to be zero? Or if not, how do I do multiple equations?

Apologizes if this is way simpler than I'm thinking.

Re: Least squares regression for nonlinear system of equations

peng_liu — Mon, 11 Dec 2023 16:22:07 GMT

One way that I suggest is an old trick by defining an appropriate lost function. In your case the lost function can be:

(s-f1(...))^2 + (s-f2(...))^2 + (s-f3(...))^2 + ...

Each element is the squared error of an equation in your systems. You sum them up, and minimize the overall error. This approach converts a nonlinear least squares problem to a loss function optimization problem.

Meanwhile, I think it is still doable using nonlinear least squares for a system of equations. Here are the steps:

Expand your data table. For each row, you make copies of it, and the total number of copies the the same as your number of equations in your system. This add-in might be useful for this task. You need to create a Freq column, whose cell values are the number of equations.
Now you need to create that not-so-simple formula column. Depending on how your arrange the rows from the last step, you may need to write the formula differently. But the goal is the following. For each copy of the original row, assign it to one equation in your system. So, in the end, every copy of a row is associated with a distinct equation in your system. You probably need to figure out how to associated row number with equation number. If the copies are in consecutive rows, as the result from the above add-in, you may need the "Modulo" function to make the mapping.

In the end, use the Nonlinear as you know to solve the least squares problem.

Re: Least squares regression for nonlinear system of equations

David_Burnham — Mon, 11 Dec 2023 18:48:49 GMT

I have some code that I use to model a moisture adsorption process - it seeks equilibrium conditions as described by multiple physical models - I'll see if I can make a generic version of the code that I can share on here.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Fri, 15 Dec 2023 14:48:39 GMT

Hi @peng_liu thanks for the answer and sorry for the delay in response! Dissertation + 2 month old baby make you forget to follow up on things.

What would you chose as the s? I don't have any s measured, and in the matlab code, we guess an s, and then update. How would you suggest doing this?

As for the least squares method, that looks interesting. Do you have a table you've made for another project I could look at that does a similar thing? For all my scripting, I've not done much with row based formulas and I've failed spectacularly when I have.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Fri, 15 Dec 2023 14:49:38 GMT

Hi @David_Burnham - yea if you had a generic version of that, that would be extremely helpful.

Re: Least squares regression for nonlinear system of equations

peng_liu — Fri, 15 Dec 2023 19:11:08 GMT

Congratulations!

So, "s" is an unknown parameter here. In that case, rewrite all your equations to something like this: 0 = f(...) - s.

Or if you have known quantities in your equations, move them to the left hand side. The left hand side will be your Y when you use the Nonlinear platform.

I have no examples, so I made up one. Attached: nonsense system of eqns.jmp

It has

a Y column, known quantities.
a formula column, the stuffs on the right hand size of your equations
a label column indicates odd rows are corresponding to the first equation and the even rows are corresponding to the second equation
other known quantities that are part of the equation on the right hand side.

Look at the formula in the "formula" column to see how to dispatch two equations in a single column.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Mon, 18 Dec 2023 20:37:20 GMT

Thanks for the guidance @peng_liu!

I've put together a table and it seems to work if I have a couple of rows, but if I try to do more than a few, it gives me this:

Is this indeed an issue with just having too many rows? The error seems to persist if I cut down the rows to the minimum (3 in this case) and I have to go make a new table. Do you have any thoughts on this? Here's the table I've made with a barebones version of the model so its not cumbersome. I've also put the parameters into their own columns (Phi0, sigma0, pHf) and expanded intermediate formulas to make looking at the column formula easier.

Re: Least squares regression for nonlinear system of equations

peng_liu — Mon, 18 Dec 2023 21:01:34 GMT

I am not sure how to run Nonlinear platform using this data table. Which is Y, which is X?

But if "Formula" column is X, I don't think it would work. The formula of X must use Parameters, not Table Variables.

The following is what the X column looks like in that "nonsense data table".

Your "Formula" column does not have anything listed under Parameters.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Mon, 18 Dec 2023 21:54:52 GMT

Hey @peng_liu - Yea so because the parameters were making the formula 100 times as long, I dumped them into the Phi0, sigma0, and pHf columns (columns 6-8 after the column with a "." as the title), and then was selecting the "Expand intermediate formulas" box at the bottom of the nonlinear platform. That way I can use the parameters but they aren't making the formula column impossible to work with, since there are so many "expanded categories" for each parameter.

Does that make sense?

The attached screenshot should show those columns and their location in the formula, in case that makes it clearer.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Mon, 18 Dec 2023 22:12:15 GMT

@peng_liu I will also say that I've found that 1) numerical derivatives only option does seem to get it to go through to the main platform, and 2) that when I have parameters in more than one of the equations, it seems to give that error. Not sure if that tells you anything. I just can't find a reference to that error so I'm stumped.

Re: Least squares regression for nonlinear system of equations

peng_liu — Tue, 19 Dec 2023 00:46:51 GMT

Thanks for explaining, @ehchandlerjr I understand what you are doing now. What you are doing is a nice trick for such a complicated problem. Something new to me!

I looked at the error message. You need to check "Numerical Derivatives only".

You have 300+ parameters, the underlying symbolic derivative is hitting a limit.

I see you have 337 rows, but 339 parameters. This subset won't get a solution. Hope your full set does not have the issue.

Meanwhile, while I am looking at your setup, something is interesting. I don't know your subject. But in your setup, Phi0, sigma0, and pHf are discrete functions of pH0. Are the relationships between the three and pH0 known? If continuous relationships (with fewer unknown parameters) are known among them, you might largely reduce the number of parameters.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Tue, 19 Dec 2023 12:42:01 GMT

@peng_liuok that makes more sense. Do you happen to know the limit on number of terms or parameters in the symbolic derivative code? Because I'm thinking I can just have a column with a few numbers and then segment the calculation using the "by" feature. The symbolic derivative is just so much faster.

And hmmm. Not sure where the new parameters came from. I'll check it when I get in. Thanks for noticing that. Three related questions: 1) how did you see the number of parameters and 2) is there a way to bulk delete parameters when in the column formula menu? As far as I can tell, you have to right click, click delete one by one. 3) is there a way to use a table to input the parameters and their bounds? This would really help speed.

Thanks!

Re: Least squares regression for nonlinear system of equations

peng_liu — Tue, 19 Dec 2023 14:01:50 GMT

The limit is not in terms of number of parameters. But the number of parameters is a factor in your case. So, you have to try and see.

I see your system has three distinct types of parameters: Phi, sigma, pHf. They are all conditioned on value of pH0. Therefore, the parameters are independent by different pH0. So, you are right, do a "By" fit can reduce the complexity.

P.S. "By" is not working as I wished. All rows see all parameters in the column. I don't see it is possible to let individual rows in By to see just a subset of parameters.

I see the number of parameters after launching the Nonlinear. There is a able of parameters. I right click and create a table out of it, count rows. For the last two rows, I would suggest just edit the JSL formula. Open up the formula editor of the column. Copy it, and paste into a JSL editor. After editing, copy and paste back. By such, you can edit the list of parameters:

The "1" section is your list of parameters. The "2" is the actual formula.

Re: Least squares regression for nonlinear system of equations

peng_liu — Tue, 19 Dec 2023 17:45:05 GMT

Follow up on how to use "By".

Seems there is a way to use By and reduce the number of parameters to take advantage of independence among rows.

I attach v2 of the "nonsense system". First explain what I set up.

I set up two systems of linear equations. Row 1 and 2 are system "a", the Group column. And row 3 and 4 are system "b".

Each linear system has two parameters "b0" and "b1", but they are different between systems. And I use the parameters, x1, and x2 to produce "SimY". I copy SimY to Y for my analysis.

Now look at the "formula" column, I do not differentiate "b0" ad "b1" by Group.

Now run Nonlinear with By = Group. And here is the report:

It finds separate, correct, solutions for respective systems.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Tue, 19 Dec 2023 19:29:55 GMT

@peng_liu Yea I had run into the by issue as well. You're solution looks good! However, when I try to apply it to my table, it errors out. Do you see a glaring issue in the attached table? the Phi0, sigma0, and pHf are all the same as before. However, I made a "cyclic categorization" column for the "by" method you mentioned, but then have a "parameter by column". I use the levels of this column to expand parameters by categories. Since :Cyclic Categorization and :Parameter by column are orthogonal (or whatever the correct word is), this should allow the parameters to iterate over the whole curve at slight offsets (pH0 is the x axis), and then I can just put them together again at the end. However, like I said, its failing and only updating one of the levels of the parameters, and not sure why.

Could this be an issue with how the by interacts with parameters expanded by categories?

Re: Least squares regression for nonlinear system of equations

peng_liu — Tue, 19 Dec 2023 20:25:54 GMT

You have missing values in Column3, which explains why only one group has result. But the "formula" is not set up as I recommended. I attach test.v2 to illustrate. With this approach, you no longer needs the Phi0, sigma0, and pHf in the middle.

Re: Least squares regression for nonlinear system of equations

ehchandlerjr — Tue, 19 Dec 2023 20:52:51 GMT

@peng_liu Oh duh. Not sure how I missed that many missing rows. Thanks for noticing that.

So maybe I'm misunderstanding, but since pH0 is my x axis, does your solution just mean the by column is the x column? like if you have 100 x values, you have 100 "groups" so to speak?

If so, wouldn't the expanding by columns version be better since you are minimizing the error across multiple points rather than just a self contained minimization for each one? Or is point by point a better way to deal with loss functions? I'm kinda just fitting the matlab code I had to JMP, but don't really have expertise in minimization algorithms, so whatever the best technique is, I'm down for.

P.S. And maybe I didn't make it clear, but there should be a unique set of values of each of the parameters for each pH0 value (which is the dependent variable). Which is why I thought I needed to expand by categories.

Thanks!

Re: Least squares regression for nonlinear system of equations

peng_liu — Wed, 20 Dec 2023 02:06:20 GMT

I see your "formula" column in the original table depends on pH0. But the "formula" column in your test.jmp does not depend on pH0. I guess that is a misunderstand between us. Let me still use your original data, so Phi0, sigma0, pHf depend on pH0, not "Parameter by column".

Let's look at the first three rows:

Their pH0 = 0.2. Because of that, the formula of the first row can be denoted by:

f1(Phi0_pH0_0.2, sigma0_pH0_0.2, pHf0_pH0_0.2).

The 2nd and 3rd row formulas are:

f2(Phi0_pH0_0.2, sigma0_pH0_0.2, pHf0_pH0_0.2) and

f3(Phi0_pH0_0.2, sigma0_pH0_0.2, pHf0_pH0_0.2). The three rows form Group 1.

Now look at row 4, 5, 6. Their pH0 = 0.3. So the three formulas are:

f1(Phi0_pH0_0.3, sigma0_pH0_0.3, pHf0_pH0_0.3),

f2(Phi0_pH0_0.3, sigma0_pH0_0.3, pHf0_pH0_0.3), and

f3(Phi0_pH0_0.3, sigma0_pH0_0.3, pHf0_pH0_0.3). The three rows form Group 2.

Group1 and Group2 do not share same set of parameters. They are independent. Minimizing errors for Group1 has nothing to do with minimizing errors for Group2. So, we can optimize group by group.

If you keep parameter set in the three intermediate columns as Phi0_pH0_0.2, Phi0_pH0_0.3, Phi0_pH0_0.4, etc. each row sees all of them. You can see that if you run Nonlinear by pH0, and every group will have hundreds parameters listed. But among them, only three are relevant for each row, and every three rows.

What I did is creating three parameters only, inside of the column "formula". I named them paramPhi0, paramSigma0, parampHf. Even there are just three, when we run Nonlinear by "pH0", the platform will create different sets of the three parameters by "pH0". It literally creates those hundreds of parameters for you by "pH0". And each little Nonlinear by pH0 just need to worry about three parameters.