You say, "suborders would be the random effect because the fixed effect depends on it." Are you talking about nested levels. For example, in this case
![nested.PNG nested.PNG](https://community.jmp.com/t5/image/serverpage/image-id/41075iD2BA0AFCCDD06EA1/image-size/large?v=v2&px=999)
If the levels of Factor B are such that level 1 in the first row is the first and level 1 in the third row is first, but they are not the same level, then Factor B is nested in Factor A. Think of a case where two parts are test at both levels of Factor A, but they are not the same parts. There are then actually 4 parts tested. On the other hand, if Factor B = 1 is the same for Factor A = 1 or 2, then Factor A and Factor B are crossed. Using the same example, there are only two parts that are each tested under both conditions of Factor A.
I need to know which case (crossed or nested) to be sure that you specified the fixed effects correctly.
I would still try to include Sub-order as one of the fixed effects. If that works, then you do not need GLMM. You can use GLM personality instead.