Combining DOE and First-Principles Science to Maximize Yield and Minimize Impurity with Curve DOE (2021-US-30MP-877)

[#1]

Level: Intermediate

Brian Taylor, Statistician, AstraZeneca
Chris Gotwalt, Chief Data Scientist, JMP

Recent releases of JMP Pro have greatly expanded the modeling of functional response data with the Functional Data Explorer (FDE). FDE fits flexible smoothing splines to the response trajectories, extracts functional principle components (FPC) scores from the spline models, and then applies DOE methods to these FPCs. This empirical, data-centric approach ignores the knowledge of subject matter experts who may know the equations that describe the response curves based on first-principles science. In JMP Pro 16, Curve DOE is a new addition to the Fit Curve platform. With it, one chooses an equation suggested by the first-principles subject matter knowledge. The first-principles model is then fit separately to the individual response curves; the parameters of these nonlinear equations are modeled using generalized regression, just as FDE models FPC scores. By directly incorporating engineering and scientific knowledge via these equations, models more realistically extrapolate to new combinations of factor settings not present in the data, while also being more robust to missing data and functions sampled over different ranges of X. Applying this approach to pharmaceutical product development (simultaneously to product yield and impurity reaction curves) illustrates the ease of use and practical statistical advantages of using model selection criteria.

Auto-generated transcript...

Speaker	Transcript
Peter Polito	Alright I'm in the zoom meeting.
Brian T	hi Peter.
Peter Polito	hey Brian.
Brian T	There we go.
	perfect right.
Peter Polito	All right.
Brian T	Give me a second X, I need to.
	Close oh yeah.
	I'll look at all those types of things.
Peter Polito	Take your time you let me know when you're ready.
Brian T	yeah, so this is a slightly.
	Maybe some more complicated I don't know I'm doing I'm co presenting the talk with Chris gold, but he's actually on holiday so he's pre recorded his part for me to push play, and he did say I needed to change some settings in zoom for it for the audio to work on his recording.
Peter Polito	yeah so when you start your.
	Let me just do that real fast to remind myself when you start your share you want to click share sound, but you don't want to opt in click optimize for video clip that actually.
	doesn't help, despite it sounding like it should.
Brian T	Right, so I just got to try that again I just need to say share.
Peter Polito	In the lower left there'll be two checkboxes one says share sound.
	Okay, it says optimized for video clip and you want to click the share sound.
Brian T	Just share sound okay.
	Okay that's.
Peter Polito	perfect and what might be good is before you start.
	let's just share that video and make sure it works Okay, so that, in the middle of your talk when you do it we we know what took.
Brian T	yeah yeah absolutely and the other thing was if I put my slide deck into presenter mode, will the video see my presenter mode and my slides or is it just.
Peter Polito	um, I think, well, it should show just your slides but, again, we cannot test that before you start.
	cast to make sure.
Brian T	Okay, no worries because I've yeah.
	OK cool so I'll I'll just test the video, first of all.
	Okay.
Brian T	So I I can hear the.
	I can hear the video in my headset.
Peter Polito	I am not seeing anything yet.
Brian T	Right, so I why I think I need to do is share.
	Do that share that you said.
Peter Polito	You want to share a screen, not just an application.
Brian T	So I'll just share the whole screen yep okay.
	There we go right sorry right so that's that and then.
	He said that I needed to change the audio on.
Peter Polito	In the end, when you do that share screen in the lower left it will say share our share computer audio I think.
Brian T	just looking now and.
	Share sound.
	They go yeah it's in the in the middle section so I'll just just sound right okay let's.
	give this a go now.
	Thanks Brian now I'm going to demonstrate the analysis of these four curve of linear responses product product impure yeah starting material.
Peter Polito	And I can hear you just fine this analysis mom than I had originally thought it would be.
Brian T	Loud or soccer has made.
Peter Polito	No, no that's fine bro.
Brian T	17 I'll call these things out as I go and I want to thank clay for.
	dedication okay let's just with that back to the beginning again ready for.
	that's it done.
Peter Polito	All right, and then we need to there's a few settings they've asked us to just.
	optimize.
	Let me analyze up.
	And, before I do that, I just want to confirm your talk is combining do you see it right here, combining do we first principles, science and maximum to maximize yield and minimizing period with curve DOE, and then I just need a little disclaimer here.
	You understand this is being recorded for the use of chump discovery summit conference and will be available publicly in the JMP user community do you give permission for this recording and use.
Brian T	So yes, when will it be.
	announced when when will it be available to the junk community.
Peter Polito	I think they release them, not more than a week before.
Brian T	Okay.
Peter Polito	Now that's fine certain.
Brian T	As long as as long as I get the opportunity, just to say go ahead because well I'll need to do is share this recording or the full slide deck to get it kind of approved stuff I've had a pre approval.
	So that's why I can record it, but then, as long as I've got two or three weeks, I will I will get this through.
Peter Polito	And yeah so you have about one month.
	I guess.
	The first week of October and I'll let.
	You know that we have a little asterisk next to yours and that she needs to confirm with you, before it gets shared.
Brian T	Yes, yes please yeah.
	Okay, but there shouldn't be any problems with it it's just processes within companies.
Peter Polito	No, I totally get I work a lot with oil industry and.
	Getting anything approved can take a long time.
Brian T	yeah yeah cool.
	So I'm right.
Peter Polito	We just need to check your display settings real fast.
Brian T	If you got shot.
	I was just going to see if, when I go to presenter whether.
Peter Polito	it's still sells my goal and presenter I see.
	It and I don't I see the whole presenter mode your notes and everything.
Brian T	Right okay so.
Peter Polito	Is there a way is it showing just one slide On your other monitor maybe you could just flip flop things.
Brian T	That I've just got on one screen at the moment it's just quite.
	A large screen okay so that's fine if it's.
	I'll just I'll just do off and some printed some slides some.
	handouts just.
Peter Polito	Just in case of this adventure three.
Brian T	Okay right so that's not that's not that's a no go Okay, let me.
	write okey dokey so sorry go ahead with your.
Peter Polito	yeah so in the lower left if you're on windows 10 there should be like a type here to search and if you just type in display settings.
	Show.
Peter Polito	And it's going to be a change resolution it's up, I think we get from here so on display resolution.
	Lets you have a much.
	bigger than others so scale.
Brian T	looks but set it to 19 2010 at.
Peter Polito	1030 yeah.
	OK now we'll make it easier for folks to read.
	Your slides and everything I know it's probably looks kind of weird to you.
Brian T	yeah that's fine yeah it's I've got a 34 inch screens our lovely yeah I guess it's.
	Because the company is giving me a best screen for home how that works.
Peter Polito	So tell us back to the office you.
	know where is the Office for you what city.
Brian T	it's leads, where I live in Leeds but it's medical field so.
	The big city would be Manchester.
	yeah.
Peter Polito	I embarrassingly I know of leads because of the WHO and their live album but I don't know much beyond that.
Brian T	So if you if you know that I live five five minutes away with where they performed in a massive.
Peter Polito	Oh.
Brian T	In the city.
Peter Polito	that's awesome.
Brian T	place I think it'd be roundhay Park, which is.
	Okay, was probably done life is just an open air that's pretty cool yeah.
Peter Polito	Very cool.
Brian T	yeah right Okay, so I get that stuff done.
Peter Polito	And then it's like your cell phone off and you sounds like you already closed out looking at anything else that might make some weird noises in the background.
Brian T	yeah I'm just gonna.
	just going to double check everything off so I've just got the video clip zoom and.
	Get rid of.
Brian T	These.
Peter Polito	And then will you be showing everything in PowerPoint or will you be going between PowerPoint and JMP.
Brian T	know so a lot of chris's video is almost screenshots of what he would do and JMP, but I have a near the end of my so I'm I'm starting an intro then I'm cutting into chris's video, and then I come back and finish off talking about some things, and when I get to that point.
	I will then.
	have already got to open.
	well.
	So I will just open this up and then why I'm actually after is.
	To talk about this.
	Okay profiles, I will just talk a little bit about that.
	And then I have to cut back to my slide deck again just to finish off a summary.
Peter Polito	Okay.
Brian T	So being about.
Peter Polito	And I'm at this size everything looks good you're welcome to make those boxes, a little bit bigger if you want to take advantage of the space so folks can see it a lot easier, but they can I can see it just fine now, if this is how you like to keep it.
Brian T	yeah well yeah, this is it maximize on my screen.
	Oh yes, yeah I mean like you can drag the corner of the boxes of that production profile right.
Peter Polito	All of them bigger.
Brian T	Okay, and.
	If you know, I will try remember to do that it's probably best if I show them that it is actually coming straight from here.
Peter Polito	And then understand.
Brian T	And then I can.
	yeah something like that.
Peter Polito	And just seeing if you do not to so I don't want to micro yeah if you make it that but just want to make sure you can still see.
	The tag on that X axis.
Brian T	yeah exactly yeah.
Peter Polito	Yes, so Okay, so I will.
	I'll give you a 321 and then I'm going to go on mute and unless there is something major going on, that you don't notice in.
	Whatever I plan I'm not saying a single word until you're finished I'm just going to be here, making sure it's recording properly and you just tell me when you're finished.
Brian T	yeah no worries right so I'm just just get my bearings.
	sure.
Peter Polito	Well, I will go on mute and you start whenever you would like and good.
Brian T	Okay, thanks.
	just going to test a few things out for a guy.
	combining doing first principles science to maximize yield and minimize impurity with fit curve DOE.
	Talk touches on a variety of topics and if we do not cover them in sufficient detail, you'll have the slides and the video to refer to afterwards.
	AstraZeneca is a pharmaceutical company focusing on three main research therapy areas to deliver medicines by leveraging a variety of capabilities.
	I work in chemical development multi-skilled department to develop the process for making the active pharmaceutical ingredient in a medicine.
	We sit in pharmaceutical technology and development within operations, but we are organizationally at the interface between researching and manufacturing medicines.
	As a result, we're involved in many steps in a medicine's life cycle.
	Okay, on to the talk. To set the scene, we need to look over time to understand why there is a need now to develop experimental approaches.
	In the early 2000s when I joined AZ, a lab experiment typically constituted a single reaction and a couple of samples taken over time.
	Moving to present day, an experiment involves multiple reactions run in parallel, taking many samples from each reaction. Experiments are much more data rich, containing useful process insight if you can combine experimental approaches, analysis methods, and subject matter expertise.
	To my knowledge, there is not a clear front runner approach to how to analyze experimental designs, where the response is the chemical reaction profile over time.
	Current approaches tend to analyze subsets of the data, rather than analyzing all the data available, and I wonder what process learning has missed.
	The gap in what is currently done compared to how I would like to analyze data raises two questions. What analysis approach can make use of all the time course reaction profile data?
	Can we refine the analysis to consider subject matter knowledge?
	This leads us on to our grand presentation title, which translates into what we do in JMP, which is analyze DOE reaction profiles with fit curve and curve DOE.
	The DOE case study used is a package of work from where I work, chemical development,
	just the details anonymized to allow me to share. In this case, a four factor design was set up with multiple samples taken from each reaction in the design.
	There are response target criteria we are interested in achieving and, ideally, all of them, but the most important to comply with is product impurity staying below 2%.
	I originally looked at functional data explorer as a solution to analyzing reaction profiles.
	FDEs are very flexible, and this can be a strength and a weakness, as unrealistic profiles are approximated. As these profile approximations are carried through to the DOE analysis, the response predictions can be unrealistic, exceeding maximum or minimum bounded levels.
	Working with the subject matter expert, we know quite a lot about what to expect a reaction profile to look like and how responses might be linked. For example,
	when we sample a reaction, we expect the starting material, the starting material impurity, and the product, and product impurity to add up to 100%, providing nothing else is formed.
	Input ingredients must balance with output components. The mass balance must be maintained.
	from the starting material impurity or formed from the product.
	Starting material impurity is not always guaranteed to increase over time and could reduce if it reacts on to form the product impurity. The process insight also inputs into what shape to expect a response...a response's reaction profile to exhibit.
	This led to the idea of if we accept a reduction in the flexibility in profile shapes approximated, can we fit more realistic curve approximations more closely aligned to first principle understanding?
	The analysis output is then in a form more relatable to the way scientists think.
	JMP has a variety of preloaded formulas, which can be matched to the expected response profiles for the given chemistry investigated.
	responses, factors, timelines.
	We fit various pre loaded curves, compare them, select the best fitting, ideally, according to some criteria involving analysis metrics and subject matter expertise.
	Curve DOE analyzes the curve formula parameter coefficients, which JMP convert in the background into original DOE responses to show in the prediction profiler, along with the DOE factored. So pretty neat.
	Chris will now demonstrate the approach, pointing out what we encountered along the way, and where we ended up with the analysis.
Chris Gotwalt	product, product impurity, starting material, and starting material impurity.
	I would also like to add that this analysis was rather more challenging than I had originally thought it would be. Some of these are things that Clay Barker has made much easier in JMP Pro 17.
	I will call these things out as I go and I want to thank Clay for his responsiveness and dedication to making fit curve easier to use.
	For those of you that are familiar with the functional data explorer, curve DOE works in a similar way. We have a repeated measures response, as we see here on the right of the screen. For each of the response curves, we fit a nonlinear model.
	In this case, a two parameter logistic curve.
	This gives us one location parameter (theta) and one scale parameter estimate (tau) per response curve in the DOE. We then model the extracted thetas and taus as responses using generalized regression.
	I generally use least squares with forward selection, which is the default. As we'll see, there are times when it is advantageous to use other distributions, like the log normal, when the distribution
	of the parameters needs to be strictly positive. What we end up with is an overall model that incorporates both time and our DOE factors for an overall model that predicts the response curve. Here are the four response curves in our data.
	I'm going to start with the analysis of the product response. Once we bring up the data table, we find the fit curve platform under the specialized modeling submenu of the analyze menu in JMP Pro.
	We load up the columns. Product is the response, sample time is the X, or regressor variable, here. Our grouping variable is experiment name.
	And then we have the DOE factors as supplementary variables. Now let's look at the curves of the product response and see
	which of the nonlinear models that are supported by the fit curve platform fit that kind of shape. We can see that the logistic curves fit it reasonably well,
	as well as a couple others. So you go to the red triangle menu. These look like sigmoid curves, so we're going to go there.
	Logistic curves...we're going to choose the logistic three parameter model. If we Alt then right click, we can actually choose a whole bunch of different models and have them be fitted together easily.
	The four parameter logistic model appears to be doing better than any of the other models using the AICc model selection criteria, but note that the R squares for all the models
	tend to be pretty high, so the logistic four parameter model has an R squared of .998, the three parameter logistic is not far behind at .993.
	And the Weibull growth is at .978, so maybe those three are all contenders for for the model that we would use.
	Let's go ahead and take a look at the four parameter logistic. Initially, here we see our overall plot of all the response curves.
	And then we see the individual response curves overlaying on the data points. If we look at the parameter estimates of this four parameter logistic model,
	we see that there are lower asymptotes that are very large negative values.
	And so, this is troublesome because that would indicate that the starting value of the amount of product would be equal to this large negative value, not zero, and so this is in conflict with what we know about the mechanism that's generating the data.
	If we click on curve DOE and take a look at the curve DOE analysis that connects the four parameter logistic parameters to the DOE factors, we immediately see that we're getting
	pretty crazy predictions, and so this might not be a good model for us to use.
	Here are some other examples of nonlinear model fits where it looks initially pretty good, but when you look at the
	evaluations of the prediction formula between the data points, we see some crazy stuff going on, which means that we should probably not use these models for a curve DOE analysis.
	We're going to go ahead and simplify things a bit and take a look at the three parameter logistic model fit.
	So the three parameter logistic model has a initial asymptote of zero, which in this case makes sense for the product data here, because
	the amount of product that we would have at the beginning of the interaction
	should be zero. And as we see, the fit looks pretty good when we look at the overlay of the model and the data points on the right. We proceed to do the curve DOE analysis by going to the red triangle menu
	that brings up the curve DOE analysis and the CDOE profiler. When we look at the actual by predicted plot of the curve DOE analysis, we see
	things look reasonably good along the 45 degree line, but we see these troubling patterns, where it looks like some individual functions are going at a 90 degree angle to the 45 degree line, indicating that we've got some bad fit going on here.
	And when we look at the curve DOE profiler, if we adjust some of the input variables to their extremes,
	we see that there are predictions of product time traces that are actually decreasing in time, which makes absolutely no sense.
	We need to dig in a little bit and figure out what's going on here and how to deal with this. So when we dig into the generalized regression for the individual nonlinear model parameters,
	we have one for the growth rate, one for the inflection point, and one for the asymptote of the nonlinear model.
	In this case, I think that the issue is going to be with the growth rate parameter, because that would control whether or not things can be increasing or decreasing. Drilling down into the generalized regression for growth rate,
	we can request to fit a log normal distribution,
	which will force the growth rate parameter
	to be constrained to be positive, which is what we want. I also want to point out that within the individual generalized regression advanced controls options, we can also turn effect heredity on, which is a sensible option when we're analyzing a curve DOE.
	Clay Barker has gone ahead and set it so that the default is to have effects heredity on by default
	in JMP 17, but we had to do this manually for every response and every parameter in this analysis. Now that we've done our log normal modeling of the growth rate parameter, our CDOE diagnostic plots are automatically updated.
	And we see that those troubling response curves are now following along the 45 degree line much more closely in the curve DOE actual by predicted plot and our predictions in the CDOE profiler make a lot more sense.
	Now that we're much happier with how well our model fits the product curves, we can save our prediction formula and move on to the next response.
	Moving on to the product impurity response, we found that three parameter logistic curves also modeled this reasonably well, though not quite as well as with just the product curves.
	In any case, the analysis was very similar and so I'm just going to move on to the starting materials model.
	Taking a look at the starting material response curves, we see that in contrast to the product and product impurity curves,
	these are monotone decreasing. This is suggesting that an exponential curve or a logistic curve with a negative growth rate would be a reasonable model for this data, so let's take a look at those.
	So in the model comparison report, we see that the exponential model with three parameters does the best, followed by the four parameter logistic using the AICc. And the R squared for these two models is pretty similar.
	However, if we look at the overlay plot of the model fits for the four parameter logistic, it's pretty clear that as time increases for some of these curves, the starting material prediction is going to go negative.
	Whereas for the three parameter exponential, that isn't really happening. So in this case, we're going to go with a three parameter exponential.
	So we're in a situation where the model selection criteria and our common sense when looking at the models
	are suggesting the same thing.
	We proceed to the curve DOE analysis of our three parameter exponential model.
	Like the product model that we did initially, overall the model looks good but, but some of the experiments are getting predictions that go completely the wrong way in our actual by predicted plot.
	And when we play around with the curve DOE profiler, we find that there are combinations of factor settings that are predicting rapidly increasing amounts of starting material in time,
	which doesn't make any sense. The cause for this is because we have fit a normal or least squares type model to the growth rate parameters,
	but all of our growth rate parameters are negative. The problem is that our normal or least squares model cannot force the growth rate parameter to stay negative.
	And unfortunately in GenReg version 16, there is no distribution that will keep the predictions always negative. So in this case,
	I'm going to have to offroad it a little bit to get this model to work, but Clay Barker has added a negative log normal distribution to GenReg
	so that it's easier to model strictly negative data in version 17 Pro.
	So to deal with this, I went ahead and saved the prediction formula for the overall CDOE model, knowing that I was going to have to make a correction to the growth rate parameter.
	I created a data table with the DOE factor data and the growth rates, and then used the formula column to create a new column for negative growth rate. Model
	a log normal distribution in GenReg for this negative growth rate.
	Save the prediction formula, and then I went ahead and did some formula column monging, where I removed the least squares model from fit curve and replaced it with a GenReg model for negative growth rate, and that way,
	I was able to get a model that fits starting material very well. The last of the responses we were modeling was starting material impurity and this one ended up actually being the most difficult one to work with.
	This is because it had a kind of a peak type behavior for a number of the runs.
	And for some of the other ones, it had this monotone increasing type behavior. And so we had a lot of going on with a single response here, and
	none of the models in fit curve were doing a really great job then when we started applying curve DOE to those nonlinear models.
	The best thing we could find were the by exponential models, which led to very high R squares, in as far as the nonlinear modeling component of the analysis went. But when it came time to apply curve DOE,
	we got crazy predictions, on the orders of thousands, when it should have been much, much smaller than that. And these were just not useful models, no matter how hard we tried within fit curve. Fortunately,
	we're able to use the functional data explorer, which is...which uses splines, which are much more flexible. And when we did that,
	the P splines worked very, very well for fitting the curves. And then applying functional DOE analysis to those P splines
	led to models that were really good on the actual by predicted plots. And ultimately we got a functional DOE profiler that did a good job and led to believable results, even at the corners of the design space.
	So we saved the prediction formula and I constructed a profiler for these four responses and was able to send it off to Brian, who's going to talk about what he was able to do with it next.
Brian T	Thank you, Chris, for the demonstration and highlighting the challenges faced along the way. We still have work to do to figure out what is an effective way of selecting the best fit curve model when there are a few equally good candidates.
	Chris arrived at the same logistic three P model for product, product imp, a three P exponential for starting material, and a functional data explorer p spline for the starting material imp.
	Let's take a look at product. There are two good model candidates, so which to choose? Here are some points that arose when I spoke to the subject matter expert.
	The exponential 3P is comparable to first order kinetic equation, but the reaction profile shapes are sigmoidal, suggesting a more complex behavior them first order kinetics.
	The logistic 3P is maybe more appropriate, as it is better approximating the sigmoidal curve feature.
	But in future, try taking more samples early after time zero to see if this helps better discriminate between how good the two formulas are at approximating profile curves between first order and sigmoidal behavior. So it's not straightforward.
	Now for start material imp, where the FDE p spline was fitted. Some material imp can be formed and consumed at different rates, to the extent where one pathway could be switched off.
	The trellis plots show this profile suggesting some experiments just have started material imp formation switched on, and others where starting material imp formation and consumption are on.
	There is, in effect, two curves to combine and express in one formula, and none of the preloaded formulas are capable of doing an adequate job.
	The curve fitting reverts to the FDE approach, so we move further away from a first principle driven curve fit to purely empirical fit to get a good curve approximation model fit.
	This slide highlights the fit curve forming a 3P logistic parameter coefficient default distribution is normal.
	But this is the most...but is this the most appropriate option? There are distributions with better generalized R squared values, so some guidance to think about going forward.
	JMP, in the background, combines those three fitted logistic parameter coefficient models and represents them in the profiler as the DOE original responses as factors change and the reaction progresses over time. One profiler is created for each response.
	Chris has been able to group those four response profilers into a single group profiler plot. This is important because we need to understand what the impact of trying to change factors to improve one response has on the other responses.
	above 95% and product in...below 2%.
	Product and product imp were prioritized responses and effective models were fit. As can be seen for starting material and staring material imp, the models are less effective, delivering implausible negative values at factor levels favorable for prod and prod imp target values.
	What is clear is that the starting material Amine is not influencing any response. It has flat lines so is unimportant in controlling response levels.
	Choosing a high base minimizes the influence of solvent level on reaction performance as it helps to flatten the solvent effects lines.
	In fact, conditions are limited for achieving both product and product imp target criteria, due to a sensitivity to base and catalyst levels.
	However, if we find factor conditions
	that work for those two
	response criteria, then there is a reaction time window that exists. Again, you will see that the sample time profile has a flat line.
	If I show you this in JMP...so Chris has provided the scripts, this is the one that we're interested in.
	What we have here, as I said, is the starting material Amine doesn't have
	an influence on these response numbers here.
	As I change it, those response numbers don't change. So that's quite useful to know that we can choose to set Amine at any level if there's another reason...another quality criteria that we need to for this reaction.
	The base,
	at high level, flattens the solvent black lines.
	As you can see, as we move them to lower base, then we've got steep gradients and curves.
	So by operating at a higher base, then we minimize the influence of solvent.
	Now, if we look...these are the two really interested shaded zones that we're after, and you can see, for both
	catalyst and base, there isn't much of their profiles that sit
	within the shaded area. So it's telling us that there are only certain...small areas of factor levels where we're going to comply with the target criteria. And this is particularly sensitive for base; I think we can only reduce it by about a tenth or two
	until we reach beyond the product imp.
	It's a little bit more
	for catalyst, a couple of tenths.
	But the challenge is changing both at the same time, and the combination is very restricted. But what you do see is, if you find conditions that work, then we have
	a very big time window that the reaction stays stable within the acceptable response criteria, so here ranging from 300 down to
	just less than half an hour.
	Okay, so summary. This talk covers our work to build an analysis workflow. Our experience has brought to our attention improvement opportunities or questions needing answering, which were touched on during our talk above an improvement for the preloaded formulas analysis step.
	Here are the improvements for the curve DOE analysis step.
	So as a final summary, two step analysis approach for DOE analysis of reaction profiles is outlined.
	There is a chance to incorporate subject matter knowledge into the analysis approach.
	The interplay of factors' influence on responses over time is complex and a visual picture is essential to appreciate how their reaction behaves.
	The approach has potential, but it's unfinished. Using JMP's available functionality highlighted a wish list of additional functionality, and Chris discovered some technical areas needing further work.
	On a personal note, the collaboration has been immensely rewarding, and I'd like to thank Chris for all of his efforts supporting the work.
	And finally, I just have a few acknowledgments for some colleagues at
	AstraZeneca. Their work is unseen but they've
	made a great contribution to this talk.
	Thank you.
	that's.
Peter Polito	Perfect Brian great job.
	Okay, all right, how do you feel bad.
Brian T	yeah I think it went well yeah.
	it's probably a bit clunky go trying to work from one.
Peter Polito	that's fine.
	we've all been there no I thought the transitions were good and you spoke very clearly and deliberately and I think it's what you did also it's very cool so.
Brian T	Thank you, no, thank you very much alright well I don't need anything else, I will again I'll let you know that you've received preliminary approval, but need final approval before they're made public and to reach out to you before that happens yeah showing them.
Peter Polito	Anything else.
	You can reach out to me.
	You can reach out to on you.
Brian T	brilliant and do I get a copy of the recording as well it's just.
Peter Polito	um yeah I think probably the easiest thing to do is once it's public is just download a copy, if you want something before then.
	I will send this off to the discovery summit team and they're probably going to do some cropping and.
	Light editing and whatnot and then.
	If you want that version then Tony can probably provide you a link to that.
Brian T	Okay that's perfect right brilliant I will thank you for persevering with getting the.
Peter Polito	yeah sorry about the hiccup.
	I wonder if, tomorrow you will get 10 emails from yourself.
Brian T	Oh, I was always pinging a few around trying to reach out because obviously it's only that I realized this morning.
	Which is in the middle of the night for you so.
	What a tireless talk happens you've only just got to work so.
Peter Polito	yeah no problem and all worked out in the end.
Brian T	yeah that's cool okay right for me okay.
	very much.
	cheers bye bye.