The smoother in the Graph Builder platform is based upon all of the data that is being included within the graph, not just the calculated points such as the mean. As illustrated below, here are 2 Graph Builders, where one has all of the raw data that the smoother is calculated on, and the second is calculated on just the mean values. The Lambda values are the same for both graphs. The first graph appears to be based upon the mean, because with the larger number of data points, the smoother is a better estimator.
So if you want the smoother to be based upon the N, you will need to calculate the N for each of your groupings before you enter into Graph Builder, and then ask for the smoother for those data.
Jim