When the best setting is outside the region

A fitted surface returns a stationary point. The experiment was run over a square with sides at ±1, which is where the equipment reaches and the process is stable, and the point that comes back is at a radius of 2.29.

It is not a recommendation. It is a place the model has been extrapolated to — the quadratic was fitted over a square of side two and is being read at two and a quarter times that square’s own radius — and it happens often.

The shape is a familiar one, and it has two close relatives. A bias-corrected persistence leaves the stationary region on a third of series and something has to be done about it; a variance estimated by subtraction returns its own boundary on a third of datasets. Here a fitted optimum leaves the region a procedure can reach on a quarter of studies. In all three the output is a well-formed number that is not a value the quantity can take, and in all three the handling is undocumented.

How often #

The stationary point is $-\tfrac12\hat{\mathbf B}^{-1}\hat{\mathbf g}$, a ratio whose denominator is an estimated curvature, so it wanders exactly where the curvature is hard to read — which is the flat surfaces, which are the interesting ones.

A quarter of studies at $\sigma = 2$, and one in eight producing a recommendation more than twice the region’s own radius away. Those are not pathological runs: the truth has a well-defined interior optimum and the design is the standard thirteen-run composite.

And the two rates are different questions. The stationary point being outside is a fact about the fit. The best point inside the region sitting on the boundary is the operational consequence, and it happens nearly twice as often — because it also happens whenever the fit reads a saddle, where there is no interior maximum to find.

The ridge #

What replaces a stationary point that cannot be run is the path of best settings at each radius, and it has a closed form.

Maximising $\hat b_0 + \hat{\mathbf g}'\mathbf x + \mathbf x'\hat{\mathbf B}\mathbf x$ subject to $|\mathbf x| = r$ gives the Lagrange condition

$$(\hat{\mathbf B} - \mu\mathbf I)\,\mathbf x = -\tfrac12\hat{\mathbf g}$$

and the family of solutions as $\mu$ varies is the ridge. Each $\mu$ gives one point and one radius; the radius falls as $\mu$ rises, so solving for a stated radius is a bisection in one variable.

Which $\mu$ matters and it is where this essay’s arithmetic was wrong first. For the maximiser on each sphere, $\hat{\mathbf B} - \mu\mathbf I$ has to be negative semidefinite, so $\mu$ sits above the largest eigenvalue. The branch below the smallest eigenvalue is the minimiser, and a ridge built on it walks downhill: at a radius of one it reaches 54.31 where the maximum is 61.69.

That error was caught by a stated requirement rather than by inspection: a ridge has to rise along itself out to the stationary point’s radius, and the first version did not.

Two routes #

A Lagrange solve is an algebraic claim about a maximisation nobody performed, so it is checked against one.

At each of four radii the ridge point is compared with the best of three thousand six hundred evaluations around the circle. The two agree to 2.4 × 10⁻⁶, sharing only the fitted coefficients: one route inverts a two-by-two matrix at a bisected $\mu$, the other evaluates a quadratic at every tenth of a degree.

The refusal beside it is the wrong branch, and it is required to fail — the minimiser must be more than half a unit below the maximiser. A check in which both branches agreed would be a check that had never been shown the case it exists to catch.

The bisection deserves one sentence about its bracket, because the bracket is where a solve like this usually goes wrong. The radius is monotone in $\mu$ on the branch above the largest eigenvalue — falling from infinity at the eigenvalue itself to zero as $\mu$ grows — so a bisection needs an upper end where the radius is already below the target. That end is found by doubling rather than assumed, which is the same lesson two routes to every number draws from a fixed bracket that returned its own endpoint instead of failing.

What to recommend #

Two rules are available once the ridge exists and they differ in one clause.

Report the stationary point, pulled back onto the boundary if it left the region. Crude: it takes the direction the fit points in and stops at the edge.

Report the maximum of the fitted surface over the region. The textbook answer: if the stationary point is inside and is a maximum, use it; otherwise the maximum is on the boundary and the ridge at the region’s radius finds it.

That is a negative result and it is the useful one. The careful rule requires an eigendecomposition, a branch choice, a bisection and a case split; the crude rule requires dividing a vector by its own length. They differ by four per cent of a quantity that is itself under one per cent of the response.

The difficulty is not which rule. It is that the fit is pointing in a direction that is only roughly right, and neither rule can improve on the direction it was given.

Why the two rules agree #

The four per cent is worth explaining rather than only reporting, because “the crude rule is nearly as good” is the kind of claim that is usually an artefact.

The two rules differ only on the studies where the fitted stationary point is outside the region and the fit reads a maximum. On those, the crude rule takes the direction from the centre to the stationary point and stops at the boundary; the careful rule takes the ridge to the boundary. Those two boundary points are close, because the ridge bends towards the stationary point as the radius grows and by the region’s edge it is most of the way there.

On every other study the two rules agree exactly: where the stationary point is inside and is a maximum, both report it, and where the fit reads a saddle the crude rule projects a point that is not a maximum of anything while the careful rule takes the boundary maximum — which is where the ordering between them comes from and why it changes with the noise.

So the four per cent is a real comparison of two genuinely different rules that happen to disagree on a minority of studies and disagree mildly there. It is not that the careful rule is unnecessary; it is that the direction the fit points in is doing almost all of the work, and both rules inherit it.

What the ridge is actually for #

The rules above use the ridge at one radius. Its value is that it is a path, and the path answers a question the point cannot.

An experimenter reading the ridge sees the fitted response at every radius from the centre out to the edge and can see where the gain stops being worth the move:

Moving from the centre to radius 0.52 buys 1.07 of response. The next half a unit buys 0.67, and the last four tenths buy 0.39. If the settings at larger radius are more expensive, less stable or further from a known operating point, the ridge prices the trade and a point recommendation does not.

That is the difference between a recommendation and a curve. A stationary point says “go here” and the ridge says “here is what each distance is worth”, and only the second survives the fit being roughly right rather than right.

Where this does not happen #

The rates above are measured on a truth that is genuinely flat in one direction, and the contrast with a well-curved one says how much of the difficulty is the surface rather than the method.

With eigenvalues of −3 and −2.5 — a clean bowl, curved in every direction — the fitted stationary point leaves the region on 2.1% of studies at the same noise, against 24.9% on the flat truth. The whole phenomenon is a phenomenon of flat surfaces.

That is not reassuring, because flat is the interesting case. A surface with strong curvature in every direction has an obvious optimum that a coarse search would find; a surface with a flat direction is where the experiment has something to say that intuition does not, and it is exactly where the fit’s stationary point wanders off. The method is least reliable on the problems it was built for, which is the shape the hierarchical field found for its own estimator and which recurs whenever a procedure’s difficulty is division by a small estimated quantity.

Why the fit points the wrong way #

The direction the ridge sets off in is the gradient at the centre, and how wrong that is has already been measured in this field.

Walking up the gradient computes the angle between the fitted gradient and the true one in closed form — $\sigma/(|\beta|\sqrt N)$ — and what that angle costs as its squared cosine, twelve per cent at twenty degrees. The ridge inherits the same error, because it starts from the same gradient, and adds a second one: the curvature that bends the path is estimated too.

So the ridge’s accuracy is bounded by the steepest-ascent field’s arithmetic, and the boundary points it recommends are boundary points chosen by a direction with a known error. The shortfalls in the cost figure — 0.27 at $\sigma = 1$ rising to 0.75 at $\sigma = 3$ — are that error arriving in response units.

The ridge is right about the fit and the fit is wrong about the truth #

One more number belongs in the account and it is deflating, which is why it is here rather than in a footnote.

On the study drawn in the first figure, the ridge ends at (0.30, 1.36) and the fit predicts 62.05 there. The truth at that setting is 59.92, and the true optimum — at (0.4, 0.3), inside the region — is worth 60.73. So the recommended setting is worse than the true optimum by 0.81, and the fit was expecting it to be better by 1.32.

Everything about the ridge is correct. It is exactly the maximum of the fitted surface on each sphere, verified against a walk, and it rises along itself as it should. What it is a ridge of is a surface estimated from thirteen runs at twice the noise, and on that study the estimate points in a direction the truth does not go.

A correct answer to the wrong question is what most of this field’s failures look like, and it is the reason the cost figure is measured against the truth rather than against the fit. An analysis that checked its own ridge against its own surface would pass every test in this essay and recommend a setting that gives up more than a unit of response.

What an experimenter should do with all of this #

Four statements, in the order they are useful.

Check whether the stationary point is in the region before reporting it. It is one comparison of a radius against a bound and it fails on a quarter of studies at twice the noise. A report that quotes a stationary point outside the region has extrapolated a quadratic fitted over ±1 to a radius of two or more, and quadratics extrapolate badly by construction.

Use the crude rule if a single setting is wanted. It is within four per cent of the careful one and it needs no eigendecomposition.

Report the ridge if anything more than a setting is wanted. It is the object that prices the move, it costs a bisection at each of twenty-five radii, and it is what a reader deciding how far to shift an operating point actually needs.

And treat all of it as a direction rather than a destination. The recommended setting on the study drawn here gives up 0.81 of response against the true optimum, and that is not fixed by any of the rules above. The response to a fit that points roughly in the right direction is another experiment centred where it points, which is what the steepest-ascent field does with a first-order fit and is the same discipline one order up.

What is claimed here, and what is not #

The claim is what to do when a fitted optimum is outside the region: that on a flat truth with an interior optimum the fitted stationary point is outside on 24.9% of studies at twice the noise and more than three units out on 11.8%; that the best point inside the region is on its boundary on 47.3%; that the ridge maximises the fitted surface on each sphere, checked against a walk to 2.4 × 10⁻⁶, on the branch above the largest eigenvalue rather than below the smallest; and that the two rules for using it differ by four per cent of a shortfall that is itself under one per cent of the response.

Every rate is two thousand studies with the same thirteen-run design. The truth is built with stated eigenvalues of −3 and −0.4 so that it is genuinely flat in one direction, which is the case at issue; on a well-curved truth the stationary point rarely leaves the region at all.

What stays out: the sampling distribution of the ridge itself, which would need an interval at each radius and is the natural next measurement; ridges in three and more factors, where the arithmetic is identical and the picture is not; the rising-ridge case, where the fitted surface has a near-zero eigenvalue and a gradient along it, so the ridge runs to infinity and the region’s boundary is the only thing stopping it; and constraints that are not a sphere — a box, a simplex, a cost budget — where the Lagrange condition is replaced by a general constrained optimisation and the closed form is gone.

Still open: the run that confirms it #

Every setting recommended in this essay is a setting the fitted surface says is good, and the fit predicts a response there. The obvious next step, and the one every response-surface text recommends, is a confirmation run: go to the recommended setting and measure it.

What that run will find is not what the fit predicts, and the gap is not noise. The setting was chosen because the fitted surface was highest there, so the fitted height at it is a maximum over a random field and is biased upwards — by 0.858 at twice the noise, which is 0.72 of the prediction’s own standard error. That is what the confirming run finds.

The check, and the refusal #

Three claims are gated. That the ridge is defined at every radius out to the region’s edge, which would fail on a fit whose curvature made the solve singular. That the fitted response rises along the ridge to the stationary point’s own radius and falls beyond it — unimodal rather than rising, because the rising version held on the frames where the stationary point was outside and failed where it was inside — a claim about one setting dressed as a claim about the family. And that the peak, where it is inside the walked range, is at that radius.

The refusal is the branch: the Lagrange solution below the smallest eigenvalue must be the minimum on the sphere, more than half a unit below the maximum. It is 54.31 against 61.69. If the two branches agreed, the essay’s whole construction would be indifferent to a choice it makes deliberately, and the first version of the arithmetic — which took the wrong one — would have produced a figure nobody could have told was wrong.