ParticleSearch Experiments: Test Search and Recommendations Without Guesswork

A useful experiment has one question, a control, one or more variants, an eligible audience, a primary metric, and guardrails. The session remains in a stable variant so the recorded experience is coherent.

Stable assignment is not personalization. The shopper is not being profiled into a supposedly ideal experience. The experiment is creating comparable observations for a defined product decision.

The value of an experiment is its counterfactual. Control estimates what would have happened without the change, while the variant estimates what happened with it. Random assignment helps make those groups comparable before the experience begins. It cannot rescue a test in which eligibility changes after exposure, the storefront delivers the wrong treatment, or a second release changes one path halfway through.

This is why assignment is a design choice rather than an implementation detail. Microsoft Research's pre-experiment guidance notes that the unit should match the product interaction and remain stable, and that network or shared-resource effects can violate a simple user-level comparison. For a merchant, the plain-language question is: can one shopper stay in one coherent experience, and can that shopper's outcome be observed without changing who qualifies?

The comparison-integrity chapter established that assignment alone cannot make a result trustworthy. Before choosing a test question, define the evidence the question must be able to produce: comparable exposure, a response tied to the hypothesis, protected guardrails, and a decision the merchant can take. This contract prevents a convenient dashboard metric from choosing the experiment after the fact.

A control and variant make a comparison more useful when the session assignment, exposure, event definition, and observation window are held steady. They do not remove seasonality, stock changes, campaigns, consent loss, or a broken event stream. Those conditions belong in the result record.

The practical question is not “which card has the larger percentage?” It is “did the variant answer the stated shopper job better, while the protected behaviours remained acceptable?” That is why a merchant should read exposure, response, guardrails, and decision together.

There is no universal “enough traffic” number. A test that tries to detect a large change in a common product action needs less evidence than a test looking for a small change in a rare order outcome. NIST's sample-size guidance makes the dependency explicit: evidence needs are set by the baseline rate, the change worth detecting, the accepted false-alarm risk, and the power to detect that change when it is real.

For merchants, the most important input is the minimum worthwhile effect. It turns a vague hope for “lift” into a business boundary. If a change would not be worth the engineering, operational, or shopper cost unless it improves the chosen outcome by a certain amount, the test must be capable of distinguishing at least that amount from noise.

Read the effect as a range, not a verdict generated by one threshold. The point estimate is the best single estimate from this sample. Its uncertainty range shows which other effects remain compatible with the evidence under the analysis assumptions. A statistically detectable difference can still be too small to matter, while a non-detectable result can still include an important benefit or harm when the sample is weak.

Keep absolute and relative differences together. Moving a rare action from one small rate to another can produce an impressive relative percentage while affecting few shopping journeys. The absolute change expresses reach; the relative change expresses scale against the baseline. Neither should be interpreted without counts and uncertainty.

The experiment earns a decision only when the comparison is valid, the observed effect is large enough to matter, the uncertainty is narrow enough for the choice, and the protected shopping paths remain acceptable. If any of those conditions fail, “no decision yet” is more useful than a winner chosen from an attractive chart.

The shareworthy learning is not that one variant won on one store. It is the mechanism the test clarified: which shopper, in which context, responded to which product decision, at what cost, and with what remaining uncertainty. That record helps the next merchant decision even when the answer is to keep the current experience.