'%20clip-path='url(%23canvas-clip)'/%3e%3cg%20clip-path='url(%23canvas-clip)'%3e%3cg%20transform='rotate(0,%20256,%20256)'%3e%3cg%20transform='translate(87%2087)%20scale(14.083333333333334%2014.083333333333334)%20translate(0%200)'%3e%3cpath%20fill='none'%20stroke='%23FFFFFF'%20stroke-linecap='round'%20stroke-linejoin='round'%20stroke-width='2'%20d='m18.064%2010.877l-4.89-7.26c-.42-.625-1.287-.803-1.936-.397a1.4%201.4%200%200%200-.41.397l-4.893%207.26C4.24%2013.715%204.9%2017.318%207.502%2019.423a7.13%207.13%200%200%200%204.168%201.572M15%2018a3%203%200%201%200%206%200a3%203%200%201%200-6%200m5.2%202.2L22%2022'/%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Product Recommendation Analytics: Events, Metrics, Attribution, and Experiments
Product recommendation analytics are trustworthy only when you can connect a specific request to the candidates returned, the candidates actually seen, the action taken, and the purchasable line item that followed.
Start with the event chain, not the dashboard. Define each denominator, preserve the anchor and candidate identity, validate the joins, and separate attributed outcomes from incremental outcomes.
A traceable recommendation path
Request
Response
Visible
Action
Outcome
The request ID connects system output to storefront opportunity. Candidate and variant IDs connect that opportunity to interaction and order lines.
Chapter 1 · Build the event chain
Measure opportunity before measuring persuasion
A click rate is meaningless when the denominator includes candidates that never entered the viewport, excludes empty recommendation requests, or blends several modules with different jobs.
The chain starts before a product card appears. Capture the request and response so a low visible-impression count can be separated into missing requests, empty candidate sets, rendering failures, and below-fold placement.
recommendation_request
A specific context asked for candidates.
Request ID, module, anchor, intent, market, locale, model or rule version
Common gap
A dashboard starts at click, so missing coverage and failed requests are invisible.
recommendation_response
The system returned a candidate set before rendering.
Request ID, candidate IDs, positions, scores or reasons, exclusions, latency
Common gap
Only the final visible cards survive, so retrieval and eligibility cannot be separated.
recommendation_impression
A module or candidate had a real opportunity to be seen.
Request ID, module, candidate, position, visibility rule, timestamp
Common gap
An API response or DOM render is counted as an impression even below the viewport.
recommendation_click or direct_add
The shopper acted on a specific candidate.
Request ID, candidate, variant, position, action, destination
Common gap
The click loses recommendation parameters when a custom card rebuilds the URL.
cart, checkout, purchase, refund
The candidate reached a downstream state under a declared attribution policy.
Line item, variant, quantity, order, value, currency, attribution evidence
Common gap
Order revenue is joined to the session but not to the recommended line item.
Define “visible” before implementation. A response, DOM node, and visible candidate are different states. Record the viewport and duration rule your team uses, then apply it consistently across modules. The exact rule can vary by product and analytics system; the definition cannot remain implicit.
Chapter 2 · Define the event contract
Keep enough context to explain every number
An event name says what happened. The surrounding fields explain which recommendation system, surface, anchor, candidate, shopper context, and purchasable record were involved. Agree on this contract before building reports.
Identity
event_id, occurred_at, session_id, visitor_id or customer_id
Deduplicate events, order them, and connect anonymous and known journeys under an explicit policy.
Recommendation request
request_id, module_id, placement, anchor_type, anchor_id, intent
Connect the candidate to the page state and shopper job that produced it.
Decision system
candidate_source, rule_version, model_version, experiment_id, variant
Explain which logic produced the set and compare changes without relying on deploy dates.
Candidate
product_id, variant_id, position, relationship_reason, eligible_state
Measure exposure and outcome for the actual purchasable record in the actual position.
Storefront context
market, locale, currency, device class, page type, cart state
Diagnose behavior that changes with publication, price, language, layout, or existing cart items.
Outcome
action, order_id, line_item_id, quantity, net sales, margin, refund state
Separate attention from purchase and gross attributed value from retained commercial value.
Use a unique event ID or a stable idempotency key. Browser retries, queue retries, page restoration, and duplicate listeners can otherwise turn one action into several events.
Keep both product and variant identity when the candidate can resolve to a variant. A product-level click attributed to the wrong order-line variant can hide stock, price, or compatibility failures.
Treat model, rule, and experiment versions as event data. A date filter cannot reliably explain which logic produced a candidate when releases overlap, caches persist, or traffic is split.
Chapter 3 · Define the metrics
Write the numerator and denominator beside the metric
“Recommendation conversion rate” can mean purchases per impression, per click, per session with recommendations, or per order. Each definition answers a different question. Use plain names and display the formula in the report.
Keep counts beside rates. Counts expose reach and data loss. Rates compare efficiency under a stable definition. Neither is sufficient on its own.
| Metric | Numerator | Denominator | It answers | It does not answer |
|---|---|---|---|---|
| Request coverage | Eligible contexts that created a recommendation request | Contexts where the module was intended to run | Is the surface wired and eligible to ask for recommendations? | Whether the candidates are relevant |
| Non-empty response rate | Requests with at least one eligible candidate | Recommendation requests | Can retrieval and eligibility produce a usable set? | Whether the shopper saw the module |
| Visible impression rate | Rendered modules or candidates that met the visibility rule | Non-empty responses | Did returned candidates receive a real opportunity to be seen? | Whether the relationship was persuasive |
| Click-through rate | Visible impressions with a recommendation click | Visible recommendation impressions | Did the relationship and presentation earn an interaction? | Whether the candidate survived product evaluation |
| Post-click add rate | Recommendation clicks followed by an attributed candidate add | Recommendation clicks | Did the destination confirm fit, price, availability, and options? | Whether the order completed |
| Attributed purchase rate | Attributed candidate purchases | The declared exposure or click base | How often did the measured recommendation path reach purchase? | How many purchases were caused by the recommendation |
| Attributed value | Net sales, units, or margin attached to attributed line items | Optional: exposure, session, order, or module | What commercial value is associated with the declared path? | Incremental lift without a valid counterfactual |
Metric definition template
Name and decision
What do we call the metric, and which decision is it allowed to influence?
Population and formula
What is included, what is excluded, and what are the numerator and denominator?
Identity and time
Is this candidate, module, session, visitor, order, or line-item based, and which clock assigns the period?
Attribution and caveat
Which interaction qualifies, how long does credit persist, and what can the metric not prove?
Chapter 4 · Understand Shopify’s reporting boundary
Use the native reports, then document what your implementation adds
As checked on July 28, 2026, Shopify’s Search & Discovery app exposes recommendation click rate, purchase rate, and a low-engagement report for recommendations on top-selling products. The app view uses the last 30 days; Shopify directs merchants to Analytics reports for other date ranges.
Shopify’s Ajax Product Recommendations API also returns product URLs with recommendation tracking parameters. If a custom theme or storefront reconstructs those URLs and drops the parameters, Shopify’s native reporting path may lose the context it expects.
Those reports are useful for storefront performance review. They do not replace a request, response, visibility, eligibility, model-version, or controlled-experiment dataset when your team needs to diagnose the recommendation system itself.
Chapter 5 · Validate data quality
Prove the events and joins before interpreting performance
A polished dashboard can faithfully visualize broken instrumentation. Run data-contract tests on the raw or modeled events before deciding that a recommendation, placement, or model is weak.
Uniqueness
Test
Each event_id is unique, or the pipeline has a documented idempotency key.
Failure signal
Retries create extra impressions, clicks, or purchases.
Referential integrity
Test
Every candidate impression and click maps to a known request and candidate.
Failure signal
Outcome reports contain orphan actions that cannot be explained.
Count ordering
Test
Within one definition, downstream unique sessions or candidates do not exceed upstream opportunity.
Failure signal
The denominator, identity, time window, or join differs between stages.
Timestamp order
Test
Request precedes response, impression precedes click, and purchase follows the qualifying interaction.
Failure signal
Client clocks, delayed delivery, or attribution logic corrupts sequence.
Candidate identity
Test
Product and variant IDs remain stable from response through order line.
Failure signal
A product-level recommendation is credited for a different or unavailable variant.
Context continuity
Test
Placement, anchor, intent, market, and experiment survive into interaction events.
Failure signal
One blended rate hides which module or system changed.
Late outcomes
Test
The pipeline updates prior cohorts when orders, cancellations, or refunds arrive later.
Failure signal
Recent periods appear artificially weak or gross value never reconciles.
A count-ordering violation is a diagnostic signal, not automatic proof of duplicate data. Different units, time windows, attribution windows, late events, or identity policies can also make downstream counts appear larger. Compare like with like before repairing the pipeline.
Chapter 6 · Diagnose the first weak handoff
Turn the funnel into a repair queue
Start with the earliest stage that does not behave as expected. A low purchase count does not justify changing the ranking model when the module rarely enters the viewport or the click loses candidate identity.
| Observed pattern | Likely layer | Inspect first |
|---|---|---|
| Expected contexts but few requests | Theme, route, eligibility trigger, consent, or instrumentation | Compare intended placements with observed request events. |
| Requests but many empty responses | Candidate retrieval, publication, stock, exclusions, cold start, or intent | Inspect returned and excluded candidate IDs for one anchor. |
| Responses but few visible impressions | Rendering, lazy loading, below-fold placement, layout, or visibility logic | Replay one response in the actual viewport and verify the impression threshold. |
| Impressions but few clicks | Relationship, heading, order, card data, repetition, or placement | Judge the first visible set in context before changing the model. |
| Clicks but few candidate adds | Price, availability, variant choice, compatibility, destination, or intent mismatch | Follow a known candidate from card to purchasable state. |
| Adds but few purchases | Checkout friction, delivery, price, cart conflict, or attribution join | Separate business drop-off from missing purchase linkage. |
| Attributed value rises but the store outcome does not | Channel substitution, broad attribution, cannibalization, or mix shift | Run or review a controlled comparison with business guardrails. |
Open the storefront while reviewing the queue. The report identifies where to inspect; the visible product relationship and trace reveal what to change. For missing or empty native modules, use the Shopify recommendation diagnostic.
For weak engagement on a working module, compare the recommendation job, candidate source, eligibility, relationship reason, card data, and page position using the placement guide.
Chapter 7 · Segment without losing the question
Break out dimensions that can change the decision
A store-wide recommendation average can show direction, but it cannot tell you whether a cart add-on, product-page comparison, or search-recovery module needs repair. Segment when the dimension maps to a different mechanism or action.
Placement and module
Product page, search recovery, cart, and post-purchase solve different jobs.
Boundary: Do not compare their rates as though they share the same opportunity.
Recommendation intent
Related, complementary, substitute, bundle, and replenishment have different actions.
Boundary: A click may be appropriate for comparison while direct add suits a simple accessory.
Anchor and category
Coverage, catalog evidence, and product economics vary by anchor.
Boundary: High-volume anchors can dominate a store-wide average.
Candidate source or version
Manual, rules, behavioral, and model candidates should be auditable separately.
Boundary: A release-date split is weaker than an event-level version.
Market, locale, and device
Eligibility, price, language, viewport, and interaction can change the path.
Boundary: Do not create tiny cuts that cannot support a stable decision.
New, returning, or known shopper
Available context and purchase history change the recommendation problem.
Boundary: Respect consent and avoid treating identity stitching as certain when it is not.
Create a repair queue with anchor, module, intent, source or version, failing stage, evidence, owner, and next test. Sort by decision value and reliable opportunity, not by the noisiest percentage.
Chapter 8 · Separate attribution from causality
Attributed revenue describes a rule; incremental lift needs a counterfactual
An attribution policy decides which recommendation interaction receives credit for an order. It is useful for journey analysis and operating reports, but another reasonable policy can assign different credit to the same order.
Write the qualifying interaction, window, scope, identity rule, order-line rule, and treatment of returns beside the metric. Distinguish the recommended product being purchased from any product being purchased after a recommendation interaction.
Causality asks a different question: what would have happened without this recommendation experience? Answer that with a valid holdout or controlled alternative, not a wider attribution window.
Attribution report
Which measured path receives credit?
- Define the qualifying impression or click.
- Define the time and identity boundary.
- Define candidate, line-item, and order scope.
- Reconcile cancellations, refunds, and late outcomes.
Incrementality study
What changed because of the experience?
- Create a credible treatment and comparison.
- Assign before exposure and preserve assignment.
- Measure a business outcome and experience guardrails.
- Report effect, uncertainty, integrity, and limitations.
Chapter 9 · Design the experiment
Test a recommendation decision, not a dashboard color
A useful experiment changes a named layer: candidate generation, eligibility, ranking, set constraints, placement, heading, card data, or action. The hypothesis should explain why that layer causes the observed failure.
Assign the experiment before recommendation exposure and record the assignment even when the module is empty. Excluding empty responses after assignment makes a treatment with higher coverage look different from the population it actually affected.
Experiment design checklist
- 1 State the decision and hypothesis before looking at outcomes.
- 2 Assign the experimental unit before recommendation exposure and keep it stable.
- 3 Record the assignment even when the module returns no candidates.
- 4 Change the layer named by the hypothesis: placement, candidate source, ranking, set constraints, card, or heading.
- 5 Choose one primary business outcome and a small set of experience guardrails.
- 6 Estimate the required sample from the baseline, minimum useful effect, allocation, and error tolerance.
- 7 Run long enough to cover the relevant shopping cycle and known calendar effects.
- 8 Inspect treatment integrity, exposure, contamination, missing events, and segment balance.
- 9 Report uncertainty and practical effect, not only whether a threshold was crossed.
- 10 Keep a decision log so a later model or merchandising change does not erase the lesson.
Do not copy a universal sample-size threshold. Required sample depends on the baseline, experimental unit, minimum effect worth acting on, allocation, variance, and accepted error. Estimate it for the primary metric, then state what the available sample can and cannot resolve.
Chapter 10 · Build a decision dashboard
Show system health, shopper response, and the repair queue
A dashboard should reveal what changed, where it changed, and what to inspect next. Use a compact overview backed by traceable detail rather than one blended revenue tile.
Recommendation health
Filters: period, placement, intent, source, version, market, locale, device
Coverage
Requests and non-empty responses
Opportunity
Visible module and candidate impressions
Response
Clicks, direct adds, and destination behavior
Outcome
Orders, net value, margin, and refunds
Stage trend
Schematic only. Use real counts from one cohort and one definition when implementing this dashboard.
Repair queue
Chapter 11 · Run the analytics audit
Trace one candidate before scaling the dashboard
One complete trace exposes more measurement risk than another layer of aggregate charts. Use a controlled context and save the event evidence at each handoff.
- 1
Choose one module and one anchor
Fix the placement, page state, market, locale, device, cart state, and expected candidate relationship.
Output: A reproducible recommendation context.
- 2
Capture request and response
Record request ID, candidate IDs, positions, intent, source or version, exclusions, and response time.
Output: Proof of what the decision system produced.
- 3
Verify a visible impression
Confirm the module and candidate meet the declared visibility rule instead of counting only a DOM render.
Output: One trustworthy opportunity-to-see event.
- 4
Follow one interaction
Click or directly add one candidate and confirm identity, context, and tracking survive.
Output: A connected request, impression, and interaction.
- 5
Trace the candidate to the order layer
Verify the candidate variant and line item can be joined under the chosen attribution policy.
Output: A documented path from candidate to commercial outcome.
- 6
Run integrity checks
Test uniqueness, joins, timestamps, count ordering, context continuity, and late outcomes.
Output: A list of data defects separated from experience defects.
- 7
Build the first repair queue
Segment by placement, intent, anchor, source, market, locale, and device only as far as the data supports.
Output: Prioritized cases with a failure layer, owner, and next test.
The output is not “analytics installed.” It is a verified trace, a metric dictionary, a data-quality report, a segmented repair queue, and a declared attribution or experiment policy.
Build the measurement system that can explain the recommendation system
A useful report preserves the path from decision context to commercial outcome. It tells you whether the system failed to produce candidates, the page failed to expose them, the relationship failed to earn action, or the measurement path failed to connect the action.
ParticleSearch carries this context through the recommendation surface. It records whether a shelf produced a visible set, returned empty, or failed; the strategy and placement in use; the product interaction; and the assigned experiment variant when a controlled comparison is running. That lets the merchant distinguish weak candidate quality from missing exposure or a broken delivery path.
The dashboard keeps recommendation evidence separate from search evidence, then connects lineage-verified recommendation touchpoints to Shopify-verified orders when the required evidence exists. Unattributed orders remain unattributed, and an attributed order is not presented as proof that the recommendation caused the purchase.
After a verified launch, the merchant should not need to reconstruct the shelf strategy from theme markup, infer exposure from clicks, or join a recommendation report to order exports by hand. ParticleSearch still needs enough traffic and a valid event path, and the merchant still chooses the business metric and guardrails. Missing evidence is reported as missing evidence, not as zero value.
For the wider strategy, continue with the ecommerce product recommendations guide. For the product-specific evidence model, read the ParticleSearch revenue-attribution guide.