Skip to main content

Decision Framework

A Decision Framework for experiments allows teams to standardize their interpretation of results and decision-making process. Once configured, it provides clear recommendations for which group to ship based on experimental outcomes. While the framework highlights recommended actions based on metric movements, it does not enforce any actions.

Where to add Decision Frameworks​

  • Decision Frameworks can be added to an experiment template. This framework will apply to all experiments created from that template
  • Decision Frameworks can be added to individual experiments ad-hoc by pressing the three-dot menu
    • Requiring a decision framework for experiments is a setting that can be configured at the project-level. This can be useful to enforce that experimenters are thoughtfully selecting decision criteria before running an experiment.

Setup​

To configure a Decision Framework, add it from your experiment, or find your desired experiment template under Settings → Templates, then navigate to the Decision Framework tab. By selecting one primary metric and one or more guardrail metrics, you can choose a recommended action based on metric performance.

Team-level Templates Settings

Three different types of recommendations can be configured:

  • Rollout Winning Group: An icon appears next to the winning group in the Make Decision button
  • Discuss: A message appears on the experiment page recommending discussion before making a ship decision
  • Do Not Roll Out: An icon appears next to the control group in the Make Decision button

recommendation

Reviews for shipping negative results​

Reviewers can be set up when a shipping decision doesn't align with the recommendations configured in the decision framework. All it takes is to toggle on "Require reviews when decision opposes recommended decision" and add reviewers in the following dialogue box.

recommendation

Once set up, a reviewer will be required in the following situations:

  • Rollout Winning Group: Shipping a treatment group that is not recommended
  • Discuss: Shipping any treatment group
  • Do Not Roll Out: No review will be required

recoomendation