Breadcrumbs

A/B testing

This guidance has been written by the IG product team for their own use, so you may need to consider if the process needs adapting for use in other teams.

Setting up a new A/B test 

To set up a new test, follow the step-by-step instructions set out here: Creating a Split Test in Contentful  

Workflow

  1. Create the B page and split test 

  • Follow steps 1–13 in the guide to build the B page and the split test component.
    Note: The preview link for the B page will no longer work once it’s linked to the split test component. Once you’ve created the split test component in draft, you can preview the test on the A page to toggle between the variants: 

  1. QA and stakeholder approval 

  • Send the A page preview link to QA for testing. 

  • Share this preview link with stakeholders for review and sign off. 

  1. Publish the test 

  • Once the test has been QA’d and approved, continue with steps 14–15 in the guide to publish the B page and split test component. 

  • Let the stakeholders know the test is live and share the split test ID with them so they can track the test in GA4.  

  • You should also let the stakeholders know not to make any tier 1 edits to the A or B variant while the test is running. If they notice a factual error that needs correcting on either variant, ask them to let you know via email.

Ending an A/B test 

Workflow for IG

Kam will advise on the test results once the Individual Giving team and their agency (8Cats) have agreed that a significant result has been reached.   

To end the test, first, archive the B page: 

  1. Open the B page in Contentful 

  1. Archive the component that was changed for the test
    For example – if you created a duplicate hero component to test a new hero image, you should now archive this component. 

  1. Remove all other linked components on the B page (do not archive or delete, as they are used elsewhere) 

  1. Archive the page 

Then, archive the split test component: 

  1. Open the split test component in Contentful 

  1. Remove both linked pages (Variant A and Variant B) 

  1. Archive the split test component 

Note: Once you’ve archived the split test component, check the A page preview to make sure the test is no longer showing up. If you can still see the toolbar for toggling between the two variants, then you may need to ask Sam to rebuild/refresh the preview.

Next steps – dependent on test results  

  • If variant A has outperformed variant B (or results are inconclusive):
    No further action is needed. You’ve already archived the B page and the split test component, and variant A is already the live version of the page.  

  • If variant B has outperformed variant A:
    Update the A page so it reflects the winning version.
    For example, if you A/B tested two hero images, replace the hero image on the A page with the image tested on the B page.