A/B Testing As a Way to Increase Engagement … and Inventiveness

the letters A and B from a children’s book
Detail from Funny Alphabet (ca.1850)

A/B testing (or split testing) is when you simultaneously use two different versions of the same idea to determine which one gets a better engagement rate. Version A versus Version B gets you the A/B in A/B testing.

  • No need to limit to two choices: A/B/C testing is a reasonable venture, but trying any more than that dilutes the test’s effectiveness.

Do you prefer Coca-Cola in a bottle (A) or in a can (B)?

Do you sign up for my newsletter if I prod (A) or I plea (B)?

A. Subscribe to my newsletter

B. Please please subscribe to my newsletter, please!

One of the most common A/B tests for internal communications is for email newsletter subject lines. The idea is to test between two headline ideas on a segment of your population, say 20% of employees, to determine which line garners the most open rates. The winning subject is then dished out to the remaining 80% of recipients.

Mailchimp provides several examples of how their clients learn from A/B testing:

  • What day of the week gets better open rates?
  • Does a subject line with an incentive or a teaser work best?
  • Does including your company name in your subject line increase engagement?
  • Is it better to use your name as the from name, or your company’s name?
  • Does the time of day a campaign is sent affect the click rate?
  • Are recipients more likely to click a linked image or linked text?
  • Do recipients prefer a campaign that contains a GIF or one with static images?
  • What time to send the email
  • Whether to include an emoji in the subject line

“If you double the number of experiments you do per year, you’re going to double your inventiveness.” — Jeff Bezos

A/B testing is one of the easiest ways you can determine which editorial option will resonate most with your employees. You can (and should!) A/B test lots of things, such as:

  • Headlines
  • Lead images
  • Newsletter subject lines
  • Calls to action (e.g., “click here” vs. “learn more”)
  • Colors (e.g., an orange button vs. a red button)
  • Enticements to get bigger crowds (e.g., serving lunch vs. giving out swag)

Tips and Best Practices for A/B Testing

  • Ensure data reliability for the A/B testing solution
  • Conduct an acceptable test before starting
  • Test one variable at a time
  • Conduct one test at a time
  • Adapt number of variations to volume
  • Wait to have a statistical reliability before acting
  • Let tests run long enough
  • Know when to end a test
  • Measure multiple indicators
  • Take note of marketing actions during a test
  • Segment tests

The list above comes from AB Tasty, a very geeky and engrossing website if you’re into A/B testing. (You’re welcome.)

Here’s a case study from Airbnb, which used A/B testing on a customer’s booking flow — from searching to contacting hosts to receiving a reply to actual booking. It’s nerdy, but worth a review.

Lastly, don’t become complacent. Over time your employees will have gotten used to the winning A or B idea. Check in again in six or twelve months with another A/B test to make sure the tactic you’ve been using remains a winning one.




Many internal comms teams don’t have an editorial strategy. I’m here to fix that. Newsletter: https://mistereditorial.substack.com/.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Digital Content Clutter and Fragmentation

5 Must Have Digital Marketing Skills for Small Business Owners

Doing Your Own Digital Marketing: Pros and Cons

Top 5 Reasons Your Ecommerce Website visitors will be Abandoning Orders

5 Easy LinkedIn Hacks for Small-Businesses

Are your leads ready for a call? A quick look at qualifying

DIY Website for Beginners: How to Start Your Business Online | FREE

Three rocks CROs can rely on in a “hybrid” world

Three rocks CROs can rely on in a “hybrid” world

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mister Editorial

Mister Editorial

Many internal comms teams don’t have an editorial strategy. I’m here to fix that. Newsletter: https://mistereditorial.substack.com/.

More from Medium

Design and Intuition: Learning to follow my heart while keeping food on the table

Five Powerful Ways to Engage Your Customers in 2022 — PxidaX

How good storytelling could improve your Product Designin

My Big STupid Outside-In Failure