Closed Bug 1555756 Opened 6 years ago Closed 6 years ago

MoCo_GLOBAL_FF_2019_ACCT_HTML_accounts_ALL_EN_EML (trailhead)

Categories

(Marketing :: Email, task)

Product:
Marketing
Component:
Email
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dkessler, Unassigned)

Details

Description should include:

SFMC folder path & email name
--tbd

--

Intended audience for send (Subscription, Language[s], Country/ies if applicable - otherwise Global, specific engagement segments if applicable - i.e. High/Med/Low/No)

--FX EN ALL

LCM will finalize audience targeting, but if you have recommendations or requests please provide

Target send date (ensure appropriate placeholder[s] is/are already on the Email

6/5/2019

A/B test info if applicable (i.e. subject lines, header images, etc.)

--Subject lines.

Link to your Litmus preview

Link to your Copy Doc
https://docs.google.com/document/d/1YvbHDJCR8DtqsFLmvr8cId3ajgIPfxOQtiBxS_q-SnE/edit#

Hello!

The following email has been built and a TEST sent: MoCo_GLOBAL_FF_2019_NEWS_HTML_NABrand-Promise_ALL_EN_EML - 20190603_144603

The location is: Location Content Builder\Shared Emails\01 User Initiated Sends\Firefox & You\NA Firefox Brand\2019\2019_06\MoCo_GLOBAL_FF_2019_NEWS_HTML_NABrand-Promise_ALL_EN_EML - 20190603_144603

Thank you!

Hi Daniel,

Pending your green light, I'll get the A/B test all setup for tomorrow morning.

Thanks!
Chris

Hi Daniel,

The A/B test is set to launch tomorrow morning at 8 am.

I'll report back email results at the beginning of next week.

Thanks,
Chris

Hi Daniel,

Closing this bug now. The results of the campaign are below, with more detail in the results email I sent. Looking forward to the next one!

Below are the results for the June Fx Brand Newsletter email that was sent Wednesday, 6/5/19. For this email, we did an A/B test on the subject line. I’ve attached a screenshot of the A/B testing overview for more insight.

A/B Testing Insights

For this email, we ran the following subject lines through an A/B test:

Subject Line A: Pay with your privacy no more

Subject Line B: The Firefox Personal Data Promise

Both subject lines had a similar open rate and Subject line A was the winner.

Takeaways:

This email’s subject lines performed well, near the industry average for open rate. The difference between the winning subject line and the non-winner was nominal, which says our privacy promise message is carried well by both subject lines. The CTR performed a bit better for this newsletter send, compared to last month’s. The vast majority of clicks were seen on the “Join Firefox” CTA button.

Thanks,
Chris

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED

SFMC folder path & email name

NA Firefox brand_2019_July

MoCo_Global_FF_2019_NEWS_HTML_NonFxA-JoinCTA_ALL_EN_EML

This bug is for the July NA FX Durable newsletter.

LCM will finalize audience targeting, but intent is Global_EN. Non account holders.

Target send date

6/31/2019

A/B test info if applicable (i.e. subject lines, header images, etc.)

Subject lines.

Link to your Litmus preview

https://litmus.com/checklist/tests/14064702?utm-source=V1.1

Link to your Copy Doc
https://docs.google.com/document/d/1QTAO41sUyzZF531dp8uqjNxUMcr3Na6Vtlw5RCsBIb8/edit

Status: VERIFIED → REOPENED
Resolution: FIXED → ---

Hi Daniel,

I'll be conducting the first round of QA for your recent email. Can you please grant me viewing access to your copy doc? My email is jujensen@mozilla.com. Thank you!

Cheers,

-Juli Jensen

Hi all:

We have updated the header image. I hope we are good to go.

DK

Hi Daniel,

I slacked you some feedback. I think we still have some work to do here before we're ready to finalize this email. Please let us know if you have any questions - hopefully we can get everything ironed out tomorrow.

Thanks,
Lisa

Hi Daniel,

Thank you for getting us the updated header image. We've finished QA and transferring you over to our new template standards, but unfortunately we're still working on getting the Pocket content block setup to incorporate the test we talked about. Please stay tuned for a final proof that we'll be sure to share with you before anything launches, and we'll continue to work to get this setup asap (hopefully in the next day or two.)

Thank you,
Lisa

Hi Daniel,

Final email proof is in your inbox ready for review, along with a recap of our test plan.

Thanks,
Lisa

We got the green light on the new email proof and test plan from Daniel via Slack.

We're moving forward with setting up this send for tomorrow 8/1 to FF subscribers without a Firefox account, and we'll report back on stats next week.

  • We'll be using one subject line: You Deserve Real Privacy (with pre-header: Get the rest of the Firefox family)
  • And doing a 50/50 test of two email versions to gauge click rate: One with the added Pocket articles content block for cross-promotion, and one without it (in our standard newsletter layout format)

Thank you,
Lisa

Hi Daniel,

Here is a snippet of the email Lisa and I sent you regarding your EN Firefox Newsletter that went out on Thursday 8/1/19:

Below are the results of your EN Firefox Newsletter that went out on Thursday 8/1/19, using our updated template design & footer standards.

For this send we conducted an A/B test with one version containing a "bonus" Pocket content block (cross-promoting 3 syndicated articles), and the other without (i.e. our control) in a true 50/50 split, using the same subject line across both & defining the "winner" by highest click rate.

Audience: EN-ALL Firefox subscribers who did not have a Firefox account.

Subject line: You Deserve Real Privacy
Pre-header: Get the rest of the Firefox family

Industry benchmarks*:

  • Open Rate: 17.96%
  • Click Rate: 2.71%
  • Unsubscribe Rate: 0.17%

A/B Test Results:

A: Control (standard newsletter)
Click Rate (CTR): 0.08%

B: WINNER Test (bonus Pocket content)
Click Rate (CTR): 0.92%

Overall Performance
Delivered: 1,995,523
Open Rate: 19.59%
Click Rate (CTR): 0.86%
Unsubscribe Rate: 0.05%

Key Takeaways:

  • The CTR was 15% higher for our Pocket cross-promotion version compared to our standard newsletter send.

  • As for overall clicks, our Control version had 28% of its unique clicks (4,537) on the primary Join CTA button, whereas our Test version with the Pocket content had 21.3% of its unique clicks (4,459) on the button. However, when you look at the total button clicks between the two sends, they only differed by 78 clicks! (Suggesting that the bonus Pocket content did not, in fact, detract much from the email's primary CTA.)

  • As for the Pocket content itself, our bonus block totaled 34.4% of all unique clicks in our Test version, with the most popular article was in the first position "The History of CTRL + ALT + DELETE". It'll be interesting to see if our top/ left-aligned article (on mobile/ desktop respectively) continues to garner the most attention in another layout (i.e. a 2x2 grid test coming up soon.)

  • While both the Open and Unsubscribe rates here performed well against industry benchmarks, the overall CTR was quite low. Previous sends to the full Firefox EN audience for this program (2019 Fx News - EN) this year have averaged a 1.48% clickthrough rate, with the highest CTR (2.70%) seen by the Auto Block campaign sent on 4/10/19. Since this latest send was only to non-Firefox account holders, it can be assumed that our account-holders are typically a more engaged segment and help drive up the CTR when included. Another aspect to consider is how well the message resonates with the audience. The Auto Block campaign provided an immediate value (blocking auto-play videos), whereas here we were asking them to take an action and join the Firefox family/ use more of our services.

  • The other really interesting thing to note here is that the % of total clicks on the Unsubscribe link in the footer varied drastically between the two sends. We saw 11.5% (on the Control version link) vs 5.8% (on the Pocket test link), which is extremely promising and allows us to assume that the additional content did not annoy our readers, but in fact added value.

Please note: We'll be continuing to dig in to downstream results as well (i.e. attributable Firefox account creations, new Pocket users, etc.) and do a deep dive recap of all of our relating Pocket tests and results at a later date once we gather more data.

*These are 2019 industry averages for benchmark comparison--not Mozilla/Firefox averages.

Thank you!

Cheers,

-Juli

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.