Closed Bug 1461416 Opened 7 years ago Closed 7 years ago

CRM/Email Marketing Request: IRL Season 3 Emails

Categories

(Marketing :: Email, task)

Product:
Marketing
Component:
Email
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mthayer, Assigned: jkleinsasser)

Details

>> Mozilla Team/Department Mozilla Conscious Choosers Durable Team >> What kind of relationships are you looking to develop? A relationship with Mozilla supporters and Firefox users who are interested in taking back their power online and learning how to do so by listening to IRL Season 3. >> How will you nurture these relationships? Email messages to those who are interested (as decided through list segmentation). >> How will success be measured? Podcast downloads and subscribers. >> What is the risk to not doing this? Less conscious choosers listening to IRL > less conscious choosers understanding Mozilla > less conscious choosers choosing our products and services that make the Web better. >> What's the timeline for launch? Episode 1 email launch: July 3 Episode 2 email launch: July 17 Episode 3 email launch: July 31 Episode 4 email launch: August 14 Episode 5 email launch: August 28 Episode 6 email launch: September 12 Episode 7 email launch: September 25 >> Any other info we should know? Season 3 email copy document: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit#
Assignee: nobody → lwright
Thanks for filing, Michaela! Placeholders for all of the above target send dates have now been added to our Email Comms Calendar.
Assignee: lwright → nobody
Thanks, Michaela! When can we expect copy for the first email to launch next Tuesday (3rd)? Looking for copy here: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit Info on process here: https://docs.google.com/document/d/1tH6s2utEwORV-l2fzX-5Kznt23PBVqsCXX13k4kjV48/edit Steve will be leading on this. Thanks!
Assignee: nobody → ssegovia
Flags: needinfo?(mthayer)
Yep! Indeed. I'll have the copy in the copy doc by tomorrow.
Flags: needinfo?(mthayer)
Hi folks – just a quick update. We'd love to send out a German localized email as well with this send. We'll give you copy for both the EN and DE versions tomorrow. If the timeline needs to shift for the German email to go out, no worries. We're curious if the German version will perform well. If it does, we'll continue with localized IRL emails. If it doesn't, we won't. (Note: the top listened-to podcasts in Germany are in English, so we hypothesize this may be a fit.)
Hi Jess and Steve, Good morning! Copy is all ready to go for the IRL Episode 1 email send: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit?ts=5b31ee59. In the doc, you'll see that: - We'd like to do an a/b test with copy (rather than subject lines) for the English send. And similarly; - We'd like to do an a/b test with copy (rather than subject lines) for the German send. For English, we're hoping to learn if the "we / collective" or the "you / individual" copy perspective is best for our readers. And for German, we're hoping to learn if readers prefer clicking to an English podcast or a German blog. Let me know if you have any questions, and thank you! Michaela
Thanks Michaela! I love the testing ideas and the expansion to Germany! This is bigger scope than our usual 1 email to 2 different audiences (Mozilla & Firefox) with easy subject line tests. (To do copy test, we have to code 2 different emails. This request would mean coding 8 emails, and setting up 4 a/b tests by Tuesday.) Here's what we can support: English sends on Tuesday, 7/3 - a/b copy test to Firefox audience (You'll get better testing results with the larger list) - 1 copy send to the Mozilla audience (you can pick the version you want to use, or wait until we get results from the Firefox a/b test - let us know). German in future. We *might* be able to pull into this sprint. We'll keep you posted as the sprint progresses on timing. Let me know if you have any questions and please confirm which version you want to send to the Mozilla audience (either pick ahead, or wait for Firefox winner). Thanks!
Flags: needinfo?(mthayer)
Thanks, Jess! We don't need the Firefox and Mozilla sends to be on different dates. They can even be together the same send (though, not sure that's technically possible). Does that make a difference? And, yes. We'll stay tuned on the German send. For future episodes, we are hoping to launch a German send with every English one (so long as this first one performs). Possible to get that on the calendar as tentative as we wait to see how our German audience responds to the first message? Michaela
Flags: needinfo?(mthayer)
Happy to be of help in this! :D The Firefox & Mozilla audiences will go the same day, but they are 2 different branded emails (one with "Mozilla" wordmark at the top - and the other with "Firefox" wordmark). So we can't send them all with the same send. Our goal is to code & launch 3 English emails for this sprint: 1) Firefox branded content version A 2) Firefox branded content version B 3) Mozilla branded content version <you decide or we pick from the early-results winner on Tuesday am PT time> Thanks for the heads up with German! We'll wait until we can send the first email, see how it goes, and then we can put them on the cal if it looks like something our teams can prioritize together and make happen. Thanks! Jess
Sounds good.
Hi all, I'm sharing the results for the IRL emails that went out this Tuesday, 7/3. Although we see that the Auto winner selected was "We" copy selected by SFMC within 2-3hrs of the send, but the "You" copy delivered slightly higher overall clicks in the long-run. I recommend that the "You" copy is the true winner here. Send time: 7/3/18 @ 7am-9am PST Industry average goals* Mozilla: - Open: 25.25% - Click: 2.84% Firefox: - Open: 21.56% - Click: 2.40% Firefox - EN - ALL - HIGH A/B Copy Test: A) We Copy (6.05% click) - AUTO WINNER B) You Copy (6.18% click) - OVERALL WINNER 43.68% overall open 5.95% overall unique click 116,621 total sent Mozilla - EN - IRL - ALL A/B Copy Test: A) We Copy(1.03% click) - AUTO WINNER B) You Copy (1.16% click) - OVERALL WINNER 47.09% overall open 1.06% overall unique click 227,355 total sent *These are industry averages (2015)--not Mozilla/Firefox averages--for comparable ongoing series sends.
I'm missing some vocabulary... apologies. Can you explain how to think of what "auto" means in this regard. And can you share what the open rates were as well on those? Just curious about that. Thanks, Steve!
Hi Brad, I can provide context. I set Marketing Cloud (SFMC) to automatically determine a winner 2 hours after the first send to the A/B test audience. Once SFMC auto-declares a winner with the best Click Through Rate, it sends it to the rest of the audience. In this case, the initial results from SFMC declared that the "We" copy had a higher click rate. Upon pulling the final results today, I see that the "You" copy drove the overall higher click though rate. Please let me know if there's anything else I can answer. Thanks!
Thank you Steve!
Hey Michaela! Is the gameplan for you to post the next IRL content in the doc the Tuesday before the following week's launch? ie - are you putting copy for S3 E2 here tomorrow? Thanks!
Flags: needinfo?(mthayer)
Yes ma'am! English copy is already in there. Anja is adding German copy in the morning. When it's all in there, I'll ping in the bug tomorrow.
Flags: needinfo?(mthayer)
Hi Jess and Steve, EN and DE copy for the IRL Episode 2 email is here: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit#heading=h.4mqw0nh84eqm. Let me know if you have questions! Michaela
Thanks, Michaela! Here's the gameplan: This week: Sending S3, E1 email to German *Firefox* subscribers, and excluding our Mozilla audience. (There are some German email calendar considerations happening for our Mozilla audience.) Steve will comment the actual day it's scheduled for when it's known. Next week: Tuesday, 7/17 - S3, E2 email to our regular crew of English Mozilla and Firefox subscribers who get IRL content. If the German test goes well for the Firefox audience, we'll plan to send the S3, E1 email to German-speaking Mozilla audience next sprint, likely on 7/31. Holler if you have ?s. Thanks!
This all sounds wonderful!
Hi there. That sounds amazing. I just wanted to triple check, that the link in Copy Test B in the German email is correct and working: It's https://mzl.la/S3E1 Will we get a proof before the email goes out to German audiences? Thanks so much, Anja
Jess and Steve — can you let us know the size of the German Firefox and Mozilla lists?
Hi Michaela, Below is the list size for the Mozilla and Firefox DE Audiences: ------ Firefox DE ALL: 493.3k Mozilla DE ALL: 56K
(In reply to Steve Segovia from comment #21) > Hi Michaela, > > Below is the list size for the Mozilla and Firefox DE Audiences: > > ------ > Firefox DE ALL: 493.3k > > Mozilla DE ALL: 56K Thank you!
Hi Michaela, I notice that the landing page for Episode 2 is not yet live. When will that be live? http://smarturl.it/irl_s03e02?IQid=email
On Monday when the episode releases!
Perfect! Thanks!
Hi all, I'm sharing the results for the DE IRL emails that went out this Tuesday, 7/12. These emails were delivered to the Firefox DE HIGH, MED, LOW, and NO (lapsed) audiences so that we can get a measure from our overall audience. ------- A/B Test Version B won the AB test, delivering the higher overall click through rate with 2.44%. Version B content directed towards the German blog post. Perhaps this content resonated with subscribers best because of the language presented was in German. -------- Overall Email Analysis In all HIGH/MED/LOW/NO audiences, we noticed that between 30-50% of all clicks were unsubscribes, indicating that some subscribers were interested enough to open but did not engage with the content of the email. Typically we see 10%-20% of all clicks to be unsubscribes. (Note: This is different from an unsubscribe rate). Nonetheless, this test gave us valuable information on how the DE audience engages with content. In the DE HIGH audience we saw high engagement with a 7.19% CTR, but in the DE NO (lapsed) low overall engagement 0.19% CTR. My recommendation is that if we continue to test DE IRL email content to the Firefox DE audience, we should send only to the DE HIGH audience, and perhaps include subscribers who clicked/opened this send. --------- Send time: 7/12/18 @ 7am-9am GMT +2 (Berlin Time) Industry average goals* Mozilla: - Open: 25.25% - Click: 2.84% Firefox: - Open: 21.56% - Click: 2.40% Firefox - DE - ALL A/B Copy Test: A) Podcast CTA (1.95% click) B) Blog CTA (2.44% click) - WINNER 22.59% overall open 1.69% overall unique click 474,835 total sent ------- For reference, here's how the HIGH audience performed. Firefox - DE - ALL - HIGH A/B Copy Test: A) Podcast CTA (5.54% click) B) Blog CTA (8.89% click) - WINNER 42.85% overall open 7.19% overall unique click 11,929 total sent *These are industry averages (2015)--not Mozilla/Firefox averages--for comparable ongoing series sends.
Thanks, Steve! That is so interesting. @michaela, I am wondering if we should adjust copy for the next email according to that? And also looking into numbers of how many people click through to the podcast from the blog post.
No worries. Let me know if there is anything I can clarify, or if there is any specific information that you'd like for me to look into. Have a great weekend!
IRL S3E2 Email Results. I'm sharing the results for the EN IRL emails that went out this past Tuesday, 7/17. These emails went out to the Firefox EN HIGH audience and the Mozilla IRL audience. The A/B subject line winner was "Are you being gamed?", delivering the higher open rate for both the Firefox and Mozilla audiences Send time: 7/17/18 @ 7am-9am PST Industry average goals* Mozilla: - Open: 25.25% - Click: 2.84% Firefox: - Open: 21.56% - Click: 2.40% 40.97% Firefox - EN - ALL - HIGH Subject Lines: A) Are you being gamed? (40.97%) - WINNER B) Is it all fun and games? (38.56%) 42.44% overall open 4.91% overall unique click 102,913 total sent Mozilla - EN - IRL - ALL Subject Lines: A) Are you being gamed? (46.49%) - WINNER B) Is it all fun and games? (44.946%) 45.89% overall open 1.25% overall unique click 235,799 total sent *These are industry averages (2015)--not Mozilla/Firefox averages--for comparable ongoing series sends.
Hi team, Here is the email EN copy for Episode 3 of IRL: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit#. Anja is on PTO, but will post the DE copy aligning with Episode 3 later this week. If it still works for you, sending on Tuesday next week would be ideal. Meanwhile, sounds like the Firefox High list is a good list to move forward with on the DE emails. I think we're still waiting on one DE email — will it still go out on July 31? Thanks so much, Michaela
Hi team, I will submit GER copy for Ep. 3 tomorrow Am Berlin time. Another qs: Did you already coded in ep2 email for Germany? If not, I'd love to change it a little bit and send reader to the blogpost as the A/B test results were good for those. Thanks so much Anja
Thanks for the content y'all! Per slack convos, we'll work to get the EN S3, E3 email launched next week. Heads up: We currently don't have bandwidth to support the German IRL emails at this time. Feel free to make changes to copy as needed, and we'll keep you posted when we're able to prioritize the next German sends. Thanks for bearing with us on this!
(In reply to Jessilyn Davis from comment #32) > Thanks for the content y'all! > > Per slack convos, we'll work to get the EN S3, E3 email launched next week. > > Heads up: We currently don't have bandwidth to support the German IRL emails > at this time. Feel free to make changes to copy as needed, and we'll keep > you posted when we're able to prioritize the next German sends. > > Thanks for bearing with us on this! Thanks, Jess. Clarifying question: you can't support DE IRL emails this sprint? Or at all? Or, is that TBD?
(In reply to mthayer@mozilla.com from comment #33) > Clarifying question: you can't support DE IRL emails this sprint? Or at all? > Or, is that TBD? TBD.
Okay, thanks Jess. One note that I think is important to call out here: Our OKR is this: Improve NA & German Conscious Choosers’ perception of Mozilla by 15% over 2017 year end brand perception score. We're on the hook for improving brand perception by 15% in Germany in 2H. We've identified email as one of our largest vehicles to reach people and help them understand that Mozilla does more than make Firefox. So, DE email is very important to us. Totally recognize and appreciate all of the organizational priorities, but in diligence, just wanted to call the OKR at risk so that it can be taken into account as you think about resourcing! And, of course, we'll follow your lead on what you prioritize. Michaela
Hi team! Copy / assets for IRL Episode 4: Virtual Connections is all ready here: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit#heading=h.2v93gxvldn93. You'll note that there is no DE corresponding copy. Anja is switching email gears with sights on a DE newsletter instead. One thought: we have the emails set for this episode to release on August 14. If August 20 is an option, we'd like to consider waiting. We're seeing a bit less pick up than we did last season with listens, and our production partners think that maybe a delayed email is the way to go, to remind people who have already been notified via their podcast apps that the episode is waiting. I'm not convinced of this theory, but if it's easy enough to delay the send, we could give it a try and check our numbers. Let me know. And if this date change isn't possible, no biggie. Thanks, Michaela
Thanks for the heads up on content, Michaela! The week of Aug 20 is pretty full with emails to the Mozilla crew, whereas next week it's only this IRL email launch. What about an email on Thurs/Fri/Sat (16/17/18)? Remind people about the podcast before the weekend?
Yes, that works! Thursday or Friday would be great.
Great! It's on the calendar for Friday the 17th. We'll send proofs ahead of time once our team has tackled it starting next sprint. Thanks!
IRL S3E3 Email Results. I'm sharing the results for the EN IRL emails that went out this past Tuesday, 7/31. These emails went out to the Firefox EN HIGH audience and the Mozilla IRL audience. The A/B subject line winner was "What is your attention worth?", delivering the higher open rate by a significant lead for both the Firefox and Mozilla audiences. The overall email performed well with above standard open rate as usual. This indicates that subscribers are expecting the content and are engaged enough to open. Noticeably, the Firefox Audience has a higher overall click rate than the Mozilla audience, perhaps due to our Firefox segmentation method. Send time: 7/21/18 @ 6am-8am PST Industry average goals* Mozilla: - Open: 25.25% - Click: 2.84% Firefox: - Open: 21.56% - Click: 2.40% Firefox - EN - ALL - HIGH Subject Lines: A) What is your attention worth? (43.95% ) - WINNER B) Attention economics (37.53%) 42.42%overall open 7.15% overall unique click 102,913 total sent Mozilla - EN - IRL - ALL Subject Lines: A) What is your attention worth? (46.49%) - WINNER B) Attention economics (44.946%) 49.31% overall open 1.39% overall unique click 214,459 total sent *These are industry averages (2015)--not Mozilla/Firefox averages--for comparable ongoing series sends.
Wow. The CTRs were very high, significantly more than other IRL emails we've sent, while the open rates were steady (and of course, good). Do you think the high CTR has to do with how short the email was? Or just that the topic maybe was more interesting? I was actually hesitant about this email and its length but Steph reassured me that we should try it in Slack. I'm happy we did...
Hi all, Here is the copy for IRL's Episode 5 email all about making the web and world a kinder place: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit#heading=h.13qv7hy8fs7g. I'm copying in Daniel Kessler into this bug, as I'll be on PTO through Labor Day. He'll be reviewing the email test send, and helping you with any questions or concerns you have. Very best, and thank you! Michaela
Hey y'all, Here is a link the spreadsheet that will hold IRL email results moving forward. Still deciding where to put A/B test results, stay tuned for that: https://docs.google.com/spreadsheets/d/1DwFiiPQlsitfRaN0o3oN0pXj-bnVQvrQkkiaz0VaxrI/edit#gid=64282874&range=A1
Michaela and team, In response to your comment about the high CTR for the S3E3 email, I think it is correct to assume that it was a combination of an interesting topic and a shorter email that led to higher click engagement. For shorter emails vs longer emails... I have seen increases in CTR when paragraphs have a maximum of 2-3 lines when rendering on mobile (which this email does), and the total count of paragraphs before a button CTA is three or less (which this email also has).
Assignee: ssegovia → nobody
Assignee: nobody → jkleinsasser
Hey y'all, The hero image that was provided for the episode 5 email campaign, says episode 6. Can we get this changed to episode 5 please?
Hi, I'm getting this fixed. Please hang tight. I'll def have by Wednesday morning. Daniel
Thanks Daniel.
Hi all! Here is copy for the IRL Episode 6 email — the upcoming episode is all about kids and teens online. I would go as far as to say it's our best episode yet. Here is copy: https://docs.google.com/document/d/15GuzcPNxz4Ez8vVuaq04lzSCBLTNUanUlcg8sfppo-c/edit#heading=h.b0f9fnbsi8v. Let me know what you think. I know we've planned for a send on September 12, but we could also do September 10 if that's better for your team. Let me know. Either way is good for us. Thanks, Michaela P.S. Thanks for all the insightful analytics we've been getting on IRL and MIH newsletter sends! They've come up in conversations regularly and they're super actionable.
Hi Jesse and team, We'd like to delay the launch of IRL Episode 7 from September 24 to October 4. This is so that we can align with the Firefox elections bundle that will go live then – since our Episode 7 is about elections. Any concerns with delaying the email as well? It would mean I'll submit copy into this bug on the 24th (rather than today), and you'd have a week and a half to code, stage, and send. Let me know your thoughts. And — I'm also comfortable if the email for IRL Episode 7 doesn't go out right on the 4th. Anytime in the week following would be totally fine. Let me know what you think. Best, Michaela
Flags: needinfo?(jkleinsasser)
Hey Michaela, This sounds good to me. It doesn't look like there's a conflict on October 4th, but let me get with the email squad to see what the best day would be since you're open to sending the following week. I'll keep you posted :-) -Jesse K.
Flags: needinfo?(jkleinsasser)
Sounds great.
Hey Michaela! Per the email thread today with all the Firefox emails and Election Bundle timing - do you have objection to the email going on Oct 17th to both Moz IRL + full Firefox audiences - with a PS reminder to get the elections bundle?
Flags: needinfo?(mthayer)
Sounds great to me!
Flags: needinfo?(mthayer)
Ep. 7 (elections) email copy here: https://docs.google.com/document/d/1HSKUkfxscI3kYc5xdZElrz_thgkhhiEfBUjGtUpZQes/edit. LMK if you have questions / concerns!
Jesse, Curious if you might be able to send over the IRL Kids These Days email stats. We saw not-great OR and CTR related to our newsletter on a similar topic (kids and tech), and we're hoping to dig into if it was a content topic thing or something different. Thanks so much, Michaela
Flags: needinfo?(jkleinsasser)
Hey Michaela! Just fwded you Jesse's email with the stats from S3-E6. hope this helps!
Flags: needinfo?(jkleinsasser)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.