Closed Bug 775485 Opened 12 years ago Closed 10 years ago

Send welcome email to new accounts

Categories

(developer.mozilla.org Graveyard :: Sign-in, enhancement, P1)

Product:
developer.mozilla.org Graveyard
Component:
Sign-in
Platform:
x86
macOS
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jswisher, Assigned: openjck)

References

Details

Attachments

(1 file)

Analytics developer.mozilla.org (production) Campaigns 20140806-20140813.pdf
109.85 KB, application/vnd.pdf
Details 
After an account has been created, MDN should send an email message welcoming the user. For example, it could provide links to pages on how to contribute, how to join the MDN community mailing list and IRC channel.
See Also: → 676164
Priority: -- → P3
Version: Kuma → unspecified
Component: Docs Platform → Editing
With the redesign coming up, we should map out the text of this email now so we can hit the ground running.
I made an etherpad where we can work out the text: https://etherpad.mozilla.org/MDN-welcome-email

Keep in mind that we want users to feel like they didn't just create a wiki account; they joined a community: http://www.feverbee.com/2013/01/getting-members-into-the-community-mindset.html
Severity: normal → enhancement
Component: Editing → Login
Priority: P3 → P2
Priority: P2 → P1
Assignee: nobody → jkarahalis
Blocks: mdn-gh-login
We will be able to launch this soon. Should we finalize the copy? Who might be a good person to write a friendly welcome email like this?
In comment 2 there's a link to an etherpad  with a draft email, which is probably way too long. I'll check in with Jess Davis (Engagement's email goddess) to see if she has suggestions about improving it.
We might also want to think about what subject line to use.
I shortened the email by replacing the sections with links into the Getting Started page, and added a subject line: Take the next step to get involved on MDN!
https://etherpad.mozilla.org/MDN-welcome-email
Note: we should split test these emails/subjects for click-thru rate by using utm_content parameters on the link urls.

When we upgrade to universal GA, we can also measure the open rate too.
Hey team, quick question: What tech is being used to send these emails?

I ask because you may want to use ExactTarget (the email platform we use for all of our marketing & engagement emails) that does have open and click tracking, as well as delieverability tracking on our 4 dedicated IPs (for maximum deliverability). Even if this feature doesn't launch using ExactTarget, we may want to migrate this sending in the future to unify how we as Mozilla send emails (there are several other initiatives underway to get us all migrated into one unified email platform).
Hey Jessilyn, thanks for the input.

AIUI, we typically use ET for mailing *list* emails? These are 1-time emails per individuals, so ET feels a bit heavy for this use-case. I.e., how would we generate a report from ET that rolls up all the individual emails into a single view of open/click rates?
Flags: needinfo?(jdavis)
(In reply to Luke Crouch [:groovecoder] from comment #9)
> Hey Jessilyn, thanks for the input.
> 
> AIUI, we typically use ET for mailing *list* emails? These are 1-time emails
> per individuals, so ET feels a bit heavy for this use-case. I.e., how would
> we generate a report from ET that rolls up all the individual emails into a
> single view of open/click rates?

There are 2 different ways you can send an email via ExactTarget:

1) Mass mailing list (ie newsletter)
2) Triggered email sends (ie welcome email or transactional email)

Triggered email sends is the infra we'd use for this case. For example, our welcome emails are triggered when someone signs up for a new newsletter. These individual sends all have the same Job ID, and the reporting (open/click/sends/bounces/etc.) are rolled up into one large report that can be broken down into segments of time (ie since the program was started, in the last week, last month, particular date range, etc.)
Flags: needinfo?(jdavis)
Either one works for me.

Should we submit a standard site-generated email now, and consider using ExactTarget later? Or would we prefer to hold off until we have decided?
I say go for it for now, and we should definitely keep chatting to get ExactTarget implemented for a v2. This way we don't roadblock this great idea in progress.
Thanks Jessilyn! Do you know of other sites or applications using ExactTarget triggered emails? I have an idea of how to do our open and click-thru rates using Google Analytics, but it would help to see other triggered emails' reports so we can compare the approaches.
Ali, let's pick our KPI's for the emails. I propose:

* Open rate (Sent/Opened)
* Click-thru rate (Opened/Clicked)
* "Edit an Article" goal conversion rate
* "Translate an Article" goal conversion rate

What else?
Flags: needinfo?(aspivak)
The first two are easy to track, so yes, let's use those. 

I'm not sure about how we would track the other two (and we should add "Create an Account" to the list)
Flags: needinfo?(aspivak)
(In reply to ali spivak from comment #15)
> I'm not sure about how we would track the other two (and we should add
> "Create an Account" to the list)

The email we're talking about is to be sent when a user creates an account. The creation of the account has already happened by the time this email goes out.

I suppose we could track which new users edited or translated a document within some period after creating an account, and see if that rate was higher for users who opened the email. But I don't think there's a direct click-thru path to that.
There's not a direct report, but we can use GA's assisted conversions metrics on the sign-up emails divided by emails sent, opened, and clicked. That will let us see how valuable the emails are for users wanting to edit in a 1-90 day period after they get the email.
Just a quick question to be sure I'm not mistaken. Every new user will get the e-mail in English. Is this correct?
Flags: needinfo?(jkarahalis)
As the code is currently written, yes. But, we have the ability to translate the emails and then send appropriate translated emails based on the Accept-Language header sent in the registration request.

Should that block the feature? Or can it be a follow-up?
Flags: needinfo?(jkarahalis)
I recommend to disable the welcome email to non-EN audiences until the messages have been localized, and make this piece a blocker.

Otherwise, it's going to be a spammy experience, and cause technical (marking the email as spam & hurting deliverability) and user sentiment backlash (would you feel welcome if the site was in French and got an English welcome email?).

Then as the emails are localized and setup, the localized welcome feature can be added over time.

(FWIW, ExactTarget would have the ability to send the right language based on the information passed through so this infrastructure will exist whenever we're ready to explore this integration.)
Status: NEW → ASSIGNED
I pulled up some GA stats on SUMO's 'new-contributor' emails to see if we could define what to call "success". Not sure if these are helpful or too orthogonal for us.

2014-06-06 to 2014-07-06 Behavior:
517% more pages/session
553% longer avg. session duration

Goal conversion rates:
Ask a Question: 888% more effective
Helpful votes: 661% more effective

For our purposes, we want to see similar increase engagement (pages/session, avg. session duration) for welcome email sessions. And we want to see increased effectiveness of goal conversions to Edit an Article and Translate an Article. Our page watch notification emails are 48,715% more effective at driving article edits than other traffic sources. ;)

So maybe 500% can be the target success metrics for our welcome emails:

500% more pages/session
500% longer avg. session duration
500% better Edit Article conversion rate
500% better Translate Article conversion rate

I.e., we'll continue to iterate on this to get as close to those targets as possible.
What is the "control" for the SUMO stats? Is there a set of new accounts that don't get a welcome email?

Similarly, what will be the comparison for MDN stats?
The SUMO control group is visits/sessions that came to the site from other sources. So it includes google traffic, referral traffic, etc.

That's why the effectiveness is so high.

We'll compare MDN visits/sessions from the email against other sources - search, referrals, etc.
No longer blocks: mdn-gh-login
I found a heisenbug in the code that sends the translated emails. It's proving very tough (for me) to fix by myself. However, we've finished the code that doesn't send un-translated emails. ETA on the translated emails is anywhere from 5 to 10 days. The code itself is probably easy, but I need to get fresh eyes on it.

So, one way to partially ship this sooner is to ship welcome emails for en-US users first, and then follow it with translated emails as soon as we can.

:hoosteeno, Janet - how would you feel about shipping en-US welcome emails first?
Flags: needinfo?(jswisher)
Flags: needinfo?(hoosteeno)
> So, one way to partially ship this sooner is to ship welcome emails for
> en-US users first, and then follow it with translated emails as soon as we
> can.
> 
> :hoosteeno, Janet - how would you feel about shipping en-US welcome emails
> first?

Works for me as long as we follow through. Questions:
* Is there a bug for "fix translated emails"?
* Is someone assigned to it and is there a plan for closing it?
Flags: needinfo?(hoosteeno)
So, the proposed shipping solution is:
* en-US welcome emails to en-US users
* no welcome emails to non-en-US users

I think that's OK.
Flags: needinfo?(jswisher) → needinfo?(lcrouch)
Jean-Yves, how does this sound to you?
Flags: needinfo?(jypenator)
FWIW, I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1045187 but I did *not* make it block this bug. Bug 1029071 is already done, and ensures that non-English sign-ups will not receive English welcome emails.
Flags: needinfo?(lcrouch)
(In reply to Janet Swisher from comment #28)
> Jean-Yves, how does this sound to you?

As it is a better solution than what we have now and will allow us to gather experience, it looks like a good solution.
Flags: needinfo?(jypenator)
I enabled and tested the feature on stage. My only nit-pick is that the email From: is "notifications" :(

I'll fix that.
Hmm ... from whom should these emails be? Right now it's from notifications@developer.mozilla.org, just like the article update emails.

:hoosteeno - should it be from welcome@developer.mozilla.org? 

note: we can't translate this from address (yet). i.e., we can't make it "welcome@developer.mozilla.org" for English sign-ups and "bienvenido@developer.mozilla.org" for Spanish sign-ups.
Flags: needinfo?(hoosteeno)
The "From" should appear to be a real person, even if the actual address is an alias. (Ideally, the alias should also go to a real person, in case of responses, but that may be a separate bug.) For example:

"Janet Swisher, MDN Community Manager <welcome@developer.mozilla.org>"

Obviously the "real name" portion can change in case of personnel changes.
Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/5232b472275a15bbefcdde100e219e0199ffd3ed
fix bug 775485 - welcome email tweaks

https://github.com/mozilla/kuma/commit/d03a7885cd603cd73c291fa3c5e03f5339337888
Merge pull request #2607 from groovecoder/change-welcome-email-from-775485

fix bug 775485 - welcome email tweaks
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Merged the final version of this to dev server. We changed it from "Janet Swisher, MDN Community Manager <no-reply@mozilla.org>" so users don't get internal email errors if they do reply.

We can update the "From:" without a code push later so we can easily change it.
(In reply to Luke Crouch [:groovecoder] from comment #36)
> Merged the final version of this to dev server. We changed it from "Janet
> Swisher, MDN Community Manager <no-reply@mozilla.org>" so users don't get
> internal email errors if they do reply.
> 
> We can update the "From:" without a code push later so we can easily change
> it.

This is great. Thanks!
Flags: needinfo?(hoosteeno)
Commits pushed to github-allauth at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/5232b472275a15bbefcdde100e219e0199ffd3ed
fix bug 775485 - welcome email tweaks

https://github.com/mozilla/kuma/commit/d03a7885cd603cd73c291fa3c5e03f5339337888
Merge pull request #2607 from groovecoder/change-welcome-email-from-775485
Bah, our email sender (SocketLabs) doesn't like the comma in "Janet Swisher, MDN Community Manager". Propose alternatives?

"Janet Swisher and MDN Community Team"
"MDN Community Team"

?
Flags: needinfo?(jswisher)
Based on IRC conversation, updated From to just "Janet Swisher" https://github.com/mozilla/kuma/pull/2630
Flags: needinfo?(jswisher)
Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/0e8e6234893a1e3cd42acd442039cf408027cc94
fix bug 775485 - email from Janet

https://github.com/mozilla/kuma/commit/2793063dac1f96d86ad5d093eb810b22b3f61e14
Merge pull request #2631 from groovecoder/from-janet-775485

fix bug 775485 - email from Janet
Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/936ae2c80ec9f72b945a3e0a4f5770b5f476c651
bug 775485 - remove extra IRC link from html email

https://github.com/mozilla/kuma/commit/76a188036b18d853165712bf51aac2076e83d776
Merge pull request #2635 from groovecoder/extra-irc-link-775485

bug 775485 - remove extra IRC link from html email
**VERY** early analysis (i.e., 6 sessions triggered by welcome emails) are promising:

They have a 116% better goal conversion rate than other traffic sources. Note: I expect this will climb as we look at more data.

http://screencast.com/t/HVXZt9Zb
Updated bug tree - this doesn't block GitHub, and it depends on the translated emails before we call this done.
No longer blocks: mdn-gh-login
Status: RESOLVED → REOPENED
Depends on: 1045187
Resolution: FIXED → ---
Updated analysis. With 68 sessions reporting, the welcome emails have increased to a 176% better goal conversion rate than other traffic sources. (See attached GA report) So "it's looking pretty good for this myth." [1] I'm eager to see a full month's worth of data.

However, if we want to iterate on tweaks or another version of the email, we can actually A/B test the email content to see if different copy gives better results.

[1] https://en.wikipedia.org/wiki/MythBusters#Outcomes_of_the_experiments
(In reply to Luke Crouch [:groovecoder] from comment #46)
> Updated analysis. With 68 sessions reporting, the welcome emails have
> increased to a 176% better goal conversion rate than other traffic sources.
> (See attached GA report) So "it's looking pretty good for this myth." [1]
> I'm eager to see a full month's worth of data.
> 
> However, if we want to iterate on tweaks or another version of the email, we
> can actually A/B test the email content to see if different copy gives
> better results.
> 
> [1] https://en.wikipedia.org/wiki/MythBusters#Outcomes_of_the_experiments

Wow -- this sounds fantastic.
Whoops, it's even better than I thought. I compared the welcome email 'campaign' against other campaigns - not against all traffic sources.

From 77 sessions started via a welcome email, we had 21 goal completions - a 27.27% goal conversion rate, which is 4,495% better than from other traffic sources. The only campaign that out-performs the welcome emails are the "Page watch" emails (27 sessions; 21 goal completions; 77.78% goal conversion rate; 13,000% better than other sources).

So, returning to our "success metrics" [1] ...

237% more pages/session (6 vs. 1.78)
195% longer avg. session duration (06:55 vs. 02:20)
16,716% better Edit Article conversion rate (5.19% vs. 0.03%)
22,801% better Translate Article conversion rate (3.90% vs. 0.02%)

:hoosteeno - Do we want to iterate and/or experiment on the email content to try to improve the pages/session and avg. session duration goals?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=775485#c22
Flags: needinfo?(hoosteeno)
> :hoosteeno - Do we want to iterate and/or experiment on the email content to
> try to improve the pages/session and avg. session duration goals?

Sure. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1053877 for that, it's not a blocker.

But considering the powerful effect described in comment 49, I think the next iteration should be to increase the number of people getting these emails via https://bugzilla.mozilla.org/show_bug.cgi?id=1045187.
Flags: needinfo?(hoosteeno)
Adding bug 1056950 as blocker.
Depends on: 1056950
All the blockers on this bug are launched. That means this feature (which was on our roadmap) is done. And as comment 49 points out, it is hugely impactful. Thanks, Janet! Thanks, MDN devs!
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Note: New users in locales won't start seeing welcome emails until they are translated into their locale.
Updated "success metrics" [1] with 952 sessions since launch ...

205% more pages/session (5.49 vs. 1.80)
165% longer avg. session duration (06:07 vs. 02:18)
17,710% better Edit Article conversion rate (6.09% vs. 0.03%)
10,008% better Translate Article conversion rate (1.68% vs. 0.02%)

So, a drop in the effectiveness of "Translate Article" conversion rate. What a time to land translated welcome emails, eh? :)

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=775485#c22
wow super cool!
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: