Closed Bug 900658 Opened 11 years ago Closed 11 years ago

spin up funnelcake build for mozilla.org /firstrun content testing for [en-US] only

Categories

(Release Engineering :: Release Requests, defect, P2)

Product:
Release Engineering
Component:
Release Requests
Platform:
x86
All
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cmore, Assigned: nthomas)

References

Details

Attachments

(2 files, 2 obsolete files)

[partner-repacks] Repack configs
1.91 KB, patch
Details | Diff | Splinter Review
Notes on creating stub and full installers
7.29 KB, text/plain
Details 
Per bug 853869, we have been doing a lot of A|B testing with new content on the first run experience in Firefox. We want to see how different content influences ADIs long term. It may have a positive, negative, or neutral impact on ADIs and at this point we don't know. We have done 3 A|B tests to get down to content that people interact with the most and the next step is to do a funnelcake test.

This test will happen during the lifespan of Firefox 23.

The build en-US only for windows, mac, and linux. The product in bouncer should be whatever the normal product names would be + -f[XX] where XX is the funnelcake version. This will be set up similar to the SEM campaign, but we will use GA content experiments to do the A|B splitting.
Blocks: 853869
No longer blocks: 870469
No longer depends on: 870464, 872609
How many unique funnelcake XX values do you need ? Did you mean that you'll offer the current content to vanilla builds, and the new optimised content to a single funnelcake ? If so, how will you compare ADI long term ?

Do you need a different firstrun url at all, or will 
  http://www.mozilla.com/%LOCALE%/%APP%/%VERSION%/firstrun/?f=XX
be fine ?
(In reply to Nick Thomas [:nthomas] from comment #1)
> How many unique funnelcake XX values do you need ? Did you mean that you'll
> offer the current content to vanilla builds, and the new optimised content
> to a single funnelcake ? If so, how will you compare ADI long term ?
> 
> Do you need a different firstrun url at all, or will 
>   http://www.mozilla.com/%LOCALE%/%APP%/%VERSION%/firstrun/?f=XX
> be fine ?

Good question. It would probably good to have two funnelcakes one for the control (the current version of the firstrun page) and the other funnelcake build with the updated firstrun experience.

So f=24 (for example) would be the control and f=25 would be the variant. We would vary the first run page display depending on those f values.

Anurag: Would you prefer two funnelcakes to make the A|B test cleaner?
Flags: needinfo?(aphadke)
cmore - I agree with your suggestion, let's have two funnelcakes, control + variant.
Flags: needinfo?(aphadke)
No longer blocks: 853869
Blocks: 901028
Product: mozilla.org → Release Engineering
What's left to do here, if anything?
Flags: needinfo?(nthomas)
Flags: needinfo?(chrismore.bugzilla)
(In reply to Ben Hearsum [:bhearsum] from comment #4)
> What's left to do here, if anything?

We are going to wait until the new /firstrun page is developed before spinning up the funnelcakes. What we don't want to have happen is that the page is done a bit before Firefox 24, we spin up funnelcake based on Firefox 23, and then we have to make new funnelcakes for 24 because there is not enough time left before the end of the 6 week cycle to run the rest. When we getting close to completing the new page, I will ping this bug.
Flags: needinfo?(chrismore.bugzilla)
That sounds great, thanks. Nothing to do here until then.
Flags: needinfo?(nthomas)
OS: Mac OS X → All
Priority: -- → P3
Nick: I need to verify two things.

1) Is the next incrementing number for funnelcake builds 24? The funnelcake build ids that we need to make here are 24 and 25. There will need to be 6 builds created for each 24 and 25 ids for windows, mac, and linux. 

2) Given Firefox 24 is coming the week of September 17th, we will have to do this ADI post-release. What is the earliest estimate date given the 17th release that you could have all 6 funnelcake builds created from Firefox 24?

Thanks!
Flags: needinfo?(nthomas)
(In reply to Chris More [:cmore] from comment #7)
> Nick: I need to verify two things.
> 
> 1) Is the next incrementing number for funnelcake builds 24? The funnelcake
> build ids that we need to make here are 24 and 25. There will need to be 6
> builds created for each 24 and 25 ids for windows, mac, and linux. 

24 and 25 are next. To be crystal clear
* two funnelcakes (24, 25)
* three platforms (windows, mac, linux32)
* one locale (en-US)
--> 6 builds in total ?

> 2) Given Firefox 24 is coming the week of September 17th, we will have to do
> this ADI post-release. What is the earliest estimate date given the 17th
> release that you could have all 6 funnelcake builds created from Firefox 24?

I'm on the hook for three releases due out Sep 17. If they go really well then the funnelcakes could be up on Sep 17 itself. If there are problems then it might be a few days after.  I want to avoid redoing work if we need to respin Firefox 24, which means waiting until signoff on Sep 16 for the full installers. I'll also need to create two stub installers, but they can be created before hand if I have time.

Do you want to hit or avoid the extra traffic we get around release time ?
Flags: needinfo?(nthomas)
 
> 24 and 25 are next. To be crystal clear
> * two funnelcakes (24, 25)
> * three platforms (windows, mac, linux32)
> * one locale (en-US)
> --> 6 builds in total ?
>

Exactly correct.

> Do you want to hit or avoid the extra traffic we get around release time ?

I want to avoid some of the initial spike in manual upgrade traffic, but it shouldn't matter since we will segment the traffic down to non-firefox users and manual updates won't get funnelcake either. Basically, as soon as you get Firefox 24 out the door, create the 6 funnelcake builds and ping this bug. We can enable the GA experiment at any point by literally flipping the switch. Thanks!
Nick: What is the ETA on the 6 builds being ready built off of Firefox 24?
Flags: needinfo?(nthomas)
I'll work on them today, so either today or your Monday.
Assignee: nobody → nthomas
Flags: needinfo?(nthomas)
Priority: P3 → P2
It'll be Monday, I'm having issues generating stub installers.
This was the first time we 'repacked' an en-US stub installer, so there was a bit of work figure out the right incantation.
Missed the linux & mac builds.
Attachment #808385 - Attachment is obsolete: true

        Attachment #808396 -
        Attachment is obsolete: true

    
    

    
  
Chris, I'm on PTO from tomorrow. Is everything OK here from your point of view ?
Flags: needinfo?(chrismore.bugzilla)

    
    

    
  
All good and the rest has been running for two days. :) Thanks
Flags: needinfo?(chrismore.bugzilla)

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: