1118797 - [release-promotion] Implement downstream source job

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Chris AtLee [:catlee]]

We provide tarballs and bundles of our source for each beta/release build, but should revisit if it's still necessary in today's world. Verify license requirements. Identify some important consumers (e.g. linux distros) and find out how they obtain a copy of the source. Check the number of downloads from ftp server. If we do need to continue producing source archives, then create a downstream builder to produce and upload them. In the future this may be combined with tagging, but that's not a requirement here.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I talked to gerv to see if we need still need to create source archives. The code under the MPL v2 doesn't, but we have other licenses in the tree and some of them do, notably the LGPL v2.1. Rather than keep track of the subset of code we need to provide source for we're going to keep producing full source tarballs.

I'll handle adding a scheduler that listens for a sendchange here, which is how we'll do jobs in the short term. Later on we'll probably have a taskcluster graph instead.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-custom] WIP 1

Are you OK with the direction this is going ? It's just a dummy builder at the moment which will become a real source job (maybe mozharness, maybe rejiggering SourceFactory). We'd have enable_build_promotion set on beta and release.

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 8578561 [details] [diff] [review]
[buildbot-custom] WIP 1

Review of attachment 8578561 [details] [diff] [review]:
-----------------------------------------------------------------

I'm definitely on board with the general idea of starting from scratch with release_promotion.py - hopefully it helps minimize cargo culting of cruft from release.py. (Though, with TaskCluster in the near future perhaps that doesn't matter as much.)

::: misc.py
@@ +2337,5 @@
>  
> +    if config.get('enable_build_promotion'):
> +        promotionObjects = generateBuildPromotionObjects(config,
> +            name)
> +        branchObjects = mergeBuildObjects(branchObjects, promotionObjects)

Did you purposely decide to generate these objects here (instead of in master.cfg like existing release objects)? I'm fine with either, just wondering.

::: process/release_promotion.py
@@ +27,5 @@
> +            if branchConfig['platforms'][p].get('use_mock'):
> +                mock_slaves.extend(platform_slaves)
> +    unix_slaves = [x for x in set(unix_slaves)]
> +    mock_slaves = [x for x in set(mock_slaves)]
> +    all_slaves = [x for x in set(all_slaves)]

I'm not sure if we need mock_slaves anymore. Looking at release.py it looks like unix_slaves ends up being mock_slaves in all cases:
https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.py#L367

This whole section makes me cringe, but I guess we need at least part of it because we don't define slave sets for things such as update_shipping (which runs on unix_slaves) or dummy factories...

@@ +56,5 @@
> +        branch='%s-build-promotion' % branchName,
> +        treeStableTimer=None,
> +        builderNames=topLevelBuilders,
> +        #fileIsImportant=lambda c: changeContainsProduct(c, releaseConfig['productName']) \
> +        #    and changeContainsScriptRepoRevision(c, releaseTag),

Is this commented out because you're not sure if you'll need it? It would be great not to have it...

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Did you purposely decide to generate these objects here (instead of in
> master.cfg like existing release objects)? I'm fine with either, just
> wondering.

Mainly to avoid doing it all of {builder,scheduler,universal_master_sqlite.cfg}_master.cfg. The fit with the dep builder generation seemed better than the release stuff too.
 
> I'm not sure if we need mock_slaves anymore. Looking at release.py it looks
> like unix_slaves ends up being mock_slaves in all cases:
> https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.
> py#L367

Good point, I'll tidy that up.
 
> This whole section makes me cringe, but I guess we need at least part of it
> because we don't define slave sets for things such as update_shipping (which
> runs on unix_slaves) or dummy factories...

Would you prefer to only have an all_slaves, and make as many jobs platform-agnostics as possible, or just limit to something like linux slaves ?

> @@ +56,5 @@
> > +        #fileIsImportant=lambda c: changeContainsProduct(c, releaseConfig['productName']) \
> > +        #    and changeContainsScriptRepoRevision(c, releaseTag),
> 
> Is this commented out because you're not sure if you'll need it? It would be
> great not to have it...

Leftover from a copy and paste, don't think I need it.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] WIP

I'm getting pretty close but have run into a problem with the source tarball. If I don't run tooltool then configure fails to complete when it can't find gcc in the src dir, but if tooltool is used then gcc is included in the tarball. The fix in bug 714063 was not to use tooltool, so there must be something different about how I'm running in mozharness which breaks configure. I'll look for that difference, or fall back to hardcoding another exclude at https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/upload-files.mk#781

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] WIP2

I've moved generateBuildPromotionObjects into misc.py so that I can use makeMHFactory(). Trying to use that in process/release-promotion.py created a circular import problem.

Comment 7

[Chris AtLee [:catlee]]

I think there's a configure/mozconfig option to disable the compiler environment. Would that help here?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

You probably mean --disable-compile-environement, but that's still broken even with bug 1063880. Turns out I missed setting no_tooltool=1 in the env, which means the mozconfigs don't force CC to the location tooltool provides, and instead uses the one in /tools/gcc-4.7.3-0moz1.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] builders + scheduling in misc.py

I've ended up moving this into misc.py, which addresses your earlier comments and allows me to use makeMHFactory from there. Not quite as clean, but you can't import from misc.py without creating a circular import which python rejects.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Enable scheduling

One liner to enable the previous patch.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] Add variant for source packaging

I'm leveraging the existing firefox desktop build harness (aka harness harness) to generate a source tarball and hg bundle. 

There's a new action package-source, which has just enough of build's preflight to get the source and the mozconfig. Then we run configure, with no_tooltool=1 so that it finds the system gcc and we don't need to tooltool. We don't actually need to compile anything here, just create configure from configure.in and so on. Then create the tarball (source-package) and the hg bundle. The latter specifies the revision, which comes from a buildbot property via a manual sendchange for now. Then upload to firefox/tinderbox-builds/<branch>-source>, enabling the hg bundle to be uploaded.

You can look at http://dev-master2.bb.releng.use1.mozilla.com:8710/builders/date_source/builds/29 if you'd like to see this in action.

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8585402 [details] [diff] [review]
[mozharness] Add variant for source packaging

Hijacking r? to free up jlund's resources on a merge day!

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8585402 [details] [diff] [review]
[mozharness] Add variant for source packaging

Review of attachment 8585402 [details] [diff] [review]:
-----------------------------------------------------------------

Looks straight forward to me.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #4)
> > This whole section makes me cringe, but I guess we need at least part of it
> > because we don't define slave sets for things such as update_shipping (which
> > runs on unix_slaves) or dummy factories...
> 
> Would you prefer to only have an all_slaves, and make as many jobs
> platform-agnostics as possible, or just limit to something like linux slaves
> ?

That would make it a bit cleaner, I guess, I'm not sure how realisitic it is though. It may be pretty outside the scope of what you're doing, so it's definitely not a requirement.

I think I was more just lamenting how a lot of these jobs are defined in release.py.

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 8585393 [details] [diff] [review]
[buildbotcustom] builders + scheduling in misc.py

Review of attachment 8585393 [details] [diff] [review]:
-----------------------------------------------------------------

::: misc.py
@@ +2333,5 @@
>              branchObjects['builders'].append(mozilla2_xulrunner_builder)
>  
>          # -- end of per-platform loop --
>  
> +    if config.get('enable_build_promotion'):

bug 1118796 talks uses enable_release_promotion (and the mozconfig variable is ENABLE_RELEASE_PROMOTION). I'm fine with either, but being consistent would be nice.

@@ +2334,5 @@
>  
>          # -- end of per-platform loop --
>  
> +    if config.get('enable_build_promotion'):
> +        promotionObjects = generateBuildPromotionObjects(config, name, secrets)

If you go with "release", can you rename this function too?

@@ +3392,5 @@
>          names.append(builder_name)
>      return names
> +
> +
> +def generateBuildPromotionObjects(config, name, secrets):

This got a wholllle lot cleaner, nice! Hopefully it can stay that way when misc. jobs like update stuff are here.

@@ +3406,5 @@
> +    }
> +
> +    topLevelBuilders = []
> +    pf_linux64 = config['platforms']['linux64']
> +    signing_servers = secrets.get(pf_linux64.get('dep_signing_servers'))

This will change to rel later I assume?

@@ +3441,5 @@
> +    starting_scheduler = Scheduler(
> +        name='%s_start_promotion' % name,
> +        branch='%s-build-promotion' % name,
> +        treeStableTimer=None,
> +        builderNames=topLevelBuilders,

Not necessary now, but we'll probably want a "reset schedulers" builder later if we end up using any AggregatingSchedulers.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 8585395 [details] [diff] [review]
[buildbot-configs] Enable scheduling

Review of attachment 8585395 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/project_branches.py
@@ +116,5 @@
>              'win64-debug': {},
>          },
>          'enable_valgrind': False,
>          'pgo_strategy': 'per-checkin',
> +        'enable_build_promotion': True,

r+ as long as this ends up consistently named.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8585393 [details] [diff] [review]
[buildbotcustom] builders + scheduling in misc.py

(In reply to Ben Hearsum [:bhearsum] from comment #15)
> bug 1118796 talks uses enable_release_promotion (and the mozconfig variable
> is ENABLE_RELEASE_PROMOTION). I'm fine with either, but being consistent
> would be nice.

I've made changes to be consistent with bug 118796 (ie s/build/release/).

> > +    signing_servers = secrets.get(pf_linux64.get('dep_signing_servers'))
> This will change to rel later I assume?

Right, probably the branch config will get pointed at the release signers.
 
> Not necessary now, but we'll probably want a "reset schedulers" builder
> later if we end up using any AggregatingSchedulers.

Good point.

As landed:
https://hg.mozilla.org/build/buildbotcustom/rev/adf6052198a4

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8585395 [details] [diff] [review]
[buildbot-configs] Enable scheduling

Landed using enable_release_promotion:
https://hg.mozilla.org/build/buildbot-configs/rev/fd1423e2d41b

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8585402 [details] [diff] [review]
[mozharness] Add variant for source packaging

https://hg.mozilla.org/build/mozharness/rev/519c604ea403

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

We'll have this functionality on Date once the landings in comment #17 and #18 have gone into production. 

Options for further improvement are to add a mach command to simplify the package_source in mozharness (see use of @CommandProvider in python/mozbuild/mozbuild/mach_commands.py for examples); uploading Taskcluster artifacts.

Comment 21

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/fd1423e2d41b

Comment 22

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/adf6052198a4

Comment 23

[Chris Cooper [:coop] (he/him)]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Test build in production:
 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/date-source/1427836371/

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Followups
* bug 1118798 - upload to taskcluster
* bug 1118798 - pretty up display on treeherder