set up CI builds and tests for DevEdition on mozilla-beta

RESOLVED FIXED

Status

enhancement
P1
normal
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox54 fixed, firefox55 fixed)

Details

(Whiteboard: [keep-open])

Attachments

(10 attachments, 4 obsolete attachments)

59 bytes, text/x-review-board-request
bhearsum
: review+
Details
59 bytes, text/x-review-board-request
aki
: review+
kmoir
: review+
bhearsum
: checked-in+
Details
47 bytes, text/x-github-pull-request
emorley
: review+
KWierso
: feedback+
Details | Review
4.22 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
13.81 KB, patch
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
1.38 KB, patch
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
47 bytes, text/x-github-pull-request
emorley
: review+
Details | Review
3.95 KB, patch
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
1.17 KB, patch
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
8.84 KB, patch
aki
: review+
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
We need this in order to enable release promotion for Dev Edition
(Assignee)

Comment 1

2 years ago
A few notes:
- We won't enable these for try
- We need to use separate platform names for these (like we do for debug)
- This should upload to archive.mozilla.org/pub/devedition-beta/
- They need to be configured like the current Aurora builds are (unsigned addons on, diagnostic asserts on, devedition branding et. al).
(Assignee)

Updated

2 years ago
Depends on: 1358602
(Assignee)

Updated

2 years ago
Assignee: nobody → bhearsum
(Assignee)

Comment 2

2 years ago
Posted patch add devedition mozconfigs (obsolete) — Splinter Review
Because DevEdition will be built alongside vanilla Beta, we need separate mozconfigs for it. These are simply the current Aurora mozconfigs with MOZ_REQUIRE_SIGNING set to 0 (because the default on Beta is 1, per https://dxr.mozilla.org/mozilla-beta/source/build/mozconfig.common#19).

I think this will get us pretty close to what we want. I'm not 100% sure if diagnostic asserts will be enabled by this, though.
Attachment #8861463 - Flags: review?(rail)
Attachment #8861463 - Flags: review?(clarkbw)
Comment on attachment 8861463 [details] [diff] [review]
add devedition mozconfigs

LGTM

Gijs, can you review as well? Especially about the diagnostic asserts. Thx.
Attachment #8861463 - Flags: review?(gijskruitbosch+bugs)
Attachment #8861463 - Flags: review?(clarkbw)
Attachment #8861463 - Flags: feedback+

Comment 4

2 years ago
Comment on attachment 8861463 [details] [diff] [review]
add devedition mozconfigs

Review of attachment 8861463 [details] [diff] [review]:
-----------------------------------------------------------------

Two things:

- if these are copies, please use hg cp --after to mark them as such, which makes it easier to review just the differences with the original aurora configs.
- looks like diagnostic asserts are on for early beta or earlier:

https://dxr.mozilla.org/mozilla-central/source/mfbt/Assertions.h#447

if we want to force this on for devedition, it looks like you'll need to update that file and invent another ifdef and make sure it propagates to there (and get review from an appropriate person for that code, so not me! :-) ), or just always have early_beta_or_earlier turned on for devedition (which also changes other things). Not really my call, but you asked about diagnostic asserts and this at least tells you how the patch deals with those.
Attachment #8861463 - Flags: review?(gijskruitbosch+bugs) → review-
(Assignee)

Updated

2 years ago
Depends on: 1359583
(Assignee)

Comment 5

2 years ago
(In reply to :Gijs from comment #4)
> Comment on attachment 8861463 [details] [diff] [review]
> add devedition mozconfigs
> 
> Review of attachment 8861463 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Two things:
> 
> - if these are copies, please use hg cp --after to mark them as such, which
> makes it easier to review just the differences with the original aurora
> configs.

Will do.

> - looks like diagnostic asserts are on for early beta or earlier:
> 
> https://dxr.mozilla.org/mozilla-central/source/mfbt/Assertions.h#447
> 
> if we want to force this on for devedition, it looks like you'll need to
> update that file and invent another ifdef and make sure it propagates to
> there (and get review from an appropriate person for that code, so not me!
> :-) ), or just always have early_beta_or_earlier turned on for devedition
> (which also changes other things). Not really my call, but you asked about
> diagnostic asserts and this at least tells you how the patch deals with
> those.

Thank you for the insights here. It appears that EARLY_BETA_OR_EARLIER is hardcoded in https://dxr.mozilla.org/mozilla-central/source/build/defines.sh, which will get flipped partway through the beta cycle. Changing this is out of my wheelhouse (and I don't have time to learn), so I've filed bug 1359583. I don't think that needs to block landing the initial mozconfigs - we can update later once that's fixed.
(Assignee)

Updated

2 years ago
Blocks: 1359595
(Assignee)

Comment 6

2 years ago
Rescoping this bug a bit, because enabling tests looks like it will be much trickier. I've filed bug 1359595 for enabling the tests.
(Assignee)

Comment 7

2 years ago
I haven't tested this beyond a before and after dump master diff, which looked fine. I think it's probably fine to do any staging tests of this, since it's fully additive. Worst case scenario, we can hide the builds on Taskcluster.
Attachment #8861601 - Flags: review?(rail)

Updated

2 years ago
Attachment #8861601 - Flags: review?(rail) → review+
Comment hidden (mozreview-request)

Updated

2 years ago
Attachment #8861463 - Attachment is obsolete: true
Attachment #8861463 - Flags: review?(rail)
(Assignee)

Comment 9

2 years ago
mozreview-review
Comment on attachment 8862000 [details]
Bug 1358601 - add Developer Edition mozconfigs  DONTBUILD

https://reviewboard.mozilla.org/r/133984/#review136908

Thanks for fixing the hg copy stuff!
Attachment #8862000 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 10

2 years ago
(In reply to Ben Hearsum (:bhearsum) from comment #9)
> Comment on attachment 8862000 [details]
> Bug 1358601 - add Developer Edition mozconfigs  DONTBUILD
> 
> https://reviewboard.mozilla.org/r/133984/#review136908
> 
> Thanks for fixing the hg copy stuff!

Just for posterity, my original patch was originally reviewed by clarkbw, gijs, and rail. The only blocking issue was using "hg copy" instead of adding the files entirely new, which Rail has fixed in this version.
(Assignee)

Comment 11

2 years ago
Comment on attachment 8862000 [details]
Bug 1358601 - add Developer Edition mozconfigs  DONTBUILD

I've requested that the gecko patch be autolanded. We also need to uplift to mozilla-beta. Once that's done, we can land the Buildbot patch, which should get OS X and Windows builds going.
Attachment #8862000 - Flags: approval-mozilla-beta?
Comment on attachment 8862000 [details]
Bug 1358601 - add Developer Edition mozconfigs  DONTBUILD

For Dawn project, we need CI builds and tests running on the DevEd builds off of beta repo. While we are not ready to start producing those builds yet, we are getting ready to run tests on them when the time comes. I was told that this should be a no-op for the regular Beta builds and not impact our ability to gtb 54 builds in the meantime.

Gerry, just FYI.
Flags: needinfo?(gchang)
Attachment #8862000 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment 13

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9ae481187ca5
add Developer Edition mozconfigs r=bhearsum DONTBUILD

Comment 14

2 years ago
We're sorry - something has gone wrong while rewriting or rebasing your commits. The commits being pushed no longer match what was requested. Please file a bug.
(Assignee)

Comment 17

2 years ago
mozreview-review
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134384/#review137334

Kim, it looks like you did some branch-restricted builds recently - could you have a look at the taskgraph parts of this? Aki, I think the mozharness configs are pretty straightforward, but you seem like the best person to double check them.
(Assignee)

Updated

2 years ago
Attachment #8862457 - Flags: review?(kmoir)
Attachment #8862457 - Flags: review?(aki)

Comment 18

2 years ago
mozreview-review
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137358

You need to update 
 
https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py?q=target_tasks.py&redirect_type=direct#229  so the 'mozilla_beta_tasks' include the new build tasks 

You can test that the taskgraph is generated correctly locally by downloading a parameters.yml from a recent run on m-b to ensure the new build tasks are included in the resulting target graph ./mach taskgraph target -p ~/Downloads/parameters.yml Might be worth a try run to ensure the builds work too.  I think in this case you have to temporarily update the run-on-projects to include try
Attachment #8862457 - Flags: review?(kmoir) → review-
Comment hidden (mozreview-request)

Comment 20

2 years ago
mozreview-review
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137372

::: testing/mozharness/mozharness/mozilla/building/buildbase.py:384
(Diff revision 2)
>          'valgrind' : 'builds/releng_sub_%s_configs/%s_valgrind.py',
>          'artifact': 'builds/releng_sub_%s_configs/%s_artifact.py',
>          'debug-artifact': 'builds/releng_sub_%s_configs/%s_debug_artifact.py',
>          'qr-debug': 'builds/releng_sub_%s_configs/%s_qr_debug.py',
>          'qr-opt': 'builds/releng_sub_%s_configs/%s_qr_opt.py',
> +        'devedition': 'builds/releng_sub_%s_configs/%s_devedition.py',

This line refers to the path to the config file to use here:  https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/builds

I don't see any `*_devedition.py` files in this or other patches.  Did you create them and forget to `hg add`, or do you still need to create them?
(Assignee)

Comment 21

2 years ago
mozreview-review-reply
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137358

I'll have a look at this. For what it's worth, I did do some testing locally with "./mach taskgraph", and got the following diff: https://gist.github.com/mozbhearsum/5b4134a7d1e413c498db09d36dde529a
Comment hidden (mozreview-request)
(Assignee)

Comment 23

2 years ago
mozreview-review-reply
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137372

> This line refers to the path to the config file to use here:  https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/builds
> 
> I don't see any `*_devedition.py` files in this or other patches.  Did you create them and forget to `hg add`, or do you still need to create them?

Whoops, forgot to add them! Should be fixed now.
Comment hidden (mozreview-request)

Comment 25

2 years ago
mozreview-review
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137388

Just looking at the mozharness portions, this looks good.  You may want to push to try and use the task creator to trigger a devedition build and/or a beta graph if you want to make sure it works before landing.

::: testing/mozharness/configs/builds/releng_sub_linux_configs/32_devedition.py:1
(Diff revision 4)
> +import os

We don't seem to need this import, but it's harmless.

::: testing/mozharness/configs/builds/releng_sub_linux_configs/64_devedition.py:1
(Diff revision 4)
> +import os

same here.
Attachment #8862457 - Flags: review?(aki) → review+
(Assignee)

Comment 26

2 years ago
mozreview-review-reply
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137388

Thanks Aki! I actually did do a try push, and it seemed to work: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0da6f46b606bedd978a57716b6291ae600e943a&selectedJob=94602754

Comment 27

2 years ago
mozreview-review
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

https://reviewboard.mozilla.org/r/134386/#review137394
Attachment #8862457 - Flags: review?(kmoir) → review+

Comment 28

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/9ae481187ca5
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 29

2 years ago
We've got patches reviewed for both the Linux taskcluster builds and the Buildbot based Windows and Mac ones. I intend to land these patches tomorrow morning so we can get our first set of DevEdition builds on Beta going.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [keep-open]
(Assignee)

Comment 30

2 years ago
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

This patch will enable Linux DevEdition builds on mozilla-beta.
Attachment #8862457 - Flags: approval-mozilla-beta?
(Assignee)

Comment 31

2 years ago
Attachment #8862607 - Flags: review?(wkocher)
Comment on attachment 8862607 [details] [review]
add DevEdition bits to treeherder

This looks pretty good to me, though your test is failing in travis. I don't know that you need to change the group name for the talos jobs, seeing as these are getting their own platform line.

Ed should probably give the final stamp on this, though. :)
Attachment #8862607 - Flags: review?(wkocher)
Attachment #8862607 - Flags: review?(emorley)
Attachment #8862607 - Flags: feedback+
Comment on attachment 8862607 [details] [review]
add DevEdition bits to treeherder

Travis is failing (it should have sent an email, if it didn't let me know and I'll see if I can figure out what's up :-))
Attachment #8862607 - Flags: review?(emorley)
Attachment #8862457 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment 35

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/61140ae08b48
set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds r=aki,kmoir
Comment on attachment 8862607 [details] [review]
add DevEdition bits to treeherder

Many thanks! :-)
Attachment #8862607 - Flags: review+
(Assignee)

Comment 38

2 years ago
Comment on attachment 8862457 [details]
bug 1358601: set up CI builds and tests for DevEdition on mozilla-beta - add linux devedition builds

This is on beta, too: https://hg.mozilla.org/releases/mozilla-beta/rev/fcc6c261b2c9df3684bc6cd347437dc336ab0335
Attachment #8862457 - Flags: checked-in+
(Assignee)

Comment 39

2 years ago
We've now got basic builds going for DevEdition on mozilla-beta for all platforms. There are numerous issues that need to be fixed, though:
* Channel is set wrong (beta instead of aurora)
* MAR_CHANNEL_ID and ACCEPTED_MAR_CHANNEL_IDS are set wrong
* Builds aren't clobbering
* Buildbot builds can't upload builds or logs (probably due to stage_product being set to a new thing)
* DevEdition builds are triggering tests that are identifying as vanilla Beta (very serious)
(Assignee)

Comment 40

2 years ago
(In reply to Ben Hearsum (:bhearsum) from comment #39)
> We've now got basic builds going for DevEdition on mozilla-beta for all
> platforms. There are numerous issues that need to be fixed, though:
> * Channel is set wrong (beta instead of aurora)
> * MAR_CHANNEL_ID and ACCEPTED_MAR_CHANNEL_IDS are set wrong
> * Builds aren't clobbering
> * Buildbot builds can't upload builds or logs (probably due to stage_product
> being set to a new thing)
> * DevEdition builds are triggering tests that are identifying as vanilla
> Beta (very serious)

The latter two issues are both critical, and I can't find a quick fix for either so I've backed out the buildbot patch. The gecko patch should be safe to keep in - Linux isn't suffering from either of those issues.
(Assignee)

Comment 41

2 years ago
(In reply to Ben Hearsum (:bhearsum) from comment #39)
> We've now got basic builds going for DevEdition on mozilla-beta for all
> platforms. There are numerous issues that need to be fixed, though:
> * Channel is set wrong (beta instead of aurora)

These requires changes to the mozharness config.

> * MAR_CHANNEL_ID and ACCEPTED_MAR_CHANNEL_IDS are set wrong

This needs to be fixed in browser/confvars.sh, keyed off of channel.

> * Builds aren't clobbering

Another mozharness config fix.

> * Buildbot builds can't upload builds or logs (probably due to stage_product
> being set to a new thing)

Looks like this probably requires CloudOps as well. I'm poking them about it.

> * DevEdition builds are triggering tests that are identifying as vanilla
> Beta (very serious)

Also a mozharness config change.
(Assignee)

Comment 42

2 years ago
Aki, you seem like the best reviewer for this but feel free to redirect if not. Jordan is out for a bit, so he's not an option right now.

This patch should fix the channel issues, clobbering, and extra tests. Setting stage_platform should change the branch used in the sendchange, which will get rid of the extra tests. Setting this also allows us to override the channel in branch_specifics.py. Setting force_clobber=True should fix the clobbers.
Attachment #8862999 - Flags: review?(aki)
(Assignee)

Updated

2 years ago
Depends on: 1360683
(Assignee)

Comment 43

2 years ago
(In reply to Ben Hearsum (:bhearsum) from comment #41)
> (In reply to Ben Hearsum (:bhearsum) from comment #39)
> > * MAR_CHANNEL_ID and ACCEPTED_MAR_CHANNEL_IDS are set wrong
> 
> This needs to be fixed in browser/confvars.sh, keyed off of channel.

I got some pointers on this in #build, I'll give it a try next week:
16:10 < nalexander> bhearsum: did you get an answer?  In short, you can do this in the Python configure, I expect; I'd talk to chmanchester about that.
16:11 < bhearsum> nalexander: i did not
16:11 < nalexander> bhearsum: let me see if I can find an example.
16:11 < bhearsum> that would be awesome
16:12 < nalexander> bhearsum: https://dxr.mozilla.org/mozilla-central/source/mobile/android/moz.configure#15 suggests the approach.
16:12 < nalexander> bhearsum: now, it's possible the MAR channel IDs are special, let me try to evalute that.
16:14 < nalexander> bhearsum: I don't see anything that screams "this needs to be old-configure.in" and not in moz.configure Python; I expect you can use something like that stanza (expanding the is_nightly/is_... as necessary) to achieve what you want.
16:16 < bhearsum> nalexander: does that mean i should remove them from confvars.sh and put them into https://dxr.mozilla.org/mozilla-beta/source/browser/moz.configure ?
16:17 < nalexander> bhearsum: yes, just so -- remove the old-configure.in bits, remove the confvars.sh setting, add to browser/moz.configure (since it seems to be browser/ specific).
16:18 < nalexander> bhearsum: for testing, you can do a fresh configure with `rm -f $OBJDIR/config.status && ./mach configure`; then you can examine `$OBJDIR/config.cache` to see the values produced.
16:19 < nalexander> bhearsum: ./mach environment -v | grep CHANNEL_ID does roughly the config.cache checking too.


> > * Buildbot builds can't upload builds or logs (probably due to stage_product
> > being set to a new thing)
> 
> Looks like this probably requires CloudOps as well. I'm poking them about it.


This was fixed in https://bugzilla.mozilla.org/show_bug.cgi?id=1360683.

Updated

2 years ago
Attachment #8862999 - Flags: review?(aki) → review+

Comment 45

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4d047de72843
set up CI builds and tests for DevEdition on mozilla-beta - fix stage_platform, clobbers, and update_channel. r=asasaki
Comment on attachment 8862999 [details] [diff] [review]
fix stage_platform, clobbers, update_channel

needed to let us build dev edition from beta, beta54+
Attachment #8862999 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
(Assignee)

Updated

2 years ago
Attachment #8861601 - Flags: checked-in+ → checked-in-
(Assignee)

Comment 49

2 years ago
All of the mozharness fixes are in place now. This updated buildbot-configs patch should ensure that we're able to upload, and stop the bad tests from getting triggered.
Attachment #8863357 - Flags: review?(rail)

Updated

2 years ago
Attachment #8863357 - Flags: review?(rail) → review+
(Assignee)

Updated

2 years ago
Attachment #8861601 - Attachment is obsolete: true
(Assignee)

Comment 51

2 years ago
The OS X and Windows builds are working now, but uploading to the wrong place, whoops.
Attachment #8863397 - Flags: review?(rail)
Attachment #8863397 - Flags: approval-mozilla-beta?

Updated

2 years ago
Attachment #8863397 - Flags: review?(rail) → review+

Comment 52

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8ee9c747bcf9
set up CI builds and tests for DevEdition on mozilla-beta - set stage product for buildbot based builds. r=rail
(Assignee)

Updated

2 years ago
Attachment #8863439 - Flags: review?(emorley)
Comment on attachment 8863439 [details] [review]
[treeherder] mozbhearsum:devedition2 > mozilla:master

Left a comment :-)
Attachment #8863439 - Flags: review?(emorley)
(Assignee)

Comment 56

2 years ago
Made some progress today, but also found some new issues:
* Mac and Windows builds were uploading to /pub/firefox instead of /pub/devedition (already fixed)
* Mac and Windows builds are using Nightly branding instead of DevEdition
* PGO is not enabled
(Assignee)

Comment 57

2 years ago
It looks like hardcoded in the mozconfig is how we prefer to enable PGO for release-style builds.
Attachment #8863465 - Flags: review?(rail)

Updated

2 years ago
Attachment #8863465 - Flags: review?(rail) → review+
Comment on attachment 8863439 [details] [review]
[treeherder] mozbhearsum:devedition2 > mozilla:master

Many thanks for adding the test :-)
Attachment #8863439 - Flags: review+

Comment 60

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f9fb87bbc6b
set up CI builds and tests for DevEdition on mozilla-beta - enable pgo, fix Linux treeherder symbols, use correct mozconfigs for Mac and Windows. r=rail
(Assignee)

Comment 61

2 years ago
Comment on attachment 8863465 [details] [diff] [review]
enable pgo, fix Linux treeherder symbols, and use correct mozconfigs for Mac and Windows

https://hg.mozilla.org/releases/mozilla-beta/rev/8a7f650cff30f54e60e2a55f56c2e12222352f69
Attachment #8863465 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Depends on: 1361361
(Assignee)

Comment 64

2 years ago
I looked at the latest round of builds this morning and they all look good from a branding/update channel standpoint. There's two issues remaining that I can see:
1) Complete MARs are not being generated, which are needed for release promotion
2) MAR_CHANNEL_ID and ACCEPTED_MAR_CHANNEL_IDS are still set to the Beta versions. I've moved this off to bug 1361361.
(Assignee)

Comment 65

2 years ago
While investigating the MAR issues I realized that I based the DevEdition builds off of the "opt" tasks instead of the "nightly" tasks. I'm working on fixing that, but it looks like it'll be fairly invasive. Using nightly tasks turns on beetmover, signing, and possibly balrog submission - so we need to make sure all of those things are working and won't interfere with vanilla Beta builds before making the switch.
(Assignee)

Updated

2 years ago
Depends on: 1361446
Posted file gistfile1.txt (obsolete) —
testing this attachment, please ignore
(Assignee)

Comment 67

2 years ago
Most of this is just copying in stuff from the "beta" mozconfigs to devedition, and adjusting the linux.yml parameters to match nightly style instead of opt style. I had to update some hardcoding in taskgraph generation code for artifacts, and add a crazy hack to get certs set properly. If anyone can think of a better way to do that, I'm happy to switch it.

I intend to land this on jamun before mozilla-beta, so I've added it to the list of run-on-projects for now.

Here's a diff from "mach taskgraph target-graph" w/ mozilla-beta parameters.yml before and after my changes: https://gist.github.com/mozbhearsum/3bdfed7357e4478dfb904d92900f212c. I also ran a diff w/ mozilla-inbound parameters.yml and nothing changed.
Attachment #8864113 - Flags: review?(rail)
Attachment #8864113 - Flags: review?(aki)
(Assignee)

Comment 68

2 years ago
Comment on attachment 8864111 [details]
gistfile1.txt

Ignore this - Dylan was helping me figure out why I couldn't attach anything to this bug.
Attachment #8864111 - Attachment is obsolete: true
Comment on attachment 8864113 [details] [diff] [review]
switch to nightly-style build configs, make mars, fix signing certs

Review of attachment 8864113 [details] [diff] [review]:
-----------------------------------------------------------------

The mozcondfigs look familiar to me. It'd be great if someone with in-tree scheduling knowledge took a look at the rest
Attachment #8864113 - Flags: review?(rail) → review+
(Assignee)

Updated

2 years ago
Attachment #8863397 - Flags: approval-mozilla-beta? → checked-in+
Comment on attachment 8864113 [details] [diff] [review]
switch to nightly-style build configs, make mars, fix signing certs

>diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py
>--- a/taskcluster/taskgraph/target_tasks.py
>+++ b/taskcluster/taskgraph/target_tasks.py
>@@ -202,7 +202,7 @@ def target_tasks_nightly_linux(full_task
>     and, eventually, uploading the tasks to balrog."""
>     def filter(task):
>         platform = task.attributes.get('build_platform')
>-        if platform in ('linux64-nightly', 'linux-nightly'):
>+        if platform in ('linux64-nightly', 'linux-nightly', 'linux-devedition-nightly', 'linux64-devedition-nightly'):
>             return task.attributes.get('nightly', False)
>     return [l for l, t in full_task_graph.tasks.iteritems() if filter(t)]
>

I don't think you have to touch the nightly_linux target task.  That isn't run on beta, and devedition isn't enabled on m-c.

I think you have to edit mozilla_beta_tasks, which is the on-push target task on beta... or not.  The everything-then-subtract logic there may already have enabled devedition?

>diff --git a/taskcluster/taskgraph/transforms/signing.py b/taskcluster/taskgraph/transforms/signing.py
>--- a/taskcluster/taskgraph/transforms/signing.py
>+++ b/taskcluster/taskgraph/transforms/signing.py
>@@ -106,7 +106,19 @@ def make_task_description(config, jobs):
>             # Used for l10n attribute passthrough
>             attributes['chunk_locales'] = dep_job.attributes.get('chunk_locales')
>
>-        signing_cert_scope = get_signing_cert_scope(config)
>+        # Crazy hack for DevEdition builds, which are built on mozilla-beta but don't want to use the release
>+        # signing certs. We only want to override project or this part of the process, because indexes and
>+        # other things that use it must still be set to "mozilla-beta".
>+        if dep_job.attributes.get('build_platform') in set(['linux64-devedition-nightly', 'linux-devedition-nightly']):
>+            orig_params = config.params
>+            # config.params is a ReadOnlyDict, so we need to copy it to change project.
>+            config.params = config.params.copy()
>+            config.params["project"] = "mozilla-aurora"
>+            signing_cert_scope = get_signing_cert_scope(config)
>+            config.params = orig_params

I think you have the right idea.  Two things:

1. we'll need to remember to update the platform list when we osx and windows TC are tier1.  Not a biggie.
2. We're assuming that mozilla-aurora will live on in util/scriptworker.py, after we kill aurora everywhere else.  I don't think that's a great assumption.

We should probably add a new set of config and a get_devedition_signing_cert_scope() function to util/scriptworker.py, kind of like

>    DEVEDITION_SIGNING_SCOPE_ALIAS_TO_PROJECT = [[
>        'beta', set([
>            'mozilla-beta',
>            # jamun would go here on jamun
>        ])
>    ]]
>
>    DEVEDITION_SIGNING_CERT_SCOPES = {
>        'beta': 'project:releng:signing:cert:nightly-signing',
>        'default': 'project:releng:signing:cert:dep-signing',
>    }
>
>    get_devedition_signing_cert_scope = functools.partial(
>        get_scope_from_project,
>        DEVEDITION_SIGNING_SCOPE_ALIAS_TO_PROJECT,
>        DEVEDITION_SIGNING_CERT_SCOPES
>    )

but with more docstrings and in the proper sections of the file.


(In reply to Ben Hearsum (:bhearsum) from comment #67)
> Created attachment 8864113 [details] [diff] [review]
> switch to nightly-style build configs, make mars, fix signing certs
>
> Most of this is just copying in stuff from the "beta" mozconfigs to
> devedition, and adjusting the linux.yml parameters to match nightly style
> instead of opt style. I had to update some hardcoding in taskgraph
> generation code for artifacts, and add a crazy hack to get certs set
> properly. If anyone can think of a better way to do that, I'm happy to
> switch it.
>
> I intend to land this on jamun before mozilla-beta, so I've added it to the
> list of run-on-projects for now.

I've been recommending we land without jamun or date in the release branches, and then patch them locally to enable certain functionality... that avoids crufty jamun/date references years after we stop using them for beta/nightly testing.  If you intend to clean up afterwards, that's essentially the same thing.

> Here's a diff from "mach taskgraph target-graph" w/ mozilla-beta
> parameters.yml before and after my changes:
> https://gist.github.com/mozbhearsum/3bdfed7357e4478dfb904d92900f212c. I also
> ran a diff w/ mozilla-inbound parameters.yml and nothing changed.

This is with mozilla_beta_tasks?  If you were editing nightly_linux above, then this reinforces my hunch that the nightly_linux changes aren't needed.

Minusing primarily for those two sections, but you're close!
Attachment #8864113 - Flags: review?(aki) → review-
(Assignee)

Comment 71

2 years ago
Since we're moving active work on this to Jamun, need to enable Windows and Mac DevEdition builds there.
Attachment #8864847 - Flags: review?(rail)
Comment on attachment 8864847 [details] [diff] [review]
enable devedition builds on jamun

lgtm
Attachment #8864847 - Flags: review?(rail) → review+
(Assignee)

Comment 74

2 years ago
I think builds are looking quite good on Jamun now - I've verified the branding, channel, and signing on all platforms. We've got MARs created for all platforms as well. Here's a summary of all of the changes included here:
* Base DevEdition builds off of Nightly instead of Opt. This includes parts of the mozconfigs and linux.yml.
* Add devedition entries to beetmover.py. This might be more a release-y thing, but I found it while doing this, so I figured I'd include it.
* Add goop to signing.py and scriptworker.py to use different certs for devedition.
* Set devedition channel to aurora-dev for jamun. Not strictly required on beta, but we have a permanent override for beta-dev on jamun, so it seems like we should do the same for devedition.
Attachment #8864971 - Flags: review?(rail)
Attachment #8864971 - Flags: review?(aki)
Comment on attachment 8864971 [details] [diff] [review]
devedition build upgrades after jamun testing

the mozconfigs lgtm
Attachment #8864971 - Flags: review?(rail) → review+
Comment on attachment 8864971 [details] [diff] [review]
devedition build upgrades after jamun testing

The interdiff looks good.
Could you add a docstring for the new util/scriptworker.py constants, similar to the others?  At the very least point to this bug and/or explain that we need different scopes for devedition vs fx beta.
Attachment #8864971 - Flags: review?(aki) → review+

Comment 77

2 years ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/28f449764ca0
set up CI builds and tests for DevEdition on mozilla-beta - fix build configuration and signing of DevEdition-on-beta builds. r=aki,rail
(Assignee)

Comment 78

2 years ago
Comment on attachment 8864971 [details] [diff] [review]
devedition build upgrades after jamun testing

This was pushed to inbound and beta with a couple of small changes discussed on IRC:
* Added docstrings to taskcluster/taskgraph/util/scriptworker.py and taskcluster/taskgraph/transforms/signing.py
* Added "aurora-dev" to the stub installer channel list in browser/confvars.sh

https://hg.mozilla.org/releases/mozilla-beta/rev/7868898154ed9e19d98d9aa4c53b7da57c2bd187
https://hg.mozilla.org/integration/mozilla-inbound/rev/28f449764ca0b551dd74f8ae2bbaf0ac577771ac
Attachment #8864971 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Attachment #8864113 - Attachment is obsolete: true
(Assignee)

Comment 79

2 years ago
Once the DevEdition builds on https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=7868898154ed9e19d98d9aa4c53b7da57c2bd187 are completed we've got some DevEdition builds ready for testing! There won't be updates for them yet, but it would be good to get some eyes on them to make sure the builds themselves look correct.
(Assignee)

Updated

2 years ago
No longer depends on: 1359583

Updated

2 years ago
Priority: -- → P1

Updated

2 years ago
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
Flags: needinfo?(gchang)
Depends on: 1373657
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.