Closed Bug 1267712 Opened 3 years ago Closed 3 years ago

Enable build automation on the autoland repo

Categories

(Infrastructure & Operations :: CIDuty, task)

task
Not set

Tracking

(firefox49 fixed)

RESOLVED FIXED
Tracking Status
firefox49 --- fixed

People

(Reporter: gps, Assigned: aselagea)

References

(Blocks 1 open bug)

Details

Attachments

(7 files, 2 obsolete files)

Bug 1266864 created https://hg.mozilla.org/integration/autoland. We want to stand up Firefox build automation configured similarly to inbound and fx-team.

In the past ~month, ~20% of pushes to inbound have come from autoland. Those pushes will soon go to this new autoland repo once automation is running and the Sheriffs are monitoring. So shifting some capacity from inbound to autoland does seem to make some sense. I'm not sure whether it is prudent to do that up front or wait until pushes start arriving.
Before implementing this, I think there are some aspects that need to be clarified. 

So, I'm not sure what you mean here by saying "We want to stand up Firefox build automation configured similarly to inbound and fx-team."? Does this refer to the build in commit being constrained by SETA as in the case for fx-team and m-i? 

Another thing that should be taken into consideration is how will this work given the current pool constraints? By "shifting some capacity" do you point to increasing the coalescing on m-i to allow capacity for autoland repo?

Thanks.
Flags: needinfo?(gps)
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #1)
> So, I'm not sure what you mean here by saying "We want to stand up Firefox
> build automation configured similarly to inbound and fx-team."? Does this
> refer to the build in commit being constrained by SETA as in the case for
> fx-team and m-i? 

I believe so, yes.
 
> Another thing that should be taken into consideration is how will this work
> given the current pool constraints? By "shifting some capacity" do you point
> to increasing the coalescing on m-i to allow capacity for autoland repo?

I'm not sure. This isn't my area of expertise.

According to IRC chatter, apparently I was misunderstanding how capacity works. I thought there were machines allocated to different repos. Apparently there is more or less a giant pool of machines and priorities for each repo.
Flags: needinfo?(gps)
At the moment, we're putting some effort on dealing with the current backlogs (especially when it comes to tests), so I think things will need to wait a bit on this.
Depends on: 1264417
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #3)
> At the moment, we're putting some effort on dealing with the current
> backlogs (especially when it comes to tests), so I think things will need to
> wait a bit on this.

My concern, as always, is capacity, and right now my biggest concern is the Mac test pool. I asked Alin to hold off on this until buildduty had a chance to stand-up the new Mac buildbot masters to accommodate the new 10.10 testers Amy is setting up in bug 1257963. That work is almost finished, so buildduty should be able to tackle this bug next week.

(In reply to Gregory Szorc [:gps] from comment #2)
> According to IRC chatter, apparently I was misunderstanding how capacity
> works. I thought there were machines allocated to different repos.
> Apparently there is more or less a giant pool of machines and priorities for
> each repo.

That's correct. The only distinction we have right now is between try builders and regular builders. All testers are shared, and within the regular build pool, jobs are run based on the priority of the branch (e.g. release builds are always top priority).

Further to capacity, do we plan this as a strict cutover of check-in load to the autoland repo, or is there going to be some period of overlap where both integration branches are running the same check-in load in parallel?
Alin: will you have time to pick this up this week?
Flags: needinfo?(aselagea)
So essentially we'll need to setup the same builds and tests on autoland as they are on m-i.  This will include both buildbot and taskcluster pieces. From looking at buildbot-configs, it looks like master_common.py and project_branches.py need to change.  I don't think you should enable SETA for it.  Plus there are the taskcluster pieces to enable jobs on that branch. I don't know if there are treeherder changes to enable a new branch as well, something to investigate.
Treeherder is already configured. It is waiting on jobs.
Okay, will start working on this.
Flags: needinfo?(aselagea)
Assignee: nobody → aselagea
Attached file [buildbot] bug_1267712.patch (obsolete) —
Patch to add the same builds and tests on autoland as on m-i.
Attachment #8754430 - Flags: review?(kmoir)
Attached file jobs_added.txt
[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_test]$ cat tests_new | grep 'mozilla-inbound' | wc -l
1374
[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_test]$ cat tests_new | grep 'autoland' | wc -l
1374

[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_build]$ cat builders_new | grep 'mozilla-inbound' | wc -l
26
[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_build]$ cat builders_new | grep 'autoland' | wc -l
26
m-i changes to enable TC jobs on autoland.
Attachment #8754437 - Flags: review?(wcosta)
Comment on attachment 8754430 [details]
[buildbot] bug_1267712.patch

This is good but I think there are some other buildbot related changes required before landing. You might want to search for m-i in the repos.  For example, production-branches.json needs to be updated in the tools repo.
Attachment #8754430 - Flags: review?(kmoir) → review+
Attachment #8754436 - Flags: review?(wcosta) → review+
Attachment #8754437 - Flags: review?(wcosta) → review+
Attachment #8754436 - Flags: checked-in+
Attachment #8754758 - Flags: review?(kmoir)
Attached patch [mozharness] bug_1267712.patch (obsolete) — Splinter Review
vcs_sync changes, I'm not entirely sure if this is needed though.
Attachment #8754760 - Flags: review?(kmoir)
Updated the patch as I noticed that I mistakenly set the repo path for autoland to "build/autoland" instead of "integration/autoland". The former also exists, so that was a bit confusing.
Attachment #8754773 - Flags: review?(kmoir)
Attachment #8754430 - Attachment is obsolete: true
Attachment #8754430 - Attachment is patch: false
Regarding comment #17, I think the mozilla-taskcluster patch will need to be modified too (set the repo path to "integration/autoland" and SCM level to 3). I can upload a new patch if needed. Sorry for inconveniences.
Flags: needinfo?(wcosta)
Attachment #8754758 - Flags: review?(kmoir) → review+
Attachment #8754773 - Flags: review?(kmoir) → review+
Comment on attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

I'm not up to date on the state of vcssync so I'm going to defer this review to hwine
Flags: needinfo?(hwine)
Attachment #8754760 - Flags: review?(kmoir)
Comment on attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

Greg - could you confirm that this repo should be part of gecko-dev on github

Based on that answer, I'll review.
Flags: needinfo?(hwine) → needinfo?(gps)
Attachment #8754760 - Flags: review?(hwine)
Attachment #8754760 - Flags: feedback?(gps)
Greg - this repository currently has very odd permissions on hg.m.o -- can you clarify why it is not scm_level3? or a role account?
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #18)
> Regarding comment #17, I think the mozilla-taskcluster patch will need to be
> modified too (set the repo path to "integration/autoland" and SCM level to
> 3). I can upload a new patch if needed. Sorry for inconveniences.

I am not sure what I am supposed to answer here...
Flags: needinfo?(wcosta)
This repo should *not* be part of gecko-dev on GitHub.

The repo permissions are weird on the server because we only want to allow the autoland user (and probably a few other one-off accounts) to push to it. We'll figure out another group in another bug. For purposes of TC permissions, it can be modeled as scm_level_3.
Flags: needinfo?(gps)
Comment on attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

Review of attachment 8754760 [details] [diff] [review]:
-----------------------------------------------------------------

We don't want the autoland repo to be synced. Pretty sure this patch can be dropped.
Attachment #8754760 - Flags: feedback?(gps) → feedback-
According to https://bugzilla.mozilla.org/show_bug.cgi?id=1267712#c23, we can use scm_level=3 for TC. I also updated the repo path for autoland.
Attachment #8755793 - Flags: review?(wcosta)
Attachment #8754773 - Attachment description: [buildbot] bug_1271394.patch → [buildbot] bug_1267712.patch
Attachment #8754773 - Flags: checked-in+
Attachment #8754758 - Flags: checked-in+
Attachment #8754437 - Flags: checked-in+
Looks like the repository URL is misconfigured somewhere: something is using projects/autoland instead of integration/autoland. https://treeherder.mozilla.org/#/jobs?repo=autoland
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Gregory Szorc [:gps] from comment #29)
> Looks like the repository URL is misconfigured somewhere: something is using
> projects/autoland instead of integration/autoland.
> https://treeherder.mozilla.org/#/jobs?repo=autoland

Perhaps http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#789  ?
https://hg.mozilla.org/mozilla-central/rev/6d97fe2db560
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Attachment #8756511 - Flags: review?(jlund) → review+
Comment on attachment 8756511 [details]
MozReview Request: Bug 1267712 - Add autoland configuration to mozharness r=jlund

https://reviewboard.mozilla.org/r/55206/#review51918

lgtm - one sanity check in line 

I never touched services-central so I can't vet removing that.

::: testing/mozharness/configs/builds/branch_specifics.py:300
(Diff revision 1)
>      'mozilla-inbound': {
>          'repo_path': 'integration/mozilla-inbound',
>          'stage_server': 'upload.ffxbld.productdelivery.prod.mozaws.net',
>      },
> -    'services-central': {
> -        'repo_path': 'services/services-central',
> +    'autoland': {
> +        'repo_path': 'integration/autoland',

sanity check, do we need to tweak checkout behaviour for autoland? e.g. like we do for try. or is this pretty much exactly like inbound?
services-central is dead

autoland is pretty much exactly like inbound. we only need this entry to override the default 'repo_path' that mozharness will calculate. actually - we could fix this up another way instead, by having the build script look in the buildbot properties for repo_path; it looks like it's currently ignoring it.
https://reviewboard.mozilla.org/r/55206/#review51918

> sanity check, do we need to tweak checkout behaviour for autoland? e.g. like we do for try. or is this pretty much exactly like inbound?

We should configure the upstream to mozilla-central like we do for Try.

We could generate bundles for the autoland repo. But I don't think it is necessary. (Famous last words.)
https://reviewboard.mozilla.org/r/55206/#review51918

> We should configure the upstream to mozilla-central like we do for Try.
> 
> We could generate bundles for the autoland repo. But I don't think it is necessary. (Famous last words.)

okay, in that case I think we need this too: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/branch_specifics.py#274
(In reply to Chris AtLee [:catlee] from comment #34)

> actually -
> we could fix this up another way instead, by having the build script look in
> the buildbot properties for repo_path; it looks like it's currently ignoring
> it.

I kind have had this by design. I prefer to have everything configured in mozharness so we can control things in tree rather than depend on reconfigs. So with that in mind, I made it so mozharness looks in its own configs for everything and doesn't depend on buildbot_properties.json or w/e for anything: https://dxr.mozilla.org/build-central/source/buildbotcustom/misc.py#721

maybe we should remove repo_path from buildbot-configs so that it is more clear?
Attachment #8755793 - Flags: review?(wcosta) → review+
Attachment #8755793 - Flags: checked-in+
Attachment #8754760 - Attachment is obsolete: true
Attachment #8754760 - Flags: review?(hwine)
https://hg.mozilla.org/mozilla-central/rev/836c207f205f
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
I'm not seeing TC automation at https://treeherder.mozilla.org/#/jobs?repo=autoland :/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 1278847
TC automation was activated in bug 1278847. Calling this one resolved.
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.