bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Enable build automation on the autoland repo

RESOLVED FIXED

Status

Infrastructure & Operations
CIDuty
RESOLVED FIXED
2 years ago
2 months ago

People

(Reporter: gps, Assigned: aselagea)

Tracking

(Blocks: 1 bug)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(7 attachments, 2 obsolete attachments)

(Reporter)

Description

2 years ago
Bug 1266864 created https://hg.mozilla.org/integration/autoland. We want to stand up Firefox build automation configured similarly to inbound and fx-team.

In the past ~month, ~20% of pushes to inbound have come from autoland. Those pushes will soon go to this new autoland repo once automation is running and the Sheriffs are monitoring. So shifting some capacity from inbound to autoland does seem to make some sense. I'm not sure whether it is prudent to do that up front or wait until pushes start arriving.
(Assignee)

Comment 1

2 years ago
Before implementing this, I think there are some aspects that need to be clarified. 

So, I'm not sure what you mean here by saying "We want to stand up Firefox build automation configured similarly to inbound and fx-team."? Does this refer to the build in commit being constrained by SETA as in the case for fx-team and m-i? 

Another thing that should be taken into consideration is how will this work given the current pool constraints? By "shifting some capacity" do you point to increasing the coalescing on m-i to allow capacity for autoland repo?

Thanks.
Flags: needinfo?(gps)
(Reporter)

Comment 2

2 years ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #1)
> So, I'm not sure what you mean here by saying "We want to stand up Firefox
> build automation configured similarly to inbound and fx-team."? Does this
> refer to the build in commit being constrained by SETA as in the case for
> fx-team and m-i? 

I believe so, yes.
 
> Another thing that should be taken into consideration is how will this work
> given the current pool constraints? By "shifting some capacity" do you point
> to increasing the coalescing on m-i to allow capacity for autoland repo?

I'm not sure. This isn't my area of expertise.

According to IRC chatter, apparently I was misunderstanding how capacity works. I thought there were machines allocated to different repos. Apparently there is more or less a giant pool of machines and priorities for each repo.
Flags: needinfo?(gps)
(Assignee)

Comment 3

2 years ago
At the moment, we're putting some effort on dealing with the current backlogs (especially when it comes to tests), so I think things will need to wait a bit on this.
(Assignee)

Updated

2 years ago
Depends on: 1264417

Comment 4

2 years ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #3)
> At the moment, we're putting some effort on dealing with the current
> backlogs (especially when it comes to tests), so I think things will need to
> wait a bit on this.

My concern, as always, is capacity, and right now my biggest concern is the Mac test pool. I asked Alin to hold off on this until buildduty had a chance to stand-up the new Mac buildbot masters to accommodate the new 10.10 testers Amy is setting up in bug 1257963. That work is almost finished, so buildduty should be able to tackle this bug next week.

(In reply to Gregory Szorc [:gps] from comment #2)
> According to IRC chatter, apparently I was misunderstanding how capacity
> works. I thought there were machines allocated to different repos.
> Apparently there is more or less a giant pool of machines and priorities for
> each repo.

That's correct. The only distinction we have right now is between try builders and regular builders. All testers are shared, and within the regular build pool, jobs are run based on the priority of the branch (e.g. release builds are always top priority).

Further to capacity, do we plan this as a strict cutover of check-in load to the autoland repo, or is there going to be some period of overlap where both integration branches are running the same check-in load in parallel?

Comment 5

2 years ago
Alin: will you have time to pick this up this week?
Flags: needinfo?(aselagea)

Comment 6

2 years ago
So essentially we'll need to setup the same builds and tests on autoland as they are on m-i.  This will include both buildbot and taskcluster pieces. From looking at buildbot-configs, it looks like master_common.py and project_branches.py need to change.  I don't think you should enable SETA for it.  Plus there are the taskcluster pieces to enable jobs on that branch. I don't know if there are treeherder changes to enable a new branch as well, something to investigate.
(Reporter)

Comment 7

2 years ago
Treeherder is already configured. It is waiting on jobs.
(Assignee)

Comment 8

2 years ago
Okay, will start working on this.
Flags: needinfo?(aselagea)
(Assignee)

Updated

2 years ago
Assignee: nobody → aselagea
(Assignee)

Comment 9

2 years ago
Created attachment 8754430 [details]
[buildbot] bug_1267712.patch

Patch to add the same builds and tests on autoland as on m-i.
Attachment #8754430 - Flags: review?(kmoir)
(Assignee)

Comment 10

2 years ago
Created attachment 8754431 [details]
jobs_added.txt
(Assignee)

Comment 11

2 years ago
[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_test]$ cat tests_new | grep 'mozilla-inbound' | wc -l
1374
[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_test]$ cat tests_new | grep 'autoland' | wc -l
1374

[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_build]$ cat builders_new | grep 'mozilla-inbound' | wc -l
26
[aselagea@dev-master2.bb.releng.use1.mozilla.com autoland_build]$ cat builders_new | grep 'autoland' | wc -l
26
(Assignee)

Comment 12

2 years ago
Created attachment 8754436 [details] [review]
Adding entries for autoland repo to mozilla-taskcluster
Attachment #8754436 - Flags: review?(wcosta)
(Assignee)

Comment 13

2 years ago
Created attachment 8754437 [details] [diff] [review]
[TC] bug_1267712.patch

m-i changes to enable TC jobs on autoland.
Attachment #8754437 - Flags: review?(wcosta)
Comment on attachment 8754430 [details]
[buildbot] bug_1267712.patch

This is good but I think there are some other buildbot related changes required before landing. You might want to search for m-i in the repos.  For example, production-branches.json needs to be updated in the tools repo.
Attachment #8754430 - Flags: review?(kmoir) → review+
Attachment #8754436 - Flags: review?(wcosta) → review+
Attachment #8754437 - Flags: review?(wcosta) → review+
(Assignee)

Updated

2 years ago
Attachment #8754436 - Flags: checked-in+
(Assignee)

Comment 15

2 years ago
Created attachment 8754758 [details] [diff] [review]
[tools] bug_1267712.patch
Attachment #8754758 - Flags: review?(kmoir)
(Assignee)

Comment 16

2 years ago
Created attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

vcs_sync changes, I'm not entirely sure if this is needed though.
Attachment #8754760 - Flags: review?(kmoir)
(Assignee)

Comment 17

2 years ago
Created attachment 8754773 [details] [diff] [review]
[buildbot] bug_1267712.patch

Updated the patch as I noticed that I mistakenly set the repo path for autoland to "build/autoland" instead of "integration/autoland". The former also exists, so that was a bit confusing.
Attachment #8754773 - Flags: review?(kmoir)
(Assignee)

Updated

2 years ago
Attachment #8754430 - Attachment is obsolete: true
Attachment #8754430 - Attachment is patch: false
(Assignee)

Comment 18

2 years ago
Regarding comment #17, I think the mozilla-taskcluster patch will need to be modified too (set the repo path to "integration/autoland" and SCM level to 3). I can upload a new patch if needed. Sorry for inconveniences.
Flags: needinfo?(wcosta)

Updated

2 years ago
Attachment #8754758 - Flags: review?(kmoir) → review+

Updated

2 years ago
Attachment #8754773 - Flags: review?(kmoir) → review+
Comment on attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

I'm not up to date on the state of vcssync so I'm going to defer this review to hwine
Flags: needinfo?(hwine)
Attachment #8754760 - Flags: review?(kmoir)
Comment on attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

Greg - could you confirm that this repo should be part of gecko-dev on github

Based on that answer, I'll review.
Flags: needinfo?(hwine) → needinfo?(gps)
Attachment #8754760 - Flags: review?(hwine)
Attachment #8754760 - Flags: feedback?(gps)
Greg - this repository currently has very odd permissions on hg.m.o -- can you clarify why it is not scm_level3? or a role account?
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #18)
> Regarding comment #17, I think the mozilla-taskcluster patch will need to be
> modified too (set the repo path to "integration/autoland" and SCM level to
> 3). I can upload a new patch if needed. Sorry for inconveniences.

I am not sure what I am supposed to answer here...
Flags: needinfo?(wcosta)
(Reporter)

Comment 23

2 years ago
This repo should *not* be part of gecko-dev on GitHub.

The repo permissions are weird on the server because we only want to allow the autoland user (and probably a few other one-off accounts) to push to it. We'll figure out another group in another bug. For purposes of TC permissions, it can be modeled as scm_level_3.
Flags: needinfo?(gps)
(Reporter)

Comment 24

2 years ago
Comment on attachment 8754760 [details] [diff] [review]
[mozharness] bug_1267712.patch

Review of attachment 8754760 [details] [diff] [review]:
-----------------------------------------------------------------

We don't want the autoland repo to be synced. Pretty sure this patch can be dropped.
Attachment #8754760 - Flags: feedback?(gps) → feedback-
(Assignee)

Comment 25

2 years ago
Created attachment 8755793 [details] [review]
Fix scm_level and repo path for autoland

According to https://bugzilla.mozilla.org/show_bug.cgi?id=1267712#c23, we can use scm_level=3 for TC. I also updated the repo path for autoland.
Attachment #8755793 - Flags: review?(wcosta)
(Assignee)

Updated

2 years ago
Attachment #8754773 - Attachment description: [buildbot] bug_1271394.patch → [buildbot] bug_1267712.patch
Attachment #8754773 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Attachment #8754758 - Flags: checked-in+

Updated

2 years ago
Attachment #8754437 - Flags: checked-in+
(Reporter)

Comment 28

2 years ago
Looks like the repository URL is misconfigured somewhere: something is using projects/autoland instead of integration/autoland. https://treeherder.mozilla.org/#/jobs?repo=autoland
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Gregory Szorc [:gps] from comment #29)
> Looks like the repository URL is misconfigured somewhere: something is using
> projects/autoland instead of integration/autoland.
> https://treeherder.mozilla.org/#/jobs?repo=autoland

Perhaps http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#789  ?
Created attachment 8756511 [details]
MozReview Request: Bug 1267712 - Add autoland configuration to mozharness r=jlund

Review commit: https://reviewboard.mozilla.org/r/55206/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/55206/
Attachment #8756511 - Flags: review?(jlund)

Comment 32

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/6d97fe2db560
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox49: --- → fixed
Resolution: --- → FIXED

Updated

2 years ago
Attachment #8756511 - Flags: review?(jlund) → review+
Comment on attachment 8756511 [details]
MozReview Request: Bug 1267712 - Add autoland configuration to mozharness r=jlund

https://reviewboard.mozilla.org/r/55206/#review51918

lgtm - one sanity check in line 

I never touched services-central so I can't vet removing that.

::: testing/mozharness/configs/builds/branch_specifics.py:300
(Diff revision 1)
>      'mozilla-inbound': {
>          'repo_path': 'integration/mozilla-inbound',
>          'stage_server': 'upload.ffxbld.productdelivery.prod.mozaws.net',
>      },
> -    'services-central': {
> -        'repo_path': 'services/services-central',
> +    'autoland': {
> +        'repo_path': 'integration/autoland',

sanity check, do we need to tweak checkout behaviour for autoland? e.g. like we do for try. or is this pretty much exactly like inbound?
services-central is dead

autoland is pretty much exactly like inbound. we only need this entry to override the default 'repo_path' that mozharness will calculate. actually - we could fix this up another way instead, by having the build script look in the buildbot properties for repo_path; it looks like it's currently ignoring it.
(Reporter)

Comment 35

2 years ago
https://reviewboard.mozilla.org/r/55206/#review51918

> sanity check, do we need to tweak checkout behaviour for autoland? e.g. like we do for try. or is this pretty much exactly like inbound?

We should configure the upstream to mozilla-central like we do for Try.

We could generate bundles for the autoland repo. But I don't think it is necessary. (Famous last words.)
https://reviewboard.mozilla.org/r/55206/#review51918

> We should configure the upstream to mozilla-central like we do for Try.
> 
> We could generate bundles for the autoland repo. But I don't think it is necessary. (Famous last words.)

okay, in that case I think we need this too: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/branch_specifics.py#274
(In reply to Chris AtLee [:catlee] from comment #34)

> actually -
> we could fix this up another way instead, by having the build script look in
> the buildbot properties for repo_path; it looks like it's currently ignoring
> it.

I kind have had this by design. I prefer to have everything configured in mozharness so we can control things in tree rather than depend on reconfigs. So with that in mind, I made it so mozharness looks in its own configs for everything and doesn't depend on buildbot_properties.json or w/e for anything: https://dxr.mozilla.org/build-central/source/buildbotcustom/misc.py#721

maybe we should remove repo_path from buildbot-configs so that it is more clear?
Attachment #8755793 - Flags: review?(wcosta) → review+
(Assignee)

Updated

2 years ago
Attachment #8755793 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Attachment #8754760 - Attachment is obsolete: true
Attachment #8754760 - Flags: review?(hwine)

Comment 39

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/836c207f205f
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 40

2 years ago
I'm not seeing TC automation at https://treeherder.mozilla.org/#/jobs?repo=autoland :/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

2 years ago
Depends on: 1278847
(Reporter)

Comment 41

2 years ago
TC automation was activated in bug 1278847. Calling this one resolved.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED

Updated

2 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.