1418671 - Failing lint builds - source-test-file-metadata-bugzilla-components

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Natalia Csoregi [:nataliaCs]]

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=145775119&repo=autoland

Seems that it belongs to this push:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=306fff0f8a9d1fe8b7eeafd86c68166d78288767

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This Linting opt source-test-file-metadata-bugzilla-components (Bugzilla) has this failure:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4f92cfb21a2e8c3a3c2cb31fa065cb318a786df7&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&filter-searchStr=Bugzilla&selectedJob=145775119
It complains about a missing bugzilla component on every comm-central file on a mozilla trunk push.
Missing Bugzilla component: comm/...

The machine ran 2 web platform tests before and before that a comm-* ESlint job:
https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-t-linux-xlarge/workers/us-west-1/i-0d8a4b8c971388f38

https://tools.taskcluster.net/groups/TBLbZWJCTOa0yYbUlUKmNg/tasks/c6YSkIl-Tn2kFk0WOuV0pw/runs/0
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=306fff0f8a9d1fe8b7eeafd86c68166d78288767

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/0d601be171e5
Disable linting temporarily due to cache poisoning. r=me

Comment 3

[Tom Prince [:tomprince]]

Attachment: Bug 1418671: Use a separate cache for builds that checkout comm-central;

Since Firefox and Thunderbird both checkout mozilla-central, it is reasonable
to share the hg pool. However, since `hg purge` doesn't delete subrepositories,
they should not share even read-only checkouts.

This adds a seperate cache for the shared pool, and updates the run-task
version so that old and new checkout caches don't get mixed. Additionally, this
creates new checkout caches for tasks that are going to have comm-central
checked out.

Review commit: https://reviewboard.mozilla.org/r/201028/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/201028/

Comment 4

[Tom Prince [:tomprince]]

I've turned off linting jobs on comm-central while a fix is pending.

Comment 5

[Intermittent Failures Robot]

2 failures in 762 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2

Platform breakdown:
* lint: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1418671&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

https://reviewboard.mozilla.org/r/201028/#review206468

The shared store needs to be on the same volume as the checkouts to ensure things don't get out of sync. If they weren't on the same store, you could end up in a situation where the checkout cache has a changeset that isn't in the store cache. While I think robustcheckout will or could deal with this, it is a bad situation because it requires the working directory to be blown away.

Comment 7

[Tom Prince [:tomprince]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

https://reviewboard.mozilla.org/r/201028/#review206468

All the build jobs already don't use the same volume, since they do a checkout in the workspace cache, but still point to the checkout cache for the shared store.

Comment 8

[Gregory Szorc [:gps]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

https://reviewboard.mozilla.org/r/201028/#review206468

Then the build tasks are buggy. The reason we aren't bit by this is because we only run a single concurrent task on the build workers. For the workers where we can run multiple tasks concurrently (like decision workers), the checkout is on the same cache as the store.

I'm *very* nervous about breaking this standard and mixing and matching checkouts and stores. I may just add a sanity check to run-task that fails the task if cache spanning is detected.

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

https://reviewboard.mozilla.org/r/201028/#review211244

This still gets an r- from me because of cache spanning. To support comm-central, let's put something in the cache name (not necessarily the filesystem path) to delimit which *scope* caches belong to.

Comment 10

[Tom Prince [:tomprince]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

https://reviewboard.mozilla.org/r/201028/#review206468

Thanks for the more detailed explanation.

Comment 11

[Tom Prince [:tomprince]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/201028/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8929799 [details]
Bug 1418671: Use a separate cache for builds that checkout comm-central;

https://reviewboard.mozilla.org/r/201028/#review213958

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d38774aa733f
Use a separate cache for builds that checkout comm-central; r=gps

Comment 14

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/d38774aa733f

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5b6d18ff9a4f
Re-enable lints; r=me

Comment 16

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.