Implement new taskcluster-based rooting hazard analysis jobs

RESOLVED FIXED in Firefox 48

Status

()

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: sfink, Assigned: sfink)

Tracking

(Blocks: 2 bugs)

unspecified
mozilla48
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox48 fixed)

Details

Attachments

(5 attachments, 4 obsolete attachments)

(Assignee)

Description

3 years ago
I would like to switch all hazard analysis jobs over taskcluster and shoot the old ones in the head.
(Assignee)

Comment 1

3 years ago
Created attachment 8722739 [details] [diff] [review]
Taskcluster-based browser rooting hazard analysis

This implements the main job, the whole-browser analysis. It depends on the to-be-uploaded patch.
(Assignee)

Comment 2

3 years ago
Created attachment 8722741 [details] [diff] [review]
Extract out hazard logic to a separate file

Split out the hazard analysis logic into a separate script to be shared with the already-reviewed mulet analysis.
Attachment #8722741 - Flags: review?(garndt)
(Assignee)

Updated

3 years ago
Attachment #8722739 - Flags: review?(garndt)
(Assignee)

Comment 3

3 years ago
Created attachment 8722742 [details] [diff] [review]
Implement shell-only hazard analysis job

...and a JS shell hazard analysis (corresponding to the current linux64-sh-haz).
Attachment #8722742 - Flags: review?(garndt)
(Assignee)

Comment 4

3 years ago
Comment on attachment 8722739 [details] [diff] [review]
Taskcluster-based browser rooting hazard analysis

r? terrence for the stuff that does the stuff.
Attachment #8722739 - Flags: review?(terrence)

Comment 5

3 years ago
Comment on attachment 8722739 [details] [diff] [review]
Taskcluster-based browser rooting hazard analysis

Review of attachment 8722739 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/tasks/builds/haz_linux.yml
@@ +15,5 @@
> +    - 'docker-worker:relengapi-proxy:tooltool.download.public'
> +
> +  payload:
> +    cache:
> +      level-{{level}}-{{project}}-build-haz-linux-workspace: '/home/worker/workspace'

the scope for this cache needs to be added to task.scopes

Updated

3 years ago
Attachment #8722741 - Flags: review?(garndt) → review+

Comment 6

3 years ago
Comment on attachment 8722742 [details] [diff] [review]
Implement shell-only hazard analysis job

Review of attachment 8722742 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/tasks/builds/haz_shell_linux.yml
@@ +15,5 @@
> +    - 'docker-worker:relengapi-proxy:tooltool.download.public'
> +
> +  payload:
> +    cache:
> +      level-{{level}}-{{project}}-build-haz-linux-workspace: '/home/worker/workspace'

I believe task.scopes should have the scope for this cache.
Comment on attachment 8722739 [details] [diff] [review]
Taskcluster-based browser rooting hazard analysis

Review of attachment 8722739 [details] [diff] [review]:
-----------------------------------------------------------------

rs=me
Attachment #8722739 - Flags: review?(terrence) → review+

Comment 8

3 years ago
Is there anything I can help with on this bug?  I think with the task.scopes updated, these things should be good to go.
(Assignee)

Comment 9

3 years ago
Oh, maybe I'll re-upload the fixed versions so I can get the happy r+ stamp. Thanks for asking.

But these are currently blocked on the gcc version. I thought we could just upgrade GCC to 4.9.3 across the board, but apparently we want some gcc 4.8 jobs still, so I'm going to have to use a different tooltool manifest for these jobs so I can get gcc 4.9.3. And there are multiple ways of doing that, and I need to get feedback from glandium.
(Assignee)

Comment 10

3 years ago
Created attachment 8725020 [details] [diff] [review]
Taskcluster-based browser rooting hazard analysis

This is the task definition I've been testing with for a while. I believe it has all scope and other problems fixed.
Attachment #8725020 - Flags: review?(garndt)
(Assignee)

Updated

3 years ago
Attachment #8722739 - Attachment is obsolete: true
Attachment #8722739 - Flags: review?(garndt)
(Assignee)

Comment 11

3 years ago
Created attachment 8725021 [details] [diff] [review]
Implement shell-only hazard analysis job

And here's the shell version. Hopefully bzexport will be smart enough to obsolete the right ones.
Attachment #8725021 - Flags: review?(garndt)
(Assignee)

Updated

3 years ago
Attachment #8722742 - Attachment is obsolete: true
Attachment #8722742 - Flags: review?(garndt)
(Assignee)

Comment 12

3 years ago
glandium - I would like to land the new hazard analysis jobs with gcc 4.9.3. As per 1029245, it seems we cannot upgrade across the board to 4.9.3, so I'd like to just use 4.9.3 for the hazard builds.

One way to do that is to continue sharing the gecko releng.manifest, but then use a hazard-specific manifest file to overwrite the gcc that it installs. This could be a gcc-only manifest file, or I could have one hazard manifest that installs both gcc and a corresponding sixgill. (It is probably safest to keep the two paired, even though sometimes a sixgill binary works across several gcc versions.)

Alternatively, I could duplicate the gecko releng.manifest and either replace its gcc, or remove it entirely (and install it with a hazard-specific manifest).

Similarly, I could make a single browser hazard manifest that installs everything needed in one go (everything from releng.manifest but gcc, plus gcc 4.9.3 and sixgill). But that would also lead to duplication, in that the shell, browser, and mulet hazard builds all require different subsets of the other packages (gtk3, rustc, etc.)

Which would you prefer?
Flags: needinfo?(mh+mozilla)

Updated

3 years ago
Attachment #8725020 - Flags: review?(garndt) → review+

Updated

3 years ago
Attachment #8725021 - Flags: review?(garndt) → review+
(In reply to Steve Fink [:sfink, :s:] from comment #12)
> glandium - I would like to land the new hazard analysis jobs with gcc 4.9.3.
> As per 1029245, it seems we cannot upgrade across the board to 4.9.3, so I'd
> like to just use 4.9.3 for the hazard builds.
> 
> One way to do that is to continue sharing the gecko releng.manifest, but
> then use a hazard-specific manifest file to overwrite the gcc that it
> installs. This could be a gcc-only manifest file, or I could have one hazard
> manifest that installs both gcc and a corresponding sixgill. (It is probably
> safest to keep the two paired, even though sometimes a sixgill binary works
> across several gcc versions.)
> 
> Alternatively, I could duplicate the gecko releng.manifest and either
> replace its gcc, or remove it entirely (and install it with a
> hazard-specific manifest).
> 
> Similarly, I could make a single browser hazard manifest that installs
> everything needed in one go (everything from releng.manifest but gcc, plus
> gcc 4.9.3 and sixgill). But that would also lead to duplication, in that the
> shell, browser, and mulet hazard builds all require different subsets of the
> other packages (gtk3, rustc, etc.)
> 
> Which would you prefer?

The latter is what looks more like what we do everywhere. If we want more fancy things like includes in tooltool, we should change that in tooltool, not by invoking tooltool manually multiple times, imho.
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 14

3 years ago
Created attachment 8725512 [details] [diff] [review]
Upgrade browser to gcc 4.9.3

Ok, then I'll switch the b2g tooltool manifest to that too, so I'm requesting review on a patch to eliminate the second tooltool invocation.

Personally, I think tooltool is already a bit complicated, and would rather have multiple invocations be the standard way of using it instead of having *the* manifest that you install. That requires instantiating a separate manifest for every relevant configuration, when varying parts are shared. Seems like the jobs ought to be the repository of that logic, not materialized manifests nor complex include webs (because I'd immediately want substitution to decide which manifests to include, and the yml files already have includes + a substitution mechanism etc.)

But I'm happy to pretend that the problem is simple for now, while it still is. :-)
Attachment #8725512 - Flags: review?(mh+mozilla)
Comment on attachment 8725512 [details] [diff] [review]
Upgrade browser to gcc 4.9.3

Review of attachment 8725512 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/dev/config/tooltool-manifests/linux64/releng.manifest
@@ +13,5 @@
> +"digest" : "49627d734df52cb9e7319733da5a6be1812b9373355dc300ee5600b431122570e00d380d50c7c5b5003c462c2c2cb022494b42c4ad00f8eba01c2259cbe6e502",
> +"filename" : "sixgill.tar.xz",
> +"size" : 2628868,
> +"unpack" : true
> +},

You don't need sixgill on the normal builds anymore, do you?
Attachment #8725512 - Flags: review?(mh+mozilla) → feedback+
(Assignee)

Comment 16

3 years ago
Created attachment 8729176 [details] [diff] [review]
Clobber builds

I was seeing some mysterious problems that I wanted to blame on reusing old analysis results, so it seems safer to nuke them just in case. I'm not convinced it actually mattered, but I guess it's better to be safe.
Attachment #8729176 - Flags: review?(terrence)
(Assignee)

Comment 17

3 years ago
Created attachment 8729182 [details] [diff] [review]
Upgrade browser to gcc 4.9.3

> You don't need sixgill on the normal builds anymore, do you?

I never did, but this is the mulet hazard analysis and I was using the same manifest for mulet browser build and mulet hazard build. It seems you'd really rather duplicate all the manifests so they can be single-purpose, so I've updated the patch to do that.
Attachment #8729182 - Flags: review?(mh+mozilla)
(Assignee)

Updated

3 years ago
Attachment #8725512 - Attachment is obsolete: true
Comment on attachment 8729182 [details] [diff] [review]
Upgrade browser to gcc 4.9.3

Review of attachment 8729182 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/tooltool-manifests/linux64/hazard.manifest
@@ +1,3 @@
> +[
> +{
> +"version": "gcc 4.9.3",

Why is this file not marked as the copy of another?
Attachment #8729182 - Flags: review?(mh+mozilla)
(Assignee)

Comment 19

3 years ago
Created attachment 8729676 [details] [diff] [review]
Upgrade browser to gcc 4.9.3
Attachment #8729676 - Flags: review?(mh+mozilla)
(Assignee)

Updated

3 years ago
Attachment #8729182 - Attachment is obsolete: true
Attachment #8729176 - Flags: review?(terrence) → review+
Comment on attachment 8729676 [details] [diff] [review]
Upgrade browser to gcc 4.9.3

Review of attachment 8729676 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, the copy marker makes it easier to review.
Attachment #8729676 - Flags: review?(mh+mozilla) → review+
You need to log in before you can comment on or make changes to this bug.