1282468 - disable buildbot asan builds and tests on trunk

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

Now that tests are running in tc in bug 1281501

Comment 1

[Kim Moir [:kmoir] ET]

And make asan tc builds tier 1.  

:gbrown I'm not sure of the criteria to make a tc build tier 1 but I think having tests and builds running would be a good first step.  Let me know if there are other criteria. I'll add a patch to disable the buildbot asan builds on trunk

Comment 2

[Geoff Brown [:gbrown]]

I plan to compare tc vs buildbot asan tests over the next 48 hours or so and perhaps make some adjustments: I think there may be some intermittent time-outs that could be avoided, etc.

I have some concern about making more (beyond Linux 64 Debug) tc builds/tests tier 1 until there is less of a gap between bb and tc features -- bug 1080265. I'm mostly concerned about the lack of "Add missing jobs" support on treeherder, and SETA.


For general tier 1 criteria, I only know https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy really, but :jmaher has experience from Linux 64 Debug:

:jmaher -- Can you list out things we should check / hoops to be jumped through before proceeding here?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

we are actively working on add missing jobs and SETA, I estimate completion or a small list of remaining work by the end of July on both of those.  I wouldn't want to block our transition of ASAN on these small details, it just bumps up the priority of them!

To move to Tier-1 we should have confidence in our builds/tests.  I believe that is done, but having a couple weeks of data points is important (or a try push with 30-50 retriggers of every job).  It is important for the tests that we don't have a greater level of intermittent failures than running in buildbot.

Another Tier-1 change is to ping the sheriffs and let them know- it is good to have 1 week of visibility there.  It is also good to post to dev.platform and dev.tree-management with the intention about 1 week out.

This should be exciting getting this to tier-1

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: bug1282468.patch

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: bug1282868build.diff

builder diff
tests diff looks very sane too

Comment 6

[Kim Moir [:kmoir] ET]

Comment on attachment 8766005 [details] [diff] [review]
bug1282468.patch

we don't need to land this until some other issues have been worked out, but it would be good to get it reviewed

Comment 7

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #2)
> I plan to compare tc vs buildbot asan tests over the next 48 hours or so and
> perhaps make some adjustments: I think there may be some intermittent
> time-outs that could be avoided, etc.

I landed some changes to avoid some of the most common failures.

I remain concerned about bug 1275963, a mochitest-clipboard failure that seems more frequent on tc than on buildbot.

Comment 8

[Justin Wood (:Callek)]

Comment on attachment 8766005 [details] [diff] [review]
bug1282468.patch

Review of attachment 8766005 [details] [diff] [review]:
-----------------------------------------------------------------

Looks straight forward, minor nits that I think improve code readability and speed.

::: mozilla-tests/config.py
@@ +3520,5 @@
>              continue
>          for slave_platform in ['ubuntu64_vm', 'ubuntu64_vm_lnx_large']:
>              BRANCHES[name]['platforms'][platform][slave_platform]['debug_unittest_suites'] = []
>  
> +#Bug 1282468 - disable buildbot asan builds on trunk

nit: Space after comment marker, to adhere to pep8 (I know this isn't the only place that pep8 is violated, but might as well prevent a new mistake)

@@ +3527,5 @@
> +        if platform not in ['linux64-asan']:
> +            continue
> +        if 'linux64-asan' in platform:
> +            del branch['platforms'][platform]
> +

This can be simplified with:

  for platform in branch['platforms'].keys():
      if platform not in ['linux64-asan']:
          continue
      del branch['platforms'][platform]

Since we don't need the extra if (we already know we want to delete here, and the `'str' in 'string'` would of course match stuff like 'linux64-asan-something-else' doing a string match, rather than an "array" match...

::: mozilla/config.py
@@ +2935,5 @@
>          if platform not in ['linux64-debug']:
>              continue
>          del branch['platforms'][platform]
>  
> +#Bug 1282468 - disable buildbot asan builds on trunk

pep8 again

@@ +2939,5 @@
> +#Bug 1282468 - disable buildbot asan builds on trunk
> +for name, branch in items_at_least(BRANCHES, 'gecko_version', 50):
> +    for platform in branch['platforms'].keys():
> +        if 'linux64-asan' in platform:
> +            del branch['platforms'][platform]

same structure as the mozilla-tests is probably useful here. for much the same reason.

Comment 9

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #7)
> I remain concerned about bug 1275963, a mochitest-clipboard failure that
> seems more frequent on tc than on buildbot.

That issue is resolved now.

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1282468-2.patch

unbitrotten patch

Comment 11

[Kim Moir [:kmoir] ET]

gbrown: just confirming that we are ready to turn off the asan buildbot tests and builds?

Comment 12

[Geoff Brown [:gbrown]]

Yes, I think that is fine -- the tests seem to run as well on tc as on buildbot now.

:bc -- Any concerns? jmaher thought there might be an issue with builds for bughunter??

:kmoir -- Perhaps check with sheriffs too.

Comment 13

[Geoff Brown [:gbrown]]

Will you be changing the TC asan tier in this bug? https://dxr.mozilla.org/mozilla-central/rev/763fe887c37cee5fcfe0f00e94fdffc84a41ea1c/taskcluster/taskgraph/transforms/tests/all_kinds.py#39

Comment 14

[Kim Moir [:kmoir] ET]

Attachment: bug1282468tier1.patch

Yes, I will update tier to 1.  Trying this patch on try now.

Also renamed opt_linux64-asan.yml to opt_linux64_asan.yml to match other platforms

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug1282468tier1-2.patch

Comment 16

[Kim Moir [:kmoir] ET]

Comment on attachment 8779033 [details] [diff] [review]
bug1282468tier1-2.patch

https://treeherder.mozilla.org/#/jobs?repo=try&revision=eecfe93ae185&selectedJob=25308826

Comment 17

[Phil Ringnalda (:philor)]

Personally, I'd rather see this depend on bug 1263815.

Comment 18

[Geoff Brown [:gbrown]]

(In reply to Kim Moir [:kmoir] from comment #16)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=eecfe93ae185&selectedJob=25308826

Looks like you introduced a couple of flake8 errors (one from bug 1283879, I believe).

Comment 19

[Bob Clary [:bc] (inactive)]

Bughunter Linux workers are currently down. When they come back up I'll use tc builds. Thanks for the heads up.

Comment 20

[Kim Moir [:kmoir] ET]

Kwierso/Tomcat: Any concerns with disabling asan builds and tests on buildbot not that they are running on tc?

Comment 21

[Geoff Brown [:gbrown]]

Comment on attachment 8779033 [details] [diff] [review]
bug1282468tier1-2.patch

Review of attachment 8779033 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/taskgraph/transforms/tests/all_kinds.py
@@ +35,5 @@
>      specify a tier otherwise."""
>      for test in tests:
>          # only override if not set for the test
>          if 'tier' not in test:
> +            if test['test-platform'] == 'linux64/debug' or test['test-platform'] == 'linux64-asan/opt':

Line too long flake8 error here.

Comment 22

[Kim Moir [:kmoir] ET]

Attachment: bug1282468tier1-3.patch

patch updated to remove pep8 issue

Comment 23

[Kim Moir [:kmoir] ET]

Attachment: bug1282468tier1-4.patch

Comment 24

[Carsten Book [:Tomcat]]

fine for me

Comment 25

[Kim Moir [:kmoir] ET]

thanks Tomcat will land patches

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a50ec658a90e
disable buildbot asan builds and tests on trunk r=gbrown

Comment 27

[Kim Moir [:kmoir] ET]

Attachment: bug1282468-3.patch

Comment 29

[Kim Moir [:kmoir] ET]

verified on inbound, will close bug once the tc changes land on m-c

Comment 30

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/a50ec658a90e