Closed Bug 1282468 Opened 6 years ago Closed 6 years ago

disable buildbot asan builds and tests on trunk

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set
normal

Tracking

(firefox51 fixed)

RESOLVED FIXED
Tracking Status
firefox51 --- fixed

People

(Reporter: kmoir, Assigned: kmoir)

References

Details

Attachments

(3 files, 5 obsolete files)

Now that tests are running in tc in bug 1281501
And make asan tc builds tier 1.  

:gbrown I'm not sure of the criteria to make a tc build tier 1 but I think having tests and builds running would be a good first step.  Let me know if there are other criteria. I'll add a patch to disable the buildbot asan builds on trunk
Assignee: nobody → kmoir
Flags: needinfo?(gbrown)
I plan to compare tc vs buildbot asan tests over the next 48 hours or so and perhaps make some adjustments: I think there may be some intermittent time-outs that could be avoided, etc.

I have some concern about making more (beyond Linux 64 Debug) tc builds/tests tier 1 until there is less of a gap between bb and tc features -- bug 1080265. I'm mostly concerned about the lack of "Add missing jobs" support on treeherder, and SETA.


For general tier 1 criteria, I only know https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy really, but :jmaher has experience from Linux 64 Debug:

:jmaher -- Can you list out things we should check / hoops to be jumped through before proceeding here?
Flags: needinfo?(gbrown) → needinfo?(jmaher)
we are actively working on add missing jobs and SETA, I estimate completion or a small list of remaining work by the end of July on both of those.  I wouldn't want to block our transition of ASAN on these small details, it just bumps up the priority of them!

To move to Tier-1 we should have confidence in our builds/tests.  I believe that is done, but having a couple weeks of data points is important (or a try push with 30-50 retriggers of every job).  It is important for the tests that we don't have a greater level of intermittent failures than running in buildbot.

Another Tier-1 change is to ping the sheriffs and let them know- it is good to have 1 week of visibility there.  It is also good to post to dev.platform and dev.tree-management with the intention about 1 week out.

This should be exciting getting this to tier-1
Flags: needinfo?(jmaher)
Attached patch bug1282468.patch (obsolete) — Splinter Review
Summary: disable buildbot asan builds on trunk → disable buildbot asan builds and tests on trunk
Attached file bug1282868build.diff
builder diff
tests diff looks very sane too
Comment on attachment 8766005 [details] [diff] [review]
bug1282468.patch

we don't need to land this until some other issues have been worked out, but it would be good to get it reviewed
Attachment #8766005 - Flags: review?(fkang)
(In reply to Geoff Brown [:gbrown] from comment #2)
> I plan to compare tc vs buildbot asan tests over the next 48 hours or so and
> perhaps make some adjustments: I think there may be some intermittent
> time-outs that could be avoided, etc.

I landed some changes to avoid some of the most common failures.

I remain concerned about bug 1275963, a mochitest-clipboard failure that seems more frequent on tc than on buildbot.
Attachment #8766005 - Flags: review?(bugspam.Callek)
Comment on attachment 8766005 [details] [diff] [review]
bug1282468.patch

Review of attachment 8766005 [details] [diff] [review]:
-----------------------------------------------------------------

Looks straight forward, minor nits that I think improve code readability and speed.

::: mozilla-tests/config.py
@@ +3520,5 @@
>              continue
>          for slave_platform in ['ubuntu64_vm', 'ubuntu64_vm_lnx_large']:
>              BRANCHES[name]['platforms'][platform][slave_platform]['debug_unittest_suites'] = []
>  
> +#Bug 1282468 - disable buildbot asan builds on trunk

nit: Space after comment marker, to adhere to pep8 (I know this isn't the only place that pep8 is violated, but might as well prevent a new mistake)

@@ +3527,5 @@
> +        if platform not in ['linux64-asan']:
> +            continue
> +        if 'linux64-asan' in platform:
> +            del branch['platforms'][platform]
> +

This can be simplified with:

  for platform in branch['platforms'].keys():
      if platform not in ['linux64-asan']:
          continue
      del branch['platforms'][platform]

Since we don't need the extra if (we already know we want to delete here, and the `'str' in 'string'` would of course match stuff like 'linux64-asan-something-else' doing a string match, rather than an "array" match...

::: mozilla/config.py
@@ +2935,5 @@
>          if platform not in ['linux64-debug']:
>              continue
>          del branch['platforms'][platform]
>  
> +#Bug 1282468 - disable buildbot asan builds on trunk

pep8 again

@@ +2939,5 @@
> +#Bug 1282468 - disable buildbot asan builds on trunk
> +for name, branch in items_at_least(BRANCHES, 'gecko_version', 50):
> +    for platform in branch['platforms'].keys():
> +        if 'linux64-asan' in platform:
> +            del branch['platforms'][platform]

same structure as the mozilla-tests is probably useful here. for much the same reason.
Attachment #8766005 - Flags: review?(bugspam.Callek) → review+
Attachment #8766005 - Flags: review?(fkang)
(In reply to Geoff Brown [:gbrown] from comment #7)
> I remain concerned about bug 1275963, a mochitest-clipboard failure that
> seems more frequent on tc than on buildbot.

That issue is resolved now.
Attached patch bug1282468-2.patch (obsolete) — Splinter Review
unbitrotten patch
Attachment #8766005 - Attachment is obsolete: true
gbrown: just confirming that we are ready to turn off the asan buildbot tests and builds?
Flags: needinfo?(gbrown)
Yes, I think that is fine -- the tests seem to run as well on tc as on buildbot now.

:bc -- Any concerns? jmaher thought there might be an issue with builds for bughunter??

:kmoir -- Perhaps check with sheriffs too.
Flags: needinfo?(gbrown) → needinfo?(bob)
Attached patch bug1282468tier1.patch (obsolete) — Splinter Review
Yes, I will update tier to 1.  Trying this patch on try now.

Also renamed opt_linux64-asan.yml to opt_linux64_asan.yml to match other platforms
Attached patch bug1282468tier1-2.patch (obsolete) — Splinter Review
Attachment #8779019 - Attachment is obsolete: true
Personally, I'd rather see this depend on bug 1263815.
(In reply to Kim Moir [:kmoir] from comment #16)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=eecfe93ae185&selectedJob=25308826

Looks like you introduced a couple of flake8 errors (one from bug 1283879, I believe).
Bughunter Linux workers are currently down. When they come back up I'll use tc builds. Thanks for the heads up.
Flags: needinfo?(bob)
Kwierso/Tomcat: Any concerns with disabling asan builds and tests on buildbot not that they are running on tc?
Flags: needinfo?(wkocher)
Flags: needinfo?(cbook)
Comment on attachment 8779033 [details] [diff] [review]
bug1282468tier1-2.patch

Review of attachment 8779033 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/taskgraph/transforms/tests/all_kinds.py
@@ +35,5 @@
>      specify a tier otherwise."""
>      for test in tests:
>          # only override if not set for the test
>          if 'tier' not in test:
> +            if test['test-platform'] == 'linux64/debug' or test['test-platform'] == 'linux64-asan/opt':

Line too long flake8 error here.
Attachment #8779033 - Flags: review?(gbrown) → review+
Attached patch bug1282468tier1-3.patch (obsolete) — Splinter Review
patch updated to remove pep8 issue
Attachment #8779033 - Attachment is obsolete: true
Attachment #8779424 - Attachment is obsolete: true
fine for me
Flags: needinfo?(cbook)
thanks Tomcat will land patches
Flags: needinfo?(wkocher)
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a50ec658a90e
disable buildbot asan builds and tests on trunk r=gbrown
Attachment #8778913 - Attachment is obsolete: true
Attachment #8779463 - Flags: checked-in+
Attachment #8779755 - Flags: checked-in+
Duplicate of this bug: 1253301
verified on inbound, will close bug once the tc changes land on m-c
https://hg.mozilla.org/mozilla-central/rev/a50ec658a90e
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.