1184018 - Performance regression auto-bisect tool for B2G

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Performance, defect, P3)

Description

[Ting-Yu Chou [:ting] (away)]

As now we have Raptor monitor application startup, we should have an auto-bisect tool to find the culprit once there's a regression to make life easier.

There're some similar tools and related bugs:

  bug 1042550: regression detection for every gaia commit, nothing has been done
  bug 482536 (funfuzz): use js shell to run a test script to decide whether the build is good or bad, for firefox
  b2ghaystack: a script uses jenkins to build gecko for each commit in the range, see also bug 1038293
  mozregression: interactive script, test existed builds manually and input good/bad, output is commit range, for firefox
  mozcommitbuilder: support both interactive and condition script to bisect local builds, for firefox

Comment 1

[Ting-Yu Chou [:ting] (away)]

Based on [1], Raptor execute performance tests against devices for every new build generated by mozilla-central or b2g-inbound. So the tool will need to build before each bisect.

[1] https://hacks.mozilla.org/2015/06/performance-testing-firefox-os-with-raptor/

Comment 2

[Ting-Yu Chou [:ting] (away)]

Prototype: https://github.com/janus926/b2gbisectbot

The ideal goal is to make this a service and integrate with either Raptor or bugzilla so it will auto-bisect once a regression is detected.

Eli, how does Raptor detect regression and report to bugzilla? I couldn't find the code, the node module seems only reports to InfluxDB. Thanks.

Comment 3

[:Eli Perelman]

Shouldn't Taskcluster have all the builds you are looking for? Not sure, but thought I would pose the question.

Raptor's regression detection mechanism is two-fold: one repo is used for detecting and reporting regressions[1], the other is for filing the bugs and ensuring legitimacy[2]. 

[1] https://github.com/eliperelman/phanalysis
[2] https://github.com/eliperelman/raptor-bugs

Comment 4

[Ting-Yu Chou [:ting] (away)]

(In reply to :Eli Perelman from comment #3)
> Shouldn't Taskcluster have all the builds you are looking for? Not sure, but
> thought I would pose the question.

Thanks for the information, I'll figure this out.

Comment 5

[Ting-Yu Chou [:ting] (away)]

From ftp.mozilla.org I didn't see gecko/gaia builds for every revision. Also per https://goo.gl/uMcKWV, we're moving from ftp.mozilla.org to Taskcluster, but didn't mention that we will have builds for every revision with this transition.

I've sent an email https://goo.gl/QjSn3E to confirm this.

Comment 6

[Ting-Yu Chou [:ting] (away)]

I've just talked to Kanru. So we have builds for every push, not every revision. And he thinks it's good enough to bisect by push, so he didn't expect to use git or hg bisect.

Comment 7

[Ting-Yu Chou [:ting] (away)]

Hmm... but if Raptor already tests every m-c and b2g-inbound build, we will need to build revisions locally to bisect. I am confused.

Comment 8

[Ting-Yu Chou [:ting] (away)]

(In reply to Ting-Yu Chou [:ting] from comment #7)
> Hmm... but if Raptor already tests every m-c and b2g-inbound build, we will
> need to build revisions locally to bisect. I am confused.

For a huge push such as https://goo.gl/JJa0dG, we definitely need to bisect revisions.

And I can find builds for pushes on Taskcluster, with index:

  root.gecko.v1.b2g-inbound.revision.linux.xxxx...,
  root.gecko.v1.mozilla-central.revision.linux.xxxx...

Comment 9

[Ting-Yu Chou [:ting] (away)]

The regression range Raptor reports could across multiple pushes. So we can at first bisect pushes by exist builds, then bisect revisions by git/hg.

Comment 10

[Bobby Chien]

[Tracking Requested - why for this release]:

Comment 11

[Bobby Chien]

accident in comment 10. Reopen this.

Comment 12

[Ting-Yu Chou [:ting] (away)]

It finds the correct culprit in bug 1200504!

Comment 13

[Ting-Yu Chou [:ting] (away)]

In bug 1204837, the output of bisect bot is:

  Bisecting: bad
  changeset 262462:4968d9c21b40: bad
  The first bad revision is:
  changeset:   262462:4968d9c21b40
  parent:      262461:0baf29d2d17a
  parent:      262451:c69e31de9aec
  user:        Wes Kocher <[e-mail]>
  date:        Mon Sep 14 17:27:31 2015 -0700
  summary:     Merge m-c to b2ginbound, a=merge

  Not all ancestors of this changeset have been checked.
  Use bisect --extend to continue the bisection from
  the common ancestor, 6da01e6fcdf0.

The command used: $ node index.js 0baf29d2d17a19aa 13d2e06d95f07770 47f0ac6cf4c4525c 02a15ec81e0bece9 raptor.sh communications/dialer 3600.000

Comment 14

[BMO Automation]

Firefox OS is not being worked on