Closed
Bug 1184018
Opened 9 years ago
Closed 6 years ago
Performance regression auto-bisect tool for B2G
Categories
(Firefox OS Graveyard :: Performance, defect, P3)
Tracking
(tracking-b2g:backlog)
RESOLVED
WONTFIX
tracking-b2g | backlog |
People
(Reporter: ting, Unassigned)
References
Details
As now we have Raptor monitor application startup, we should have an auto-bisect tool to find the culprit once there's a regression to make life easier. There're some similar tools and related bugs: bug 1042550: regression detection for every gaia commit, nothing has been done bug 482536 (funfuzz): use js shell to run a test script to decide whether the build is good or bad, for firefox b2ghaystack: a script uses jenkins to build gecko for each commit in the range, see also bug 1038293 mozregression: interactive script, test existed builds manually and input good/bad, output is commit range, for firefox mozcommitbuilder: support both interactive and condition script to bisect local builds, for firefox
Reporter | ||
Comment 1•9 years ago
|
||
Based on [1], Raptor execute performance tests against devices for every new build generated by mozilla-central or b2g-inbound. So the tool will need to build before each bisect. [1] https://hacks.mozilla.org/2015/06/performance-testing-firefox-os-with-raptor/
Reporter | ||
Comment 2•9 years ago
|
||
Prototype: https://github.com/janus926/b2gbisectbot The ideal goal is to make this a service and integrate with either Raptor or bugzilla so it will auto-bisect once a regression is detected. Eli, how does Raptor detect regression and report to bugzilla? I couldn't find the code, the node module seems only reports to InfluxDB. Thanks.
Flags: needinfo?(eperelman)
Comment 3•9 years ago
|
||
Shouldn't Taskcluster have all the builds you are looking for? Not sure, but thought I would pose the question. Raptor's regression detection mechanism is two-fold: one repo is used for detecting and reporting regressions[1], the other is for filing the bugs and ensuring legitimacy[2]. [1] https://github.com/eliperelman/phanalysis [2] https://github.com/eliperelman/raptor-bugs
Flags: needinfo?(eperelman)
Reporter | ||
Comment 4•9 years ago
|
||
(In reply to :Eli Perelman from comment #3) > Shouldn't Taskcluster have all the builds you are looking for? Not sure, but > thought I would pose the question. Thanks for the information, I'll figure this out.
Reporter | ||
Comment 5•9 years ago
|
||
From ftp.mozilla.org I didn't see gecko/gaia builds for every revision. Also per https://goo.gl/uMcKWV, we're moving from ftp.mozilla.org to Taskcluster, but didn't mention that we will have builds for every revision with this transition. I've sent an email https://goo.gl/QjSn3E to confirm this.
Reporter | ||
Comment 6•9 years ago
|
||
I've just talked to Kanru. So we have builds for every push, not every revision. And he thinks it's good enough to bisect by push, so he didn't expect to use git or hg bisect.
Reporter | ||
Comment 7•9 years ago
|
||
Hmm... but if Raptor already tests every m-c and b2g-inbound build, we will need to build revisions locally to bisect. I am confused.
Reporter | ||
Comment 8•9 years ago
|
||
(In reply to Ting-Yu Chou [:ting] from comment #7) > Hmm... but if Raptor already tests every m-c and b2g-inbound build, we will > need to build revisions locally to bisect. I am confused. For a huge push such as https://goo.gl/JJa0dG, we definitely need to bisect revisions. And I can find builds for pushes on Taskcluster, with index: root.gecko.v1.b2g-inbound.revision.linux.xxxx..., root.gecko.v1.mozilla-central.revision.linux.xxxx...
Reporter | ||
Comment 9•9 years ago
|
||
The regression range Raptor reports could across multiple pushes. So we can at first bisect pushes by exist builds, then bisect revisions by git/hg.
Comment 10•9 years ago
|
||
[Tracking Requested - why for this release]:
Comment 11•9 years ago
|
||
accident in comment 10. Reopen this.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Reporter | ||
Comment 12•9 years ago
|
||
It finds the correct culprit in bug 1200504!
Reporter | ||
Comment 13•9 years ago
|
||
In bug 1204837, the output of bisect bot is: Bisecting: bad changeset 262462:4968d9c21b40: bad The first bad revision is: changeset: 262462:4968d9c21b40 parent: 262461:0baf29d2d17a parent: 262451:c69e31de9aec user: Wes Kocher <wkocher@mozilla.com> date: Mon Sep 14 17:27:31 2015 -0700 summary: Merge m-c to b2ginbound, a=merge Not all ancestors of this changeset have been checked. Use bisect --extend to continue the bisection from the common ancestor, 6da01e6fcdf0. The command used: $ node index.js 0baf29d2d17a19aa 13d2e06d95f07770 47f0ac6cf4c4525c 02a15ec81e0bece9 raptor.sh communications/dialer 3600.000
Updated•9 years ago
|
Blocks: PerformanceProgram
Updated•9 years ago
|
Priority: -- → P3
Comment 14•6 years ago
|
||
Firefox OS is not being worked on
Status: REOPENED → RESOLVED
Closed: 9 years ago → 6 years ago
Resolution: --- → WONTFIX
You need to log in
before you can comment on or make changes to this bug.
Description
•