1042173 - mozregression 0.19 regressions

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Elbart]

Attachment: mozregression1.txt

1.)
Before, mozregression would fetch the needed revisions even when there's only one day between the good- and bad-dates.

Now it just crashes, see mozregression1.txt

2.)
After the Nightly-bisection is down to two dates, the "Ensuring we have enough metadata to get a pushlog..."-text is printed twice.

See mozregression1.txt lines 12 to 28 and mozregression2.txt lines 62 to 72.

===

I don't know if this is a problem or regression of mozregression or just another case of borked timestamps on the server:

In mozregression2.txt, the timestamps of the "good" builds are progressing from older to younger, as they should, but then the "good"-timestamp goes from 1400774845 to 1400776100 and finally back to 1400767310, resulting in a range with too many entries.

Comment 1

[Elbart]

Attachment: mozregression2.txt

Comment 2

[William Lachance (:wlach)]

Thanks for testing. :) I will try to look into these issues soon, if no one else does.

Comment 3

[Elbart]

Another oddity:

Trying

> mozregression --bits=32 --persist /c/temp --inbound --good-rev=a74600665875 -
> -bad-rev=35f3fa435d2c

with 0.19 either results in

> Retrieving valid builds from u'http://inbound-archive.pub.build.mozilla.org/pub/
> mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1405599037/' generate
> d an exception: 504 Server Error: Gateway Timeout

(the timestamp is never the same)

or

> Traceback (most recent call last):
>   File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
> dule>
>     load_entry_point('mozregression==0.19', 'console_scripts', 'mozregression')(
> )
>   File "build\bdist.win32\egg\mozregression\regression.py", line 311, in cli
>   File "build\bdist.win32\egg\mozregression\regression.py", line 108, in bisect_
> inbound
>   File "build\bdist.win32\egg\mozregression\inboundfinder.py", line 76, in get_b
> uild_infos
>   File "build\bdist.win32\egg\mozregression\inboundfinder.py", line 58, in _extr
> act_paths
>   File "build\bdist.win32\egg\mozregression\utils.py", line 66, in url_links
>   File "c:\mozilla-build\python\lib\site-packages\requests-2.2.1-py2.7.egg\reque
> sts\models.py", line 773, in raise_for_status
>     raise HTTPError(http_error_msg, response=self)
> requests.exceptions.HTTPError: 503 Server Error: Service Unavailable: Back-end s
> erver is at capacity

during the stage of collecting the timestamps.

At first I thought the servers are just thrashed, so I waited a day, but nothing changed.

After downgrading to 0.18, the bisection worked on the first try without error.

Going back to 0.19, and the errors reappear.

Comment 4

[William Lachance (:wlach)]

(In reply to Elbart from comment #3)
> Another oddity:
> 
> Trying
> 
> > mozregression --bits=32 --persist /c/temp --inbound --good-rev=a74600665875 -
> > -bad-rev=35f3fa435d2c
> 
> with 0.19 either results in
> 
> > Retrieving valid builds from u'http://inbound-archive.pub.build.mozilla.org/pub/
> > mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1405599037/' generate
> > d an exception: 504 Server Error: Gateway Timeout
> 
> (the timestamp is never the same)
> 
> or
> 
> > Traceback (most recent call last):
> >   File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
> > dule>
> >     load_entry_point('mozregression==0.19', 'console_scripts', 'mozregression')(
> > )
> >   File "build\bdist.win32\egg\mozregression\regression.py", line 311, in cli
> >   File "build\bdist.win32\egg\mozregression\regression.py", line 108, in bisect_
> > inbound
> >   File "build\bdist.win32\egg\mozregression\inboundfinder.py", line 76, in get_b
> > uild_infos
> >   File "build\bdist.win32\egg\mozregression\inboundfinder.py", line 58, in _extr
> > act_paths
> >   File "build\bdist.win32\egg\mozregression\utils.py", line 66, in url_links
> >   File "c:\mozilla-build\python\lib\site-packages\requests-2.2.1-py2.7.egg\reque
> > sts\models.py", line 773, in raise_for_status
> >     raise HTTPError(http_error_msg, response=self)
> > requests.exceptions.HTTPError: 503 Server Error: Service Unavailable: Back-end s
> > erver is at capacity
> 
> during the stage of collecting the timestamps.
> 
> At first I thought the servers are just thrashed, so I waited a day, but
> nothing changed.
> 
> After downgrading to 0.18, the bisection worked on the first try without
> error.
> 
> Going back to 0.19, and the errors reappear.

I could reproduce this as well, I think the problem was that we were trying to download too many metadata files at the same time. I just changed it upstream to only do 8 at a time and it seems to work. I'll do a quick release of 0.20 with the fix.

Comment 5

[Julien Pagès (:parkouss)]

Attachment: fix sorting of inbound builds found - Bug 1042173

I think this is also related. I probably made a mistake by not reordering inbound builds found when I refactored inboundfinder.py (now that threads are involved) - https://bugzilla.mozilla.org/show_bug.cgi?id=1030314.

I found this because I'm writing some new unit tests for the inboundfinder.py file - I will make a pull request for this soon.

I'm really sorry for the mistake - now I plan to write tests for mozregression (https://bugzilla.mozilla.org/show_bug.cgi?id=999039), to limit this kind of bugs in the future.

Comment 6

[William Lachance (:wlach)]

Comment on attachment 8465325 [details] [review]
fix sorting of inbound builds found - Bug 1042173

Looks good and seeing better behaviour now. Thanks! I'll release 0.21

Comment 7

[William Lachance (:wlach)]

Released 0.21 with the fix enabled.

Comment 8

[Elbart]

First two regressions are still happening, though:

>===============================================================================
$ mozregression --bits=32 -g 2014-07-10 -b 2014-07-12
Running nightly for 2014-07-11
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
014/07/2014-07-11-03-02-01-mozilla-central/firefox-33.0a1.en-US.win32.zip
===== Downloaded 100% =====
Installing nightly
Starting nightly (revision: e1a037c085d1)
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry
', or 'exit' and press Enter): g
Got as far as we can go bisecting nightlies...
Ensuring we have enough metadata to get a pushlog...
Getting http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/07/2014-07-1
2-03-02-02-mozilla-central/firefox-33.0a1.en-US.win32.txt
Last good revision: b0701d069bf9 (2014-07-11)
First bad build: 2014-07-12
Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2014-07-11&enddate=
2014-07-12

Ensuring we have enough metadata to get a pushlog...
Getting http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/07/2014-07-1
2-03-02-02-mozilla-central/firefox-33.0a1.en-US.win32.txt
Last good revision: b0701d069bf9 (2014-07-11)
First bad build: 2014-07-12
Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2014-07-11&enddate=
2014-07-12

... attempting to bisect inbound builds (starting from previous day, to make sur
e no inbound revision is missed)
Getting http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/07/2014-07-1
0-03-02-00-mozilla-central/firefox-33.0a1.en-US.win32.txt
Getting inbound builds between cb75d6cfb004 and None
Traceback (most recent call last):
  File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
dule>
    load_entry_point('mozregression==0.21', 'console_scripts', 'mozregression')(
)
  File "build\bdist.win32\egg\mozregression\regression.py", line 325, in cli
  File "build\bdist.win32\egg\mozregression\regression.py", line 205, in bisect_
nightlies
  File "build\bdist.win32\egg\mozregression\regression.py", line 176, in bisect_
nightlies
  File "build\bdist.win32\egg\mozregression\regression.py", line 108, in bisect_
inbound
  File "build\bdist.win32\egg\mozregression\inboundfinder.py", line 67, in get_b
uild_infos
  File "build\bdist.win32\egg\mozregression\inboundfinder.py", line 34, in get_p
ushlogs
  File "c:\mozilla-build\python\lib\site-packages\requests-2.2.1-py2.7.egg\reque
sts\models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found
>===============================================================================

Comment 9

[William Lachance (:wlach)]

Hmm, indeed, that case doesn't seem to work. I guess there's something else wrong...

Comment 10

[Julien Pagès (:parkouss)]

Attachment: Fix regression introduced by Bug 1029449 - Bug 1042173

Another regression introduced by me... I feel ashame; I hope this is the last one. :) Automated tests will definitively be usefull!

Comment 11

[William Lachance (:wlach)]

Comment on attachment 8465551 [details] [review]
Fix regression introduced by Bug 1029449 - Bug 1042173

I should have caught this in review. No worries, merged. Will do yet another release.

Comment 12

[Elbart]

Yes, 0.22 is now working fine.
The "Ensuring we have enough metadata to get a pushlog"-text is still shown twice, and I think I know why.

I don't know if I'm reading the changelog properly, though.

https://github.com/mozilla/mozregression/commit/f1adf45ae5d98245548d87e139e1c91e6a6d59ad
moved the printing of the metadata-printing outside the if-clause on June 9th, but it's still there from a commit one week earlier:
https://github.com/mozilla/mozregression/blame/5cd484ca0eef4a314e8049bc1e5639b61f82a031/mozregression/regression.py#L164

???

Comment 13

[Sam Garrett]

Hurda sent a PR for this here: https://github.com/mozilla/mozregression/pull/117

I merged it in. I think that wraps up the fixes for this bug.

Comment 14

[William Lachance (:wlach)]

Cool, thanks Sam. I released mozregression 0.23 with the fix:

https://pypi.python.org/pypi/mozregression