1000422 - Don't skip Nightly-respins

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Elbart]

I.e. doing --good 2013-12-08 --bad 2013-12-10 can't be done on Windows, because 2013-12-09-03-02-02-mozilla-central was compiled with wrong options as a debug-build, and it got a respin 2013-12-09-05-34-02-mozilla-central which is not considered by the script.

Comment 1

[Sam Garrett]

@wlachance it looks like the issue is here: https://github.com/sinemetu1/mozregression/blob/master/mozregression/runnightly.py#L138 We're returning the first match to the BuildRegex, but when they are respun there can be multiple matches.

I've fixed this and the PR is here: https://github.com/mozilla/mozregression/pull/97

Functionality was added for trying the next build if there are any on the same date when the build is marked as "bad". That is handled here: https://github.com/sinemetu1/mozregression/blob/issue1000422-dont-skip-nightly-respins/mozregression/regression.py#L184

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

(In reply to Sam Garrett from comment #1)
> https://github.com/sinemetu1/mozregression/blob/master/mozregression/
> runnightly.py#L138 We're returning the first match to the BuildRegex, but
> when they are respun there can be multiple matches.

Right. This is still bug 703559, and we don't know when it will be fixed.

Comment 3

[William Lachance (:wlach)]

(In reply to Sam Garrett from comment #1)
> @wlachance it looks like the issue is here:
> https://github.com/sinemetu1/mozregression/blob/master/mozregression/
> runnightly.py#L138 We're returning the first match to the BuildRegex, but
> when they are respun there can be multiple matches.
> 
> I've fixed this and the PR is here:
> https://github.com/mozilla/mozregression/pull/97
> 
> Functionality was added for trying the next build if there are any on the
> same date when the build is marked as "bad". That is handled here:
> https://github.com/sinemetu1/mozregression/blob/issue1000422-dont-skip-
> nightly-respins/mozregression/regression.py#L184

Hey Sam, I think this is on the right track... but wouldn't a better strategy be to try to find the *latest* respin and always use that? Forcing the user to try builds that we know are bad doesn't seem right to me.

Also, could you mark your pull requests for review here? That'll make sure they go into my reminder queue. :) See: http://globau.wordpress.com/2013/10/21/github-pull-requests-and-bugzilla/

Comment 4

[Sam Garrett]

I was thinking that the windows build was the only bad build in the "first" build. All builds are bad then?

In that case we could probably remove the retry functionality and just take the most recent build in the list.

Comment 5

[William Lachance (:wlach)]

(In reply to Sam Garrett from comment #4)
> I was thinking that the windows build was the only bad build in the "first"
> build. All builds are bad then?
> 
> In that case we could probably remove the retry functionality and just take
> the most recent build in the list.

I'm actually not sure of the exact details of how nightly builds are generated, but I think it's safe to assume that any builds except the latest are bad (or at least not worth trying) on the nightly granularity. Certainly I think it's the intention that anyone downloading nightlies through other means would grab the latest. I'll needinfo bhearsum to confirm.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to William Lachance (:wlach) from comment #5)
> (In reply to Sam Garrett from comment #4)
> > I was thinking that the windows build was the only bad build in the "first"
> > build. All builds are bad then?
> > 
> > In that case we could probably remove the retry functionality and just take
> > the most recent build in the list.
> 
> I'm actually not sure of the exact details of how nightly builds are
> generated, but I think it's safe to assume that any builds except the latest
> are bad (or at least not worth trying) on the nightly granularity. Certainly
> I think it's the intention that anyone downloading nightlies through other
> means would grab the latest. I'll needinfo bhearsum to confirm.

I'm not exactly certain of the context here, but it doesn't sound right to me to assume that non-latest nightlies are "bad" (I'm curious what "bad" means here, though). Any nightly that you find a dated dir (eg, http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/04/2014-04-01-03-02-03-mozilla-central/) was the "latest" nightly at some point, and thus shipped to users as a good build.

Comment 7

[William Lachance (:wlach)]

Discussed this on irc. For the purposes of mozregression in think we can *only* test the latest respin of a nightly and get the results we'e looking for.

<wlach> bhearsum: I guess my question in https://bugzilla.mozilla.org/show_bug.cgi?id=1000422#c6 was more "is there any reason to try the not-latest nightly respin for that day when bisecting for regressions?"?
<bhearsum> wlach: what do you mean by "non-latest nightly respin"?
<bhearsum> you're specifically talking about cases where we had a failed nightly and a successful one?
<wlach> bhearsum: I guess I don't understand all the circumstances that could lead to there being multiple nightlies. But all I need for the purposes of mozregression is that abilility to answer the question "does a build on this day exhibit a problem?"
<wlach> bhearsum: would there be any reason to try running any nightly but the latest respin to answer that question?
<bhearsum> wlach: if possible, you probably want to be revision-oriented
<bhearsum> we can have multiple nightlies anytime there's something important enough landing on m-c
<pmoore> Callek: it would be helpful for me to go through some stuff with hwine-ooo now - would you be able to help with the pending linux jobs?
<bhearsum> and sometimes we have multiple ones beacuse the first one failed in some way
<wlach> bhearsum: once we've narrowed the regression range to one day, we bisect on inbound which is more revision oriented
<bhearsum> wlach: probably doesn't matter which one you pick for that case
<wlach> bhearsum: well it sounds like we want to pick up the latest one
<wlach> bhearsum: because if we don't pick the latest we might get a build with problems
<wlach> bhearsum: and there's no point in asking people to test a potentially problematic build
<bhearsum> sure
<wlach> bhearsum: ok, I'll comment in the bug, thanks

Comment 8

[William Lachance (:wlach)]

Hey Sam, is the above enough information for you to proceed? Let me know if I can get further clarifications for you.

Comment 9

[Sam Garrett]

(In reply to William Lachance (:wlach) from comment #8)
> Hey Sam, is the above enough information for you to proceed? Let me know if
> I can get further clarifications for you.

This is enough information. I'll send another PR tonight.

Comment 10

[Sam Garrett]

Attachment: Fixed Bug 1000422 - Always use latest build in case of respins

Comment 11

[William Lachance (:wlach)]

Pushed, thanks!

https://github.com/mozilla/mozregression/commit/042a1dcedf10e3ab66bf74c60be12c9083633938

Comment 12

[Elbart]

Now 0.17 is downloading the wrong file within the correct (respin-)folder:

$ mozregression --bits=32 -g 2013-12-08 -b 2013-12-10
Running nightly for 2013-12-09
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
013/12/2013-12-09-05-34-02-mozilla-central/jsshell-win32.zip
===== Downloaded 100% =====
Installing nightly
Traceback (most recent call last):
  File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
dule>
    load_entry_point('mozregression==0.17', 'console_scripts', 'mozregression')(
)
  File "build\bdist.win32\egg\mozregression\regression.py", line 276, in cli
  File "build\bdist.win32\egg\mozregression\regression.py", line 167, in bisect_
nightlies
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 274, in start
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 271, in install

  File "build\bdist.win32\egg\mozregression\runnightly.py", line 107, in install

  File "c:\mozilla-build\python\lib\site-packages\mozinstall-1.9-py2.7.egg\mozin
stall\mozinstall.py", line 89, in get_binary
    raise InvalidBinary('"%s" does not contain a valid binary.' % path)
mozinstall.mozinstall.InvalidBinary: "c:\users\user\appdata\local\temp\tmpxogvjp
\js.exe" does not contain a valid binary.

Comment 13

[William Lachance (:wlach)]

That does sound like a problem. :)

Comment 14

[Elbart]

When changing the sorting of the folder to "last modified", like
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/12/2013-12-09-05-34-02-mozilla-central/?C=M;O=A

jsshell-win32.zip becomes the last (newest) file ending with "win32.zip".
If I'm reading the code correctly, that's the only part mozreg is caring about:
https://github.com/mozilla/mozregression/blob/042a1dcedf10e3ab66bf74c60be12c9083633938/mozregression/runnightly.py#L42

Maybe adding a check for ^firefox-, ^fennec- and whatever else there is will fix this bug?

There was a similar issue filed in the github, but it's not accessible anymore:
https://github.com/mozilla/mozregression/issues/20

http://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fgithub.com%2Fmozilla%2Fmozregression%2Fissues%2F20

Comment 15

[Sam Garrett]

Attachment: added more complete regex for finding app specific builds

This filters out by app name now. Let me know if you have any issues with the @staticmethod I added.

Comment 16

[William Lachance (:wlach)]

Yup, that fix worked, thanks! Merged via pull request, will spin a new version.

Comment 17

[William Lachance (:wlach)]

FYI, just uploaded mozregression 0.18 to pypi with this fix included.

Comment 18

[Elbart]

Still not fixed, I'm afraid.

$ mozregression --bits=32 -g 2010-01-01
Running nightly for 2012-03-26
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
012/03/2012-03-26-03-11-49-mozilla-central/firefox-14.0a1.en-US.win32.zip.asc
===== Downloaded 8668% =====
Installing nightly
Traceback (most recent call last):
  File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
dule>
    load_entry_point('mozregression==0.18', 'console_scripts', 'mozregression')(
)
  File "build\bdist.win32\egg\mozregression\regression.py", line 276, in cli
  File "build\bdist.win32\egg\mozregression\regression.py", line 167, in bisect_
nightlies
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 281, in start
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 278, in install

  File "build\bdist.win32\egg\mozregression\runnightly.py", line 114, in install

  File "c:\mozilla-build\python\lib\site-packages\mozinstall-1.9-py2.7.egg\mozin
stall\mozinstall.py", line 109, in install
    '(zip, exe, tar.gz, tar.bz2 or dmg)')
mozinstall.mozinstall.InvalidSource: C:\Users\user\firefox-14.0a1.en-US.win32.zi
p.asc is not a recognized file type (zip, exe, tar.gz, tar.bz2 or dmg)

Comment 19

[Sam Garrett]

(In reply to Elbart from comment #18)
> Still not fixed, I'm afraid.
> 
> $ mozregression --bits=32 -g 2010-01-01
> Running nightly for 2012-03-26
> Downloading build from:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
> 012/03/2012-03-26-03-11-49-mozilla-central/firefox-14.0a1.en-US.win32.zip.asc
> ===== Downloaded 8668% =====
> Installing nightly
> Traceback (most recent call last):
>   File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in
> <mo
> dule>
>     load_entry_point('mozregression==0.18', 'console_scripts',
> 'mozregression')(
> )
>   File "build\bdist.win32\egg\mozregression\regression.py", line 276, in cli
>   File "build\bdist.win32\egg\mozregression\regression.py", line 167, in
> bisect_
> nightlies
>   File "build\bdist.win32\egg\mozregression\runnightly.py", line 281, in
> start
>   File "build\bdist.win32\egg\mozregression\runnightly.py", line 278, in
> install
> 
>   File "build\bdist.win32\egg\mozregression\runnightly.py", line 114, in
> install
> 
>   File
> "c:\mozilla-build\python\lib\site-packages\mozinstall-1.9-py2.7.egg\mozin
> stall\mozinstall.py", line 109, in install
>     '(zip, exe, tar.gz, tar.bz2 or dmg)')
> mozinstall.mozinstall.InvalidSource:
> C:\Users\user\firefox-14.0a1.en-US.win32.zi
> p.asc is not a recognized file type (zip, exe, tar.gz, tar.bz2 or dmg)

This is filed as a separate issue at https://bugzilla.mozilla.org/show_bug.cgi?id=1031915. It should be fixed now.

Comment 20

[Elbart]

Still only 0.18 available, which is buggy.

Comment 21

[William Lachance (:wlach)]

(In reply to Elbart from comment #20)
> Still only 0.18 available, which is buggy.

Just uploaded 0.19

Comment 22

[Elbart]

Thanks, noted regressions in Bug 1042173