Don't skip Nightly-respins

RESOLVED FIXED

Status

Testing
mozregression
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: Elbart, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
I.e. doing --good 2013-12-08 --bad 2013-12-10 can't be done on Windows, because 2013-12-09-03-02-02-mozilla-central was compiled with wrong options as a debug-build, and it got a respin 2013-12-09-05-34-02-mozilla-central which is not considered by the script.

Comment 1

4 years ago
@wlachance it looks like the issue is here: https://github.com/sinemetu1/mozregression/blob/master/mozregression/runnightly.py#L138 We're returning the first match to the BuildRegex, but when they are respun there can be multiple matches.

I've fixed this and the PR is here: https://github.com/mozilla/mozregression/pull/97

Functionality was added for trying the next build if there are any on the same date when the build is marked as "bad". That is handled here: https://github.com/sinemetu1/mozregression/blob/issue1000422-dont-skip-nightly-respins/mozregression/regression.py#L184
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
(In reply to Sam Garrett from comment #1)
> https://github.com/sinemetu1/mozregression/blob/master/mozregression/
> runnightly.py#L138 We're returning the first match to the BuildRegex, but
> when they are respun there can be multiple matches.

Right. This is still bug 703559, and we don't know when it will be fixed.
(In reply to Sam Garrett from comment #1)
> @wlachance it looks like the issue is here:
> https://github.com/sinemetu1/mozregression/blob/master/mozregression/
> runnightly.py#L138 We're returning the first match to the BuildRegex, but
> when they are respun there can be multiple matches.
> 
> I've fixed this and the PR is here:
> https://github.com/mozilla/mozregression/pull/97
> 
> Functionality was added for trying the next build if there are any on the
> same date when the build is marked as "bad". That is handled here:
> https://github.com/sinemetu1/mozregression/blob/issue1000422-dont-skip-
> nightly-respins/mozregression/regression.py#L184

Hey Sam, I think this is on the right track... but wouldn't a better strategy be to try to find the *latest* respin and always use that? Forcing the user to try builds that we know are bad doesn't seem right to me.

Also, could you mark your pull requests for review here? That'll make sure they go into my reminder queue. :) See: http://globau.wordpress.com/2013/10/21/github-pull-requests-and-bugzilla/
Flags: needinfo?(samdgarrett)

Comment 4

4 years ago
I was thinking that the windows build was the only bad build in the "first" build. All builds are bad then?

In that case we could probably remove the retry functionality and just take the most recent build in the list.
Flags: needinfo?(samdgarrett)
(In reply to Sam Garrett from comment #4)
> I was thinking that the windows build was the only bad build in the "first"
> build. All builds are bad then?
> 
> In that case we could probably remove the retry functionality and just take
> the most recent build in the list.

I'm actually not sure of the exact details of how nightly builds are generated, but I think it's safe to assume that any builds except the latest are bad (or at least not worth trying) on the nightly granularity. Certainly I think it's the intention that anyone downloading nightlies through other means would grab the latest. I'll needinfo bhearsum to confirm.
Flags: needinfo?(bhearsum)
(In reply to William Lachance (:wlach) from comment #5)
> (In reply to Sam Garrett from comment #4)
> > I was thinking that the windows build was the only bad build in the "first"
> > build. All builds are bad then?
> > 
> > In that case we could probably remove the retry functionality and just take
> > the most recent build in the list.
> 
> I'm actually not sure of the exact details of how nightly builds are
> generated, but I think it's safe to assume that any builds except the latest
> are bad (or at least not worth trying) on the nightly granularity. Certainly
> I think it's the intention that anyone downloading nightlies through other
> means would grab the latest. I'll needinfo bhearsum to confirm.

I'm not exactly certain of the context here, but it doesn't sound right to me to assume that non-latest nightlies are "bad" (I'm curious what "bad" means here, though). Any nightly that you find a dated dir (eg, http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/04/2014-04-01-03-02-03-mozilla-central/) was the "latest" nightly at some point, and thus shipped to users as a good build.
Flags: needinfo?(bhearsum)
Discussed this on irc. For the purposes of mozregression in think we can *only* test the latest respin of a nightly and get the results we'e looking for.

<wlach> bhearsum: I guess my question in https://bugzilla.mozilla.org/show_bug.cgi?id=1000422#c6 was more "is there any reason to try the not-latest nightly respin for that day when bisecting for regressions?"?
<bhearsum> wlach: what do you mean by "non-latest nightly respin"?
<bhearsum> you're specifically talking about cases where we had a failed nightly and a successful one?
<wlach> bhearsum: I guess I don't understand all the circumstances that could lead to there being multiple nightlies. But all I need for the purposes of mozregression is that abilility to answer the question "does a build on this day exhibit a problem?"
<wlach> bhearsum: would there be any reason to try running any nightly but the latest respin to answer that question?
<bhearsum> wlach: if possible, you probably want to be revision-oriented
<bhearsum> we can have multiple nightlies anytime there's something important enough landing on m-c
<pmoore> Callek: it would be helpful for me to go through some stuff with hwine-ooo now - would you be able to help with the pending linux jobs?
<bhearsum> and sometimes we have multiple ones beacuse the first one failed in some way
<wlach> bhearsum: once we've narrowed the regression range to one day, we bisect on inbound which is more revision oriented
<bhearsum> wlach: probably doesn't matter which one you pick for that case
<wlach> bhearsum: well it sounds like we want to pick up the latest one
<wlach> bhearsum: because if we don't pick the latest we might get a build with problems
<wlach> bhearsum: and there's no point in asking people to test a potentially problematic build
<bhearsum> sure
<wlach> bhearsum: ok, I'll comment in the bug, thanks
Hey Sam, is the above enough information for you to proceed? Let me know if I can get further clarifications for you.
Flags: needinfo?(samdgarrett)

Comment 9

4 years ago
(In reply to William Lachance (:wlach) from comment #8)
> Hey Sam, is the above enough information for you to proceed? Let me know if
> I can get further clarifications for you.

This is enough information. I'll send another PR tonight.
Flags: needinfo?(samdgarrett)

Comment 10

4 years ago
Created attachment 8416267 [details] [review]
Fixed Bug 1000422 - Always use latest build in case of respins
Attachment #8416267 - Flags: review?(wlachance)
Pushed, thanks!

https://github.com/mozilla/mozregression/commit/042a1dcedf10e3ab66bf74c60be12c9083633938
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Attachment #8416267 - Flags: review?(wlachance) → review+
(Reporter)

Comment 12

4 years ago
Now 0.17 is downloading the wrong file within the correct (respin-)folder:

$ mozregression --bits=32 -g 2013-12-08 -b 2013-12-10
Running nightly for 2013-12-09
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
013/12/2013-12-09-05-34-02-mozilla-central/jsshell-win32.zip
===== Downloaded 100% =====
Installing nightly
Traceback (most recent call last):
  File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
dule>
    load_entry_point('mozregression==0.17', 'console_scripts', 'mozregression')(
)
  File "build\bdist.win32\egg\mozregression\regression.py", line 276, in cli
  File "build\bdist.win32\egg\mozregression\regression.py", line 167, in bisect_
nightlies
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 274, in start
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 271, in install

  File "build\bdist.win32\egg\mozregression\runnightly.py", line 107, in install

  File "c:\mozilla-build\python\lib\site-packages\mozinstall-1.9-py2.7.egg\mozin
stall\mozinstall.py", line 89, in get_binary
    raise InvalidBinary('"%s" does not contain a valid binary.' % path)
mozinstall.mozinstall.InvalidBinary: "c:\users\user\appdata\local\temp\tmpxogvjp
\js.exe" does not contain a valid binary.
That does sound like a problem. :)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 14

4 years ago
When changing the sorting of the folder to "last modified", like
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/12/2013-12-09-05-34-02-mozilla-central/?C=M;O=A

jsshell-win32.zip becomes the last (newest) file ending with "win32.zip".
If I'm reading the code correctly, that's the only part mozreg is caring about:
https://github.com/mozilla/mozregression/blob/042a1dcedf10e3ab66bf74c60be12c9083633938/mozregression/runnightly.py#L42

Maybe adding a check for ^firefox-, ^fennec- and whatever else there is will fix this bug?

There was a similar issue filed in the github, but it's not accessible anymore:
https://github.com/mozilla/mozregression/issues/20

http://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fgithub.com%2Fmozilla%2Fmozregression%2Fissues%2F20

Comment 15

4 years ago
Created attachment 8420570 [details] [review]
added more complete regex for finding app specific builds

This filters out by app name now. Let me know if you have any issues with the @staticmethod I added.
Attachment #8420570 - Flags: review?(wlachance)
Yup, that fix worked, thanks! Merged via pull request, will spin a new version.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
Attachment #8420570 - Flags: review?(wlachance) → review+
FYI, just uploaded mozregression 0.18 to pypi with this fix included.
(Reporter)

Comment 18

4 years ago
Still not fixed, I'm afraid.

$ mozregression --bits=32 -g 2010-01-01
Running nightly for 2012-03-26
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
012/03/2012-03-26-03-11-49-mozilla-central/firefox-14.0a1.en-US.win32.zip.asc
===== Downloaded 8668% =====
Installing nightly
Traceback (most recent call last):
  File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in <mo
dule>
    load_entry_point('mozregression==0.18', 'console_scripts', 'mozregression')(
)
  File "build\bdist.win32\egg\mozregression\regression.py", line 276, in cli
  File "build\bdist.win32\egg\mozregression\regression.py", line 167, in bisect_
nightlies
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 281, in start
  File "build\bdist.win32\egg\mozregression\runnightly.py", line 278, in install

  File "build\bdist.win32\egg\mozregression\runnightly.py", line 114, in install

  File "c:\mozilla-build\python\lib\site-packages\mozinstall-1.9-py2.7.egg\mozin
stall\mozinstall.py", line 109, in install
    '(zip, exe, tar.gz, tar.bz2 or dmg)')
mozinstall.mozinstall.InvalidSource: C:\Users\user\firefox-14.0a1.en-US.win32.zi
p.asc is not a recognized file type (zip, exe, tar.gz, tar.bz2 or dmg)

Comment 19

3 years ago
(In reply to Elbart from comment #18)
> Still not fixed, I'm afraid.
> 
> $ mozregression --bits=32 -g 2010-01-01
> Running nightly for 2012-03-26
> Downloading build from:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2
> 012/03/2012-03-26-03-11-49-mozilla-central/firefox-14.0a1.en-US.win32.zip.asc
> ===== Downloaded 8668% =====
> Installing nightly
> Traceback (most recent call last):
>   File "c:\mozilla-build\python\Scripts\mozregression-script.py", line 8, in
> <mo
> dule>
>     load_entry_point('mozregression==0.18', 'console_scripts',
> 'mozregression')(
> )
>   File "build\bdist.win32\egg\mozregression\regression.py", line 276, in cli
>   File "build\bdist.win32\egg\mozregression\regression.py", line 167, in
> bisect_
> nightlies
>   File "build\bdist.win32\egg\mozregression\runnightly.py", line 281, in
> start
>   File "build\bdist.win32\egg\mozregression\runnightly.py", line 278, in
> install
> 
>   File "build\bdist.win32\egg\mozregression\runnightly.py", line 114, in
> install
> 
>   File
> "c:\mozilla-build\python\lib\site-packages\mozinstall-1.9-py2.7.egg\mozin
> stall\mozinstall.py", line 109, in install
>     '(zip, exe, tar.gz, tar.bz2 or dmg)')
> mozinstall.mozinstall.InvalidSource:
> C:\Users\user\firefox-14.0a1.en-US.win32.zi
> p.asc is not a recognized file type (zip, exe, tar.gz, tar.bz2 or dmg)

This is filed as a separate issue at https://bugzilla.mozilla.org/show_bug.cgi?id=1031915. It should be fixed now.
(Reporter)

Comment 20

3 years ago
Still only 0.18 available, which is buggy.
(In reply to Elbart from comment #20)
> Still only 0.18 available, which is buggy.

Just uploaded 0.19
(Reporter)

Comment 22

3 years ago
Thanks, noted regressions in Bug 1042173
You need to log in before you can comment on or make changes to this bug.