Closed Bug 1342952 Opened 9 years ago Closed 9 years ago

No Linux nightlies for Fx 54.0a1 after 2017-02-25

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(firefox53 unaffected, firefox54 verified, firefox55 unaffected)

Status:
VERIFIED FIXED
Milestone:
mozilla54
Tracking Flags:
Tracking Status
firefox53 --- unaffected
firefox54 --- verified
firefox55 --- unaffected

People

(Reporter: tonymec, Assigned: mtabara)

References

(
URL
)

Details

Latest L32 and L64 nightlies on ftp.m.o for Fx 54.0a1 are dated 2017-02-25. W32/64 and Mac builds are still being published. :mtabara: Callek said over IRC that you're to be in charge of this problem.
Summary: No Linux nightlies for Fx 54.0a1 after 2017-01-25 → No Linux nightlies for Fx 54.0a1 after 2017-02-25
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #0) > Latest L32 and L64 nightlies on ftp.m.o for Fx 54.0a1 are dated 2017-02-25. > W32/64 and Mac builds are still being published. > > :mtabara: Callek said over IRC that you're to be in charge of this problem. Yeah, it's a change I landed on Friday which got merged into central over the week-end that I suspect is the culprit. I'll have a look and try to unblock as soon as I can. Sorry about this.
Depends on: 1338186
Fixed the issue in bug 1338186. Manually triggered nightlies for Linux and Android platforms. Looking good so far. https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=cc0d34f0490c9dab0fda35765683c31d3a4fc950
No longer depends on: 1338186
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2) > Fixed the issue in bug 1338186. Manually triggered nightlies for Linux and > Android platforms. Looking good so far. > https://treeherder.mozilla.org/#/jobs?repo=mozilla- > central&revision=cc0d34f0490c9dab0fda35765683c31d3a4fc950 A Linux64 nightly has now appeared. Linux32 hasn't yet but I suppose it takes longer to build.
UA:"Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0" (en-US) ID:20170227203520 Built from https://hg.mozilla.org/mozilla-central/rev/cc0d34f0490c9dab0fda35765683c31d3a4fc950 This is the new Fx nightly. As can be seen, it installs and runs normally AFAICT.
While I was installing the L64 build, a L32 build has also appeared. Mihai, I don't know if you prefer to RESOLVE this as FIXED (in mozilla-central changeset cc0d34f0490c9dab0fda35765683c31d3a4fc950 ), WORKSFORME, or even DUPLICATE of bug 1338186, but I'll mark it VERIFIED. Or maybe we should wait until the nightly for tomorrow (2017-02-28) gets built at its usual appointed time.
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #5) > While I was installing the L64 build, a L32 build has also appeared. Mihai, > I don't know if you prefer to RESOLVE this as FIXED (in mozilla-central > changeset cc0d34f0490c9dab0fda35765683c31d3a4fc950 ), WORKSFORME, or even > DUPLICATE of bug 1338186, but I'll mark it VERIFIED. Or maybe we should wait > until the nightly for tomorrow (2017-02-28) gets built at its usual > appointed time. You're right, let's wait until tomorrow to make sure all is good. It's not actually a duplicate of bug 1338186 as more like a side-effect of a large set of changes that I'm pursuing there so let's mark it as FIXED/VERIFIED, either works, when the time comes. Thanks!
Well I certainly don't think it should be marked as verified yet. As far as I know a new nightly was triggered last night after the fix and yet this URL https://download.mozilla.org?product=firefox-nightly-latest&os=linux64&lang=en-US is still returning the 2017-02-25 nightly.
http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-27-20-35-20-mozilla-central/ seems to be the latest for linux x64 - maybe a problem with linking this builds ?
Weird, I see balrog is serving the 2017-02-28. That nightly is also under http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/, specifically http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-54.0a1.en-US.linux-x86_64.complete.mar Not sure how that link is still stalled.
I checked and firefox-54.0a1.en-US.linux-x86_64.tar.bz2 is also getting the correct build, which is odd because in the past usually if https://download.mozilla.org?product=firefox-nightly-latest&os=linux64&lang=en-US is incorrect that one is also.
(In reply to Carsten Book [:Tomcat] from comment #9) > http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-27-20-35-20- > mozilla-central/ seems to be the latest for linux x64 - maybe a problem with > linking this builds ? So the URL that does not work correctly ends up redirecting to http://download.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-central/firefox-54.0a1.en-US.linux-x86_64.tar.bz2 and that is wherin the issue lies. Somehow that is NOT the latest build.
(In reply to Bill Gianopoulos [:WG9s] from comment #12) > (In reply to Carsten Book [:Tomcat] from comment #9) > > http://ftp.mozilla.org/pub/firefox/nightly/2017/02/2017-02-27-20-35-20- > > mozilla-central/ seems to be the latest for linux x64 - maybe a problem with > > linking this builds ? > > So the URL that does not work correctly ends up redirecting to > http://download.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-central/ > firefox-54.0a1.en-US.linux-x86_64.tar.bz2 > > and that is wherin the issue lies. Somehow that is NOT the latest build. It seems like the file was written last night (hence today 28-02-2017): firefox-54.0a1.en-US.linux-x86_64.tar.bz2 63M 28-Feb-2017 12:11 How do you check that is not the lattest build?
This is the output from the command in the script I use to download the latest build: + wget -N --read-timeout=120 'https://download.mozilla.org?product=firefox-nightly-latest&os=linux64&lang=en-US' --2017-02-28 08:13:51-- https://download.mozilla.org/?product=firefox-nightly-latest&os=linux64&lang=en-US Resolving download.mozilla.org (download.mozilla.org)... 52.54.253.31 Connecting to download.mozilla.org (download.mozilla.org)|52.54.253.31|:443... connected. HTTP request sent, awaiting response... 302 Found Location: http://download.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-central/firefox-54.0a1.en-US.linux-x86_64.tar.bz2 [following] --2017-02-28 08:13:52-- http://download.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-central/firefox-54.0a1.en-US.linux-x86_64.tar.bz2 Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... 65.202.58.65, 65.202.58.41, 2600:1408:1d::1734:8f52, ... Connecting to download.cdn.mozilla.net (download.cdn.mozilla.net)|65.202.58.65|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 67019810 (64M) [application/x-tar] Saving to: ‘index.html?product=firefox-nightly-latest&os=linux64&lang=en-US’ index.html?product= 100%[===================>] 63.92M 2.27MB/s in 30s 2017-02-28 08:14:22 (2.14 MB/s) - ‘index.html?product=firefox-nightly-latest&os=linux64&lang=en-US’ saved [67019810/67019810] and these are the timestamps on the file that results: -rw-rw-r--. 1 wag wag 67019810 Feb 25 07:04 'index.html?product=firefox-nightly-latest&os=linux64&lang=en-US' All times here are Eastern.
I just reran my script and verified I still go the wrong build. Then I did an nslookup on archive.mozilla.org and creating a /etc/hosts file entry with that address for download.cdn.mozilla.net and reran my script and got the 2017-02-28 nightly. Evidently it is taking an inordinate amount of time for the links to update to all the cdn download systems. This also seems to have started around last Friday. Before then I always got the aurora build by around 7AM eastern and the Mozilla-central build by 9AM eastern.
Hm, I wonder whether the clocks of the Linux builders are set to the same timezone as the others' (see the respective build IDs in comment #16).
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #17) > Hm, I wonder whether the clocks of the Linux builders are set to the same > timezone as the others' (see the respective build IDs in comment #16). Ah I bet that has a bit to do with the issue. I wonder if the time is just wrong and not in the wrong timezone. that might explain why the files don;t mirror. I wonder if files dated in the future are rejected.
Myself, my build systems are in the US/Eastern timezone, but to keep the buildids consistent with the official nightlies, I do a "export TZ=PST8PDT,M3.2.0,M11,1,0" in my build procedure.
(In reply to Bill Gianopoulos [:WG9s] from comment #19) > Myself, my build systems are in the US/Eastern timezone, but to keep the > buildids consistent with the official nightlies, I do a "export > TZ=PST8PDT,M3.2.0,M11,1,0" in my build procedure. ^^^^^^^^^^^^^^^^^^^^^^^^^ TZ=PST8PDT,M3.2.0,M11.1.0
I'm in Continental Europe (Mozilla time + 9 hours except for a short time near the equinoxes when Summer Time and Daylight Savings Time don't change at the same date) and I usually see Firefox nightlies appear between noon and 4PM my time.
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #17) > Hm, I wonder whether the clocks of the Linux builders are set to the same > timezone as the others' (see the respective build IDs in comment #16). So according to the buildids and file timestamps the uploaded files were created almost 7 hours before the build began.
The nightlies that I manually triggered last night after the fix, were just Linux/Android. Not sure if this helps.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #23) > The nightlies that I manually triggered last night after the fix, were just > Linux/Android. Not sure if this helps. The issue we are seeing now, that might be at the heart of the issue, is that the buildids of the official nightlies from today are 8 hours in the future of actual time.
(In reply to Bill Gianopoulos [:WG9s] from comment #24) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #23) > > The nightlies that I manually triggered last night after the fix, were just > > Linux/Android. Not sure if this helps. > > The issue we are seeing now, that might be at the heart of the issue, is > that the buildids of the official nightlies from today are 8 hours in the > future of actual time. Hohoh..., i.e. British time zone, which happens to be Mihai's.
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #25) > (In reply to Bill Gianopoulos [:WG9s] from comment #24) > > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #23) > > > The nightlies that I manually triggered last night after the fix, were just > > > Linux/Android. Not sure if this helps. > > > > The issue we are seeing now, that might be at the heart of the issue, is > > that the buildids of the official nightlies from today are 8 hours in the > > future of actual time. > > Hohoh..., i.e. British time zone, which happens to be Mihai's. Or more likely is the system generating the BuildID is set in UTC, while the buildbot builds are setting buildid in PST, which is 8 hours off.
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #25) > (In reply to Bill Gianopoulos [:WG9s] from comment #24) > > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #23) > > > The nightlies that I manually triggered last night after the fix, were just > > > Linux/Android. Not sure if this helps. > > > > The issue we are seeing now, that might be at the heart of the issue, is > > that the buildids of the official nightlies from today are 8 hours in the > > future of actual time. > > Hohoh..., i.e. British time zone, which happens to be Mihai's. or maybe they are just set to GMT, which is the Linux xustom, instead of local (California) time as per Windows custom. I don't know the Mac custom, but considering that Mac OS X is Unix-like, I suppose the Mac builders' internal clock is set to GMT but corrected according to timezone -0800 for display.
@Justin: mid-air collision.
My understanding of how the official nightly builds are intended to work is that even accounting for incorrect timezone configuration, that would still leave a discrepancy between the correct buildid of 20170228030203 and 20170228030308. It is my understanding that all official Firefox mozilla-central nightly builds for all platforms are supposed to be triggered by a single source that provides both the changset to use for checkout and the timestamp to use for the buildid.
i.e. the normally triggered at 3:02 Mozilla time builds are all supposed to be off the same changeset and have identical buildids.
In any event though I think this issue has been fixed and the buildid issue should be a new bug. I would file it myself if I had a clue what the correct component might be.
I filed Bug 1343339 on this issue.
(In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my attention) from comment #27) > (In reply to Tony Mechelynck [:tonymec]. (NEEDINFO me if you want my > attention) from comment #25) > > (In reply to Bill Gianopoulos [:WG9s] from comment #24) > > > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #23) > > > > The nightlies that I manually triggered last night after the fix, were just > > > > Linux/Android. Not sure if this helps. > > > > > > The issue we are seeing now, that might be at the heart of the issue, is > > > that the buildids of the official nightlies from today are 8 hours in the > > > future of actual time. > > > > Hohoh..., i.e. British time zone, which happens to be Mihai's. > > or maybe they are just set to GMT, which is the Linux xustom, instead of > local (California) time as per Windows custom. I don't know the Mac custom, > but considering that Mac OS X is Unix-like, I suppose the Mac builders' > internal clock is set to GMT but corrected according to timezone -0800 for > display. The buildid on official builds is supposed to be in mozilla standard time. not GMT or UTC or any other locale.
So since the other bug was closed as invalid the bug stops here. There still an issue. builds are not available until several hours later that they should be via advertised URLs
I tried to file a bug on that issue and it got closed as INVALID.
I guess they thought it was FAKE NEWS!
I guess we can close this.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #37) > I guess we can close this. Ah, oh, yeah; and then there was even a merge
Status: RESOLVED → VERIFIED
status-firefox53: --- → unaffected
status-firefox54: affected → verified
status-firefox55: --- → unaffected
Target Milestone: --- → Firefox 54
Component: Build Config → General
Product: Firefox → Firefox Build System
Target Milestone: Firefox 54 → mozilla54
You need to log in before you can comment on or make changes to this bug.