Closed Bug 1222878 Opened 9 years ago Closed 8 years ago

comm-central-l10n and comm-aurora-l10n builds are not updated after the s3 migration: 'get mar' step fails

Categories

(Release Engineering :: General, defect)

defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: t.matsuu, Assigned: aleth)

References

Details

Attachments

(3 files, 3 obsolete files)

For building mozilla-central-l10n, mar.exe is downloaded from ftp.mozilla.org.

From https://archive.mozilla.org/pub/firefox/nightly/2015/12/2015-12-28-03-02-13-mozilla-central-l10n/mozilla-central-win32-l10n-nightly-1-unknown-bm74-build1-build12.txt.gz:
08:00:33     INFO - Downloading http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/mar-tools/win32/mar.exe to c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\build\mozilla-central\obj-l10n\dist\host\bin\mar.exe
Searching for productdelivery in mxr-build it seems that Thunderbird is using staging servers for the l10n dep builds. Reading bug 1221391 it seems that some changes are required to get l10n builds running. It works in Firefox because that uses mozharness.

https://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#3493 says that mar tools are downloaded from self.stageServer, which is set through the stage_server config var, which is set to upload.tbirdbld.productdelivery.prod.mozaws.net at https://mxr.mozilla.org/build/source/buildbot-configs/mozilla/thunderbird_production_config.py#28

Possibly using download_base_url for the mar tools and previous mar download instead of stage_server would work in most cases, but download_base_url for try builds is http://archive.mozilla.org/pub/thunderbird/try-builds and there may be other side-effects. Maybe a different fix was first anticipated in bug 1221391?

rail, I'd appreciate if you could give us a hint here :)
Flags: needinfo?(rail)
Summary: comm-central-l10n build is not updated after 2015-10-08 → comm-central-l10n and comm-aurora-l10n builds are not updated after the s3 migration
Attached patch buildbotcustom changes - v1 (obsolete) — Splinter Review
I was thinking about something like this. It won't quite work yet because before the previous mar was retrieved using ssh and ls, which probably won't work anymore. Getting mar and mbsdiff should work though.

Any preferences on how I should retrieve the previous mar? If I hardcode it to being an s3 host, then I could probably use a library or REST call to get the bucket contents. There are also libraries about.
Assignee: nobody → philipp
Status: NEW → ASSIGNED
Attachment #8706091 - Flags: feedback?(rail)
Blocks: 1192287
Comment on attachment 8706091 [details] [diff] [review]
buildbotcustom changes - v1

This should do the trick. At some point Nick separated upload and download hosts.
Flags: needinfo?(rail)
Attachment #8706091 - Flags: feedback?(rail) → feedback+
What is the status of this for 45.
Flags: needinfo?(rkent)
Better to ask Fallen.
Flags: needinfo?(rkent) → needinfo?(philipp)
To resolve this issue I think the best way forward is to take the two previous patches. This will improve the situation for most buildbot steps to use the right host. It will however still break for getting previous partials. Fixing this entails listing the previous mar files, but there is no programatic way to do this afaik. The situation does not get worse though, because getting previous mars via ssh is unconditionally broken using buildbot. 

On top of the previous patches, I propose we just disable l10n nightly partials for Thunderbird and hope for the better when moving to TaskCluster, which is what this patch does.
Flags: needinfo?(philipp)
Attachment #8721663 - Flags: review?(rail)
(In reply to aleth [:aleth] from comment #8)
> What is the status of this for 45.

This does not necessarily have to be fixed for 45, but the localizers are not getting any l10n nightly builds on aurora. The only way to check their string translations is to wait for us to build a beta, or compile it themselves.
Sadly, I'm hit by this very hard. I have redone the whole Serbian translation and I don't have now a way to test it.

I have tried to build TB on Fedora 23 but the build fails because of the missing mozIDOMWindow.h header and I do not have enough expertise to fix that. Simply commenting out the #include didn't work and I do not really want to spend time building when I could spend that time testing out Thunderbird's translation.

Is there any ETA when the new beta will be released?
Fixing this issue does not help for beta, since there are only aurora and central l10n nightly builds. beta 2 was just built and should be available for auto update soon. You can check here if you don't want to wait for auto-updates: http://ftp.mozilla.org/pub/thunderbird/candidates/45.0b2-candidates/build1/
Excellent! Thanks for the tip Philipp.
I think most localizers are hit, especially since they work in Aurora. A workaround is to use en-US Aurora (now that’s been fixed) and replace its language files by the localized ones.

If only there was at least one localized Aurora version (or langpack) per cycle available as long as automated builds fail to build, that would be great and maybe even more reliable. Would that be possible?

And how about SeaMonkey? SM testing is kind of stuck even more for our locale since there are no en-US Win Aurora builds available as of Nov 17.
Yea. Using beta for testing out translations won't work. Just started up beta, my translations aren't in it. :(
Attachment #8721663 - Flags: review?(rail) → review+
Comment on attachment 8706092 [details] [diff] [review]
buildbot-configs changes - v1

Nick, can you look at these?
Attachment #8706092 - Flags: review?(rail) → review?(nthomas)
Attachment #8706091 - Flags: review?(rail) → review?(nthomas)
There is some news on it?

The pt_BR l10n team translate 100% of Aurora strings, but can't test it. It causes links on https://www.mozilla.org/en-US/thunderbird/earlybird/all/ (using https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-aurora-l10n/ as root) broken.
Comment on attachment 8706091 [details] [diff] [review]
buildbotcustom changes - v1

I think you could make these patches much more compact by leaving stage_server/stageServer alone (ie the actual upload host), and using downloadBaseUrl to get mar tools. If you really need it you could add ftp_server/ftpServer variables for NightlyRepackFactory, and pass archive.mozilla.org. That should also avoid changing release automation shared with Firefox, which doesn't pass checkconfig with these patches.

>@@ -3485,18 +3488,17 @@ class NightlyRepackFactory(BaseRepackFac
>     def downloadMarTools(self):
...
>-        baseURL = 'http://%s' % self.stageServer + \
>-                  '/pub/mozilla.org/%s' % self.productName + \
>+        baseURL = self.downloadBaseURL + 'http://%s' % self.stageDownloadServer + \
>                   '/nightly/latest-%s' % self.branchName + \
>                   '/mar-tools/%s' % self.platform

Not sure how this would work. downloadBaseURL instead of stageDownloadServer, right ?
Attachment #8706091 - Flags: review?(nthomas) → review-
Comment on attachment 8706092 [details] [diff] [review]
buildbot-configs changes - v1

See comments on the buildbotcustom patch for r-.

>diff --git a/mozilla/thunderbird_production_config.py b/mozilla/thunderbird_production_config.py
> # Local overrides for default values
> GLOBAL_VARS.update({
>-    'stage_server': 'upload.tbirdbld.productdelivery.prod.mozaws.net',
>+    'stage_upload_server': 'upload.tbirdbld.productdelivery.prod.mozaws.net',
>+    'stage_download_server': 'archive.mozilla.org',
>     'balrog_username': 'tbirdbld',
>     'download_base_url': 'http://archive.mozilla.org/pub/thunderbird',

I'm hopeful that download_base_url is all you really need here.

>     'try-comm-central': {
>-        'stage_server': 'upload.trybld.productdelivery.prod.mozaws.net',
>+        'stage_upload_server': 'upload.trybld.productdelivery.prod.mozaws.net',
>+        'stage_download_server': 'archive.mozilla.org',
>         'tinderbox_tree': 'Try-Comm-Central',
>         'download_base_url': 'http://archive.mozilla.org/pub/thunderbird/try-builds',

I wouldn't worry about this strange value. AFAIK we don't support l10n on try with this code but the original author was forced into defining something here.
Attachment #8706092 - Flags: review?(nthomas) → review-
For reference, we made changes in bug 1206041 to support the split of upload/download.
Blocks: 1213721
Earlybird download links have now been broken on www.mozilla.org for 3 months :(

Is there any expected resolution to this bug soon?
Blocks: 1222693
Latest comm-central-l10n failed like below:
Same revision is set both moz_revision and comm_revision.


https://archive.mozilla.org/pub/thunderbird/nightly/2016/03/2016-03-28-03-02-08-comm-central-l10n/comm-central-win32-l10n-nightly-ja-bm91-build1-build19.txt.gz

========= Started set props: buildid moz_revision comm_revision (results: 0, elapsed: 1 secs) (at 2016-03-28 05:14:26.071337) =========
(snip)
moz_revision: '63be002b4a803df1122823841ef7633b7561d873'
comm_revision: '63be002b4a803df1122823841ef7633b7561d873'
========= master_lag: 1.03 =========
========= Finished set props: buildid moz_revision comm_revision (results: 0, elapsed: 1 secs) (at 2016-03-28 05:14:27.530467) =========
(In reply to Takanori MATSUURA from comment #23)
Filed as bug 1262452.
Depends on: 1262452
Depends on: 1267523
I have changed to major as this is now well over 3 month nearly 5 month over due
Severity: normal → major
Fallen, wha tnext? Do we need the blocking bugs resolved first?
Flags: needinfo?(philipp)
(In reply to Alex Gibson [:agibson] from comment #22)
> Earlybird download links have now been broken on www.mozilla.org for 3
> months :(

https://www.mozilla.org/en-US/thunderbird/channel/ 
and
https://www.mozilla.org/en-US/thunderbird/earlybird/all/
(didn't even know that existed)
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #27)
> (In reply to Alex Gibson [:agibson] from comment #22)
> > Earlybird download links have now been broken on www.mozilla.org for 3
> > months :(
> 
> https://www.mozilla.org/en-US/thunderbird/channel/ 
> and
> https://www.mozilla.org/en-US/thunderbird/earlybird/all/
> (didn't even know that existed)

See Bug 1236045 for more info
Rail, is there someone in releng who can bug 1262452, bug 1267523?  Or is that something we, thunderbird, need to pick up?
Flags: needinfo?(philipp) → needinfo?(rail)
I'd ask someone from the Build System team regarding bug 1267523.

rebug 1262452, looks like 'make ident' in https://dxr.mozilla.org/comm-central/source/mail/locales/Makefile.in#145-147 prints gecko revision from application.ini. Sounds like another build system thing.
Flags: needinfo?(rail)
Depends on: 1267270
This has now been broken for over 6 months. Is there any resolution in sight? How long do we realistically carry on listing Earlybird as available for download on mozilla.org, given that all the links are broken.
Comment on attachment 8706091 [details] [diff] [review]
buildbotcustom changes - v1

Review of attachment 8706091 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +996,5 @@
>          else:
>              self.logUploadDir = 'tinderbox-builds/%s-%s/' % (self.branchName,
>                                                               self.stagePlatform)
>              self.logBaseUrl = 'http://%s/pub/mozilla.org/%s/%s' % \
> +                (self.stageUploadServer,

for what it's worth,  http://%s/pub/mozilla.org/%s/%s'
no longer valid.  Should be now http://%s/pub/%s/%s'

So I'm guessing:

http://%s/pub/%s/%s' % \
  (self.stageUploadServer, self.stageProduct, self.logUploadDir)

(well, at least that's what I've figured out from my
messing around with SeaMonkey's buildbot code.)

@@ +2128,1 @@
>                                          '%s' % self.latestDir +

from what I can follow, self.latestDir is mentioned in
https://hg.mozilla.org/build/buildbotcustom/file/c400d63b386d/process/factory.py#l967

  self.latestDir = '/pub/mozilla.org/%s' % self.stageProduct + \
                         '/nightly/latest-%s' % self.branchName

You need to remove mozilla.org from that path in order for
it to work.  Especially now it's based on S3..  
(I *think* it's safe to say that all pub/mozilla.org/ instances
can be just changed to /pub/
It looks like currently configure isn't even reached: e.g. https://archive.mozilla.org/pub/thunderbird/nightly/2016/08/2016-08-12-15-09-44-comm-aurora-l10n/comm-aurora-macosx64-l10n-nightly-unknown-bm85-build1-build4301.txt.gz

Is that related to the issues from comment 32?
Flags: needinfo?(ewong)
(In reply to aleth [:aleth] from comment #33)
> It looks like currently configure isn't even reached: e.g.
> https://archive.mozilla.org/pub/thunderbird/nightly/2016/08/2016-08-12-15-09-
> 44-comm-aurora-l10n/comm-aurora-macosx64-l10n-nightly-unknown-bm85-build1-
> build4301.txt.gz
> 
> Is that related to the issues from comment 32?

From what I can grasp from the traceback, no.  it's complaining that
en_revision doesn't exist in the repack.  Was this repack triggered
manually?
Flags: needinfo?(ewong)
(In reply to Edmund Wong (:ewong) from comment #34)
> (In reply to aleth [:aleth] from comment #33)
> > It looks like currently configure isn't even reached: e.g.
> > https://archive.mozilla.org/pub/thunderbird/nightly/2016/08/2016-08-12-15-09-
> > 44-comm-aurora-l10n/comm-aurora-macosx64-l10n-nightly-unknown-bm85-build1-
> > build4301.txt.gz
> > 
> > Is that related to the issues from comment 32?
> 
> From what I can grasp from the traceback, no.  it's complaining that
> en_revision doesn't exist in the repack.  Was this repack triggered
> manually?

Sorry, I meant that the 'en_revision' key didn't exist in the
build properties for this particular repack.
(In reply to Edmund Wong (:ewong) from comment #35)
> (In reply to Edmund Wong (:ewong) from comment #34)
> > (In reply to aleth [:aleth] from comment #33)
> > > It looks like currently configure isn't even reached: e.g.
> > > https://archive.mozilla.org/pub/thunderbird/nightly/2016/08/2016-08-12-15-09-
> > > 44-comm-aurora-l10n/comm-aurora-macosx64-l10n-nightly-unknown-bm85-build1-
> > > build4301.txt.gz
> > > 
> > > Is that related to the issues from comment 32?
> > 
> > From what I can grasp from the traceback, no.  it's complaining that
> > en_revision doesn't exist in the repack.  Was this repack triggered
> > manually?

Ah, manually retriggered l10n repacks don't work? I didn't know that. Thanks!
(In reply to aleth [:aleth] from comment #36)
> (In reply to Edmund Wong (:ewong) from comment #35)
> > (In reply to Edmund Wong (:ewong) from comment #34)
> > > (In reply to aleth [:aleth] from comment #33)
> > > > It looks like currently configure isn't even reached: e.g.
> > > > https://archive.mozilla.org/pub/thunderbird/nightly/2016/08/2016-08-12-15-09-
> > > > 44-comm-aurora-l10n/comm-aurora-macosx64-l10n-nightly-unknown-bm85-build1-
> > > > build4301.txt.gz
> > > > 
> > > > Is that related to the issues from comment 32?
> > > 
> > > From what I can grasp from the traceback, no.  it's complaining that
> > > en_revision doesn't exist in the repack.  Was this repack triggered
> > > manually?
> 
> Ah, manually retriggered l10n repacks don't work? I didn't know that. Thanks!

You can manually retrigger l10n repacks.. it's just that you need to
add specific properties on the buildbot page, unlike regular builds which
you can just retrigger and forget. 

With l10n repacks, you need to specify the following:

branch:  (in this case, comm-aurora)

and then in the six empty fields:

l10n_revision  <revision of the locale you've chosen>
en_revision    default
locale         <locale>
Any progress? We’re heading to a new cycle and my TB aurora is still stuck on October 18 2015, so l10n is still a pain and I keep running into bugs that I can’t get a hold of when it comes to telling whether or not they have been fixed meanwhile.

OR (/and): as written in comment 15, would it be possible to make one localized aurora TB/SM build available manually for every cycle until it gets fixed? Either way would make localizers happy, I suppose.
(In reply to Ton from comment #38)
> Any progress? We’re heading to a new cycle and my TB aurora is still stuck
> on October 18 2015, so l10n is still a pain and I keep running into bugs
> that I can’t get a hold of when it comes to telling whether or not they have
> been fixed meanwhile.

Beta l10n is now finally fixed, so that's some good news at least. I just landed a patch for bug 1267523, let's hope it works and then we might be able to make progress here.
Summary: comm-central-l10n and comm-aurora-l10n builds are not updated after the s3 migration → comm-central-l10n and comm-aurora-l10n builds are not updated after the s3 migration: 'get mar' step fails
As this was last looked at months ago due to the various bustages that were piled on top, I'm not sure how current some of the comments in this bug are today. If I understand correctly based on comment 19, this may be enough to fix the get mar step. (Somehow, I doubt it...)
Attachment #8800695 - Flags: feedback?(nthomas)
Assignee: philipp → aleth
There's also a note in comment 32 that all the .../pub/mozilla.org URLs should just be .../pub. Looking at the ftp server, this seems correct. But there quite are a lot of these...
Attachment #8800713 - Flags: review?(nthomas)
Attachment #8721663 - Attachment is obsolete: true
Attachment #8706092 - Attachment is obsolete: true
Attachment #8706091 - Attachment is obsolete: true
Comment on attachment 8721663 [details] [diff] [review]
disable comm l10n partials - v1

It's possible we still need the r+ patch from comment 10.
Attachment #8721663 - Attachment is obsolete: false
Attachment #8721663 - Flags: feedback?(nthomas)
Comment on attachment 8800713 [details] [diff] [review]
Switch TB /pub/mozilla.org URLs to /pub, buildbot-configs part

Lets not worry about this one. l10nUploadPath is leftover config and not used anywhere. base_bundle_urls points to a path which doesn't exist, with or without /mozilla.org/.
Attachment #8800713 - Attachment is obsolete: true
Attachment #8800713 - Flags: review?(nthomas)
Comment on attachment 8800703 [details] [diff] [review]
Switch /pub/mozilla.org URLs to /pub, buildbotcustom part

We can take this, but it's only cleanup and won't make any functional difference. Urls with .../mozilla.org/... still work because they redirect to the an url without it.
Attachment #8800703 - Flags: review?(nthomas) → review+
The issue here is http access to upload.tbirdbld.productdelivery.prod.mozaws.net is timed out. (Comment #1)
In mozilla-central-l10n, mar.exe is downloaded from ftp.mozilla.org. (Comment #2)
Comment on attachment 8800703 [details] [diff] [review]
Switch /pub/mozilla.org URLs to /pub, buildbotcustom part

http://hg.mozilla.org/build/buildbotcustom/rev/dafdb09f5082
Attachment #8800703 - Flags: checked-in+
Comment on attachment 8800695 [details] [diff] [review]
Use downloadBaseURL in NightlyRepackFactory.downloadMarTools

lgtm - http://hg.mozilla.org/build/buildbotcustom/rev/3b0c1d8cb4eb
Attachment #8800695 - Flags: review+
Attachment #8800695 - Flags: feedback?(nthomas)
Attachment #8800695 - Flags: checked-in+
Comment on attachment 8721663 [details] [diff] [review]
disable comm l10n partials - v1

I think we could get away without this, because the step which ssh's into the upload host is allowed to fail without burning the rest of the job. This is what's happening for en-US nightlies right now. Disabling partials does save a little bit of compute time, so I took the opportunity to do that for en-US on c-c, c-a, and c-esr45 too:
http://hg.mozilla.org/build/buildbot-configs/rev/b4018e35492b
Attachment #8721663 - Flags: feedback?(nthomas) → feedback+
Looks like it worked! However, almost every repack now ends up in its own folder on the ftp server, as the timestamp in the folder name now differs:

https://archive.mozilla.org/pub/thunderbird/nightly/2016/10/

I don't understand how the patches that landed here could have had that effect.
Flags: needinfo?(nthomas)
Depends on: 1310229
(In reply to aleth [:aleth] from comment #52)
> I don't understand how the patches that landed here could have had that
> effect.

Bug 1310229 might be a related side effect.
Depends on: tb-taskcluster
(In reply to aleth [:aleth] from comment #53)
> (In reply to aleth [:aleth] from comment #52)
> > I don't understand how the patches that landed here could have had that
> > effect.
> 
> Bug 1310229 might be a related side effect.

My guess is the fact we now get as far as doRepack() breaks the build id and the path issue in comment 52 follows from it.

That would mean we should resolve this bug fixed and deal with this in bug 1310229.
No longer depends on: tb-taskcluster
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
No longer depends on: 1310229
I haven't had a chance to check the jobs, but I strongly suspect they aren't picking up the en-US buildID, so the log upload for the l10n job uses the current time instead.
Flags: needinfo?(nthomas)
(In reply to Nick Thomas [:nthomas] from comment #55)
> I haven't had a chance to check the jobs, but I strongly suspect they aren't
> picking up the en-US buildID, so the log upload for the l10n job uses the
> current time instead.

Bug 1310229 fix this?
(In reply to Takanori MATSUURA from comment #56)
> (In reply to Nick Thomas [:nthomas] from comment #55)
> > I haven't had a chance to check the jobs, but I strongly suspect they aren't
> > picking up the en-US buildID, so the log upload for the l10n job uses the
> > current time instead.
> 
> Bug 1310229 fix this?

It should, yes.
Good to see this fixed for TB, thanks for that.

But how about SeaMonkey builds? Asked twice above but to no avail.
http://ftp.mozilla.org/pub/seamonkey/nightly/latest-comm-aurora-l10n/ still shows a long listing with old builds (causing slow page loading, could need cleanup) and langpacks to use with en-US aurora aren’t updated either afaik, nor unofficial builds.
(In reply to Ton from comment #59)
> But how about SeaMonkey builds? Asked twice above but to no avail.
> http://ftp.mozilla.org/pub/seamonkey/nightly/latest-comm-aurora-l10n/ still
> shows a long listing with old builds (causing slow page loading, could need
> cleanup) and langpacks to use with en-US aurora aren’t updated either afaik,
> nor unofficial builds.

Please file a separate bug for this in the Seamonkey product, Seamonkey uses its own forked version of buildbot as far as I know. (I can't answer your question, but this bug is resolved).
Blocks: 1322344
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: