1222693 - Thunderbird partial.mar updates for nightly comm-central builds

Status: RESOLVED DUPLICATE of bug 1410512

(Thunderbird :: Build Config, defect, P5)

Description

[hannu.nyman]

Thunderbird Nightly and Earlybird builds are missing the partial .mar update files. Only the >40MB sized "complete.mar" updates are currently available. So Thunderbird downloads the full update every day.

http://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-aurora/
http://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central/

The last partial update can be found for October 19th/20th, just before the ftp.m.o. move to S3, as discussed in Bug 1213721. So this is likely related to that change.

Last Earlybird build with partial .mar  files:
http://ftp.mozilla.org/pub/thunderbird/nightly/2015/10/2015-10-20-00-41-33-comm-aurora/

Comment 1

[Magnus Melin [:mkmelin]]

Yes, there's been various bustages (release automation infra and code).

Comment 2

[Wayne Mery (:wsmwk)]

should be all better now. 
let us know if it reappears

Comment 3

[Richard Marti (:Paenglab)]

Not better, partial mar are still missing.

Comment 4

[hannu.nyman]

Same here. So far each daily Thunderbird Earlybird update has been a full update of ~40.8 MB.

Last update was today to version Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Thunderbird/44.0a2 ID:20151205004008 CSet: 4c2aa4d60adda56a258bd04a77dd7c9364805603

Comment 5

[Wayne Mery (:wsmwk)]

my bad. partial.mar are indeed missing 
https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/
https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-aurora/

Comment 6

[hannu.nyman]

The partial *.mar updates are still missing for Thunderbird Earlybird. Each daily update is a full update, which is annoying. 

Apparently the partial updates are either not generated or alternatively, they are not copied to the correct place.

It is almost four months since the move to aws infra, so apparently this is the "new normal" for Thunderbird users :-(

Comment 7

[Wayne Mery (:wsmwk)]

related to the move to s3?

Comment 8

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Yes, I've found out about this in bug 1222878 and commented about it there too. To fix partial mars I'll need to find a way to fix buildbot code, which may mean the need for JSON directory listing access to the aws bucket behind the respective staging server (e.g. archive.mozilla.org). We may need to disable partials for the time being and hope to fix this with the TC move.

Comment 9

[Wayne Mery (:wsmwk)]

perhaps there is hope in bug 1222878 comment 21

Comment 10

[aleth [:aleth]]

l10n partials have been disabled in bug 1222878 comment 50. See also comment 8.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Funsize is the best way forward here I think, since it's a dedicated partial generation services (as used by Firefox). It may or may not be part of the taskcluster plans though.

FWIW, an interim approach would be to use boto to find the last nightly. The S3 bucket allows anonymous access to list the contents, eg

from boto.s3.connection import S3Connection
s3_conn = S3Connection(anon=True)
bucket = s3_conn.get_bucket('net-mozaws-prod-delivery-archive')

nightlies = [k.name for k in bucket.list(prefix='pub/thunderbird/nightly/latest-comm-central/')
             if k.name.endswith('complete.mar')]

Then filter for the platform of interest, and take the highest version (taking care that string sorting doesn't usually cope with X.Y.Z versioning properly). Numerous other tools could also be used, if a virtualenv with boto proves painful to implement in buildbot.

Comment 12

[Wayne Mery (:wsmwk)]

Tom how does this now fit in with TC (if it does)?

Comment 13

[Tom Prince [:tomprince]]

I think taskcluster might make things easier, but it isn't something that I'm going to worry about until after we are entirely migrated.

Comment 14

[Richard Marti (:Paenglab)]

Rob, is this something you could check?

Comment 15

[Rob Lemley [:rjl]]

We have not generated partials for anything with Taskcluster. It's on my todo list, ideally starting with 68.1. I haven't looked into how much work is involved yet though.