1051617 - Make Funsize generated partials identical across platforms and environments

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Anhad Jai Singh (:ffledgling)]

Currently the partial MARs being generated across different platforms have different hashes due to a variety of reasons:

i) LANG/LC variable settings for partial generation scripts
ii) permissions of files (possible umask problem)
iii) Other yet unknown and/or un-investigated causes.

This needs to be fixed so that we can be certain that the partials generated are identical everywhere by simply comparing hashes.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Played around with these differences too. Some brief comments:

1) LC_ALL/LANG issue mainly affects the ordering of the files in the manifests. This issue can be solved two ways:

i) there's a known bug in the script generating the partial. Moreover, we're likely to get rid of the issue when migrating to Python. Both approaches ensure *eventual* solving of this issue.

ii) LC_ALL=C is set for all builders: http://hg.mozilla.org/build/buildbot-configs/file/efdd374d3260/mozilla/config.py#l199 . We could add an application config variable (through celery, maybe flask?) to run the partial generating scripts with the same locale config.

2) The files may be different due to signature issues too. Most of the production partials are SSL signed.

Will investigate more on this.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Another observations, I'll add them here for future reference.

i) for 28 -> 29 (both Linux and Mac OS) partials are the same as content (tested with diff -r =(unwrapped) =(unwrapped)) but there is a slight difference of 264 bytes in measuring the sizes of the mar files. In this particular case I assumed the difference stands for the signature (the partials from ftp.mozilla.org/pub/mozilla.org ... ) are ssl-signed while local generated ones aren't (tested via `$MAR -T archive`)

ii) Though I thought there's a pattern, I found differences in content too (binaries included) and larger mar size differences for other firefox versions updates.
 
[29->30] 22K 
[27->28] 19K

Comment 3

[Anhad Jai Singh (:ffledgling)]

(In reply to mtabara from comment #1)
> Played around with these differences too. Some brief comments:
> 
> 1) LC_ALL/LANG issue mainly affects the ordering of the files in the
> manifests. This issue can be solved two ways:
> 
> i) there's a known bug in the script generating the partial. Moreover, we're
> likely to get rid of the issue when migrating to Python. Both approaches
> ensure *eventual* solving of this issue.

There's already an existing bug to change the bash scripts to fix this, looking at the bug it looks like I did some work on it, and then forgot about it. Adding that as a dep, and I'll try to attach an updated patch to the same. Marking it as a dependency for this bug either way.

Bug 1027871 will fix the problems when using bash. The eventual migration to python will cause this bug to not arise at all.

> 
> ii) LC_ALL=C is set for all builders:
> http://hg.mozilla.org/build/buildbot-configs/file/efdd374d3260/mozilla/
> config.py#l199 . We could add an application config variable (through
> celery, maybe flask?) to run the partial generating scripts with the same
> locale config.
> 
> 2) The files may be different due to signature issues too. Most of the
> production partials are SSL signed.
> 
> Will investigate more on this.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Thanks for pointing this out Anhad, looked for the bug initially but couldn't find it. It's good to keep an eye on all dependencies.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

> ii) Though I thought there's a pattern, I found differences in content too
> (binaries included) and larger mar size differences for other firefox
> versions updates.
>  
> [29->30] 22K 
> [27->28] 19K

With regards to this, I apologies. My testing was bad, the actual differences are < 200 bytes so we're good. Assuming there's a signature difference there, we're going to test this with more complex <hash for all files under dir> script to ensure we're 100% accurate on generating partials.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Some up-to-date news around here.
(In reply to Anhad Jai Singh (:ffledgling) from comment #0)
> Currently the partial MARs being generated across different platforms have
> different hashes due to a variety of reasons:
> 
> i) LANG/LC variable settings for partial generation scripts

We fixed this in the current implementation by forcing LC_ALL=C in the running environment of workers. This worked well for the testing we've done so far.

> ii) permissions of files (possible umask problem)
> iii) Other yet unknown and/or un-investigated causes.

iv) encountered issue: partials generated via build process are signed. To overcome this issue, we have added a script to compute hash for the contents of the mar file only, thus skipping the signature. The code lies currently here: https://github.com/MihaiTabara/build-funsize/blob/master/client/mar.py

Comment 7

[Rail Aliiev [:rail]]

I believe this is fixed now.