Make Funsize generated partials identical across platforms and environments

RESOLVED FIXED

Status

Release Engineering
Tools
--
critical
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: ffledgling, Assigned: mtabara)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3085] [funsize])

(Reporter)

Description

3 years ago
Currently the partial MARs being generated across different platforms have different hashes due to a variety of reasons:

i) LANG/LC variable settings for partial generation scripts
ii) permissions of files (possible umask problem)
iii) Other yet unknown and/or un-investigated causes.

This needs to be fixed so that we can be certain that the partials generated are identical everywhere by simply comparing hashes.
(Reporter)

Updated

3 years ago
Assignee: nobody → ffledgling
Status: NEW → ASSIGNED
Whiteboard: [funsize]
(Assignee)

Comment 1

3 years ago
Played around with these differences too. Some brief comments:

1) LC_ALL/LANG issue mainly affects the ordering of the files in the manifests. This issue can be solved two ways:

i) there's a known bug in the script generating the partial. Moreover, we're likely to get rid of the issue when migrating to Python. Both approaches ensure *eventual* solving of this issue.

ii) LC_ALL=C is set for all builders: http://hg.mozilla.org/build/buildbot-configs/file/efdd374d3260/mozilla/config.py#l199 . We could add an application config variable (through celery, maybe flask?) to run the partial generating scripts with the same locale config.

2) The files may be different due to signature issues too. Most of the production partials are SSL signed.

Will investigate more on this.
Assignee: ffledgling → mtabara
(Assignee)

Comment 2

3 years ago
Another observations, I'll add them here for future reference.

i) for 28 -> 29 (both Linux and Mac OS) partials are the same as content (tested with diff -r =(unwrapped) =(unwrapped)) but there is a slight difference of 264 bytes in measuring the sizes of the mar files. In this particular case I assumed the difference stands for the signature (the partials from ftp.mozilla.org/pub/mozilla.org ... ) are ssl-signed while local generated ones aren't (tested via `$MAR -T archive`)

ii) Though I thought there's a pattern, I found differences in content too (binaries included) and larger mar size differences for other firefox versions updates.
 
[29->30] 22K 
[27->28] 19K
(Reporter)

Comment 3

3 years ago
(In reply to mtabara from comment #1)
> Played around with these differences too. Some brief comments:
> 
> 1) LC_ALL/LANG issue mainly affects the ordering of the files in the
> manifests. This issue can be solved two ways:
> 
> i) there's a known bug in the script generating the partial. Moreover, we're
> likely to get rid of the issue when migrating to Python. Both approaches
> ensure *eventual* solving of this issue.

There's already an existing bug to change the bash scripts to fix this, looking at the bug it looks like I did some work on it, and then forgot about it. Adding that as a dep, and I'll try to attach an updated patch to the same. Marking it as a dependency for this bug either way.

Bug 1027871 will fix the problems when using bash. The eventual migration to python will cause this bug to not arise at all.

> 
> ii) LC_ALL=C is set for all builders:
> http://hg.mozilla.org/build/buildbot-configs/file/efdd374d3260/mozilla/
> config.py#l199 . We could add an application config variable (through
> celery, maybe flask?) to run the partial generating scripts with the same
> locale config.
> 
> 2) The files may be different due to signature issues too. Most of the
> production partials are SSL signed.
> 
> Will investigate more on this.
Depends on: 1027871
(Assignee)

Comment 4

3 years ago
Thanks for pointing this out Anhad, looked for the bug initially but couldn't find it. It's good to keep an eye on all dependencies.
(Assignee)

Comment 5

3 years ago
> ii) Though I thought there's a pattern, I found differences in content too
> (binaries included) and larger mar size differences for other firefox
> versions updates.
>  
> [29->30] 22K 
> [27->28] 19K

With regards to this, I apologies. My testing was bad, the actual differences are < 200 bytes so we're good. Assuming there's a signature difference there, we're going to test this with more complex <hash for all files under dir> script to ensure we're 100% accurate on generating partials.
(Assignee)

Comment 6

3 years ago
Some up-to-date news around here.
(In reply to Anhad Jai Singh (:ffledgling) from comment #0)
> Currently the partial MARs being generated across different platforms have
> different hashes due to a variety of reasons:
> 
> i) LANG/LC variable settings for partial generation scripts

We fixed this in the current implementation by forcing LC_ALL=C in the running environment of workers. This worked well for the testing we've done so far.

> ii) permissions of files (possible umask problem)
> iii) Other yet unknown and/or un-investigated causes.

iv) encountered issue: partials generated via build process are signed. To overcome this issue, we have added a script to compute hash for the contents of the mar file only, thus skipping the signature. The code lies currently here: https://github.com/MihaiTabara/build-funsize/blob/master/client/mar.py

Updated

3 years ago
Whiteboard: [funsize] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3075] [funsize]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3075] [funsize] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3080] [funsize]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3080] [funsize] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3085] [funsize]
I believe this is fixed now.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.