1262515 - Update generation for l10n - local cache working as intended?

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Axel Hecht [:Pike]]

bug 1199197 added a local cache for generating update generation.

I looked a tad at wall clock time in l10n, and realized that later diffs in update generation don't run significantly faster than the first, within a particular update generation job.

For example, https://public-artifacts.taskcluster.net/YZ00CLrZSnC73t7RW2-DWA/0/public/logs/live_backing.log, reading through that log, it seems that libxul constantly misses the local cache, and it clearly shouldn't.

Not sure if I read the logs right?

Comment 1

[Rail Aliiev [:rail]]

Our chunking logic is not optimized for caching (it was implemented before we enabled caching). Grouping the same versions in the same chunk would help here.

Comment 2

[Axel Hecht [:Pike]]

going back, https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=68c0b7d6f16ce5bb023e08050102b5f2fe4aacd8&filter-job_group_symbol=L10n&filter-job_group_symbol=Update-3&filter-job_group_symbol=Update-1&filter-job_group_symbol=Update-2&exclusion_profile=false&selectedJob=3620684 is what I'm looking at.

That's https://tools.taskcluster.net/task-inspector/#YZ00CLrZSnC73t7RW2-DWA/0 on tc, generating

public/env/Firefox-mozilla-central-48.0a1-macosx64-ar-20160405030214-20160406030221.partial.mar
public/env/Firefox-mozilla-central-48.0a1-macosx64-ast-20160405030214-20160406030221.partial.mar
public/env/Firefox-mozilla-central-48.0a1-macosx64-cs-20160405030214-20160406030221.partial.mar
public/env/Firefox-mozilla-central-48.0a1-macosx64-de-20160405030214-20160406030221.partial.mar
public/env/Firefox-mozilla-central-48.0a1-macosx64-en-GB-20160405030214-20160406030221.partial.mar

which should be the same XUL.

but the diffs don't match for /XUL, see below.

Now in bug 1199197 comment 8 you suspect mac signing? I see cache hits on linux. But in https://tools.taskcluster.net/task-inspector/#Hsp6ObI1QUOgmXWQfosXnA/0 on windows, libxul.dll also cache-misses.

diffing "Contents/MacOS/XUL"
File is not in the locale cache
'/tmp/tmpKoMyA9/to.work/Contents/MacOS/XUL.patch.bz2' -> '/tmp/fs-cache/52642f42b41983fcb7ca9ed763db52928416dbcb764af174ea8409a8ebaea91408814f2e6c16ff5c0770c3130495de8c15e0b3362cab5481003e44042a9c9b7e/271f728554177d6b5d97664e9ec586451b629a55a22d5ee5364bc71e5a6a1ea152c128108a99641c8579403c3af2582901b726d48cccf8a8e33e393c8f2959de'
/tmp/tmpKoMyA9/to.work/Contents/MacOS/XUL.patch.bz2 saved on local cache!
      patch "Contents/MacOS/XUL.patch" "Contents/MacOS/XUL"

...

diffing "Contents/MacOS/XUL"
File is not in the locale cache
'/tmp/tmpjVK5Tr/to.work/Contents/MacOS/XUL.patch.bz2' -> '/tmp/fs-cache/6ea4e569d1b329a0bfb952f1de04838790fdaee96135c1853c143581e139a5d82f6e0cc74224911745ae5c81950a8a340dddbd217ae3e5ac66475d2a66b64e88/6ae69a8ff43e93ef267145d6ae83cb9fe51960d72020b4923f5903446d799183a1c7167838dd21b70133a70281c7549865bb504bf3af98b2483fbaaa5948bd2a'
/tmp/tmpjVK5Tr/to.work/Contents/MacOS/XUL.patch.bz2 saved on local cache!
      patch "Contents/MacOS/XUL.patch" "Contents/MacOS/XUL"

....

diffing "Contents/MacOS/XUL"
File is not in the locale cache
'/tmp/tmpqE9xb7/to.work/Contents/MacOS/XUL.patch.bz2' -> '/tmp/fs-cache/4c2370c3bada24f24ebc891efb4d6870d0a239902fb2cc751daba9ce00f59ae513882884e9eea3f6fae96b9c472d4782d79666203ea43d7a6d3e06df374aa215/1e4bc8b22ea9ff65dbde86a9a62e0ea55e2895d4a9fd3bf12048f3350125698e1e0d04458e826a548d55eafbd051e18891a17072eb31cb9bcbe34918c20a1b31'
/tmp/tmpqE9xb7/to.work/Contents/MacOS/XUL.patch.bz2 saved on local cache!
      patch "Contents/MacOS/XUL.patch" "Contents/MacOS/XUL"

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Mac signing might be mucking this up ? It could be that we're re-signing XUL when we pass Firefox.tar.gz off to the signing server.