1228214 - Re-enable b2g source manifests and socorro json

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

To migrate the public-facing b2g files to S3 we needed to disable the make-socorro-json and upload-source-manifest actions, which populates directories like https://archive.mozilla.org/pub/b2g/manifests/nightly/2.6.0/2015/11/2015-11-25-03/.

We still need those files, so Socorro can scrape build IDs, and for partners & QA respectively. The problem is that the new system doesn't accomodate that easily, so we're re-implementing them asap.

The plan is for Socorro to continue as is, but simplify to deprecate the latest directories and files like socorro_emulator.json, keeping just 
 manifests/nighltly/2.6.0/YYYY/MM/YYYY-MM-DD-HH/socorro_emulator_2015-11-25-03.json

For source manifests we want to look to the future and publish them to the Taskcluster index instead, which makes it much easier to query an API by revison/pushdate/nightly date, and avoids scraping the ftp server.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] Non-working WIP

This work has stalled, and now I'm into some PTO. The current state is the attached diff from ash (last merge was m-c rev 528fc18a86d3, and ash also has some decompressed omni.ja changes that catlee has been working on).

https://treeherder.mozilla.org/#/jobs?repo=ash&filter-searchStr=b2g&revision=4bbcd673cb70

We care about the ICS emulator and flame builds, which are all failing with 
    AttributeError: 'module' object has no attribute 'NullHandler'
when the upload-taskcluster action tries to import taskcluster. This is strange because our inside-mock venv appears to be using python 2.7, which is where NullHandler was added.

NB: there are some testing changes in the diff, eg ash version of 999.0 and translate_hg_to_git set to False (the latter avoids mapper hanging nightlies)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Also NB: We're planning to stop uploading to pvtbuilds2, and just use TC artifacts, so there's a little setup work included in the patch.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

See also bug 1222231.