1313154 - adjust beetmoverscript to cope with mobile/desktop nightly graph constraints for single-locale

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Now that we're inches away from having our beetmoverworkers deployed to production (that is: working on date project branch, uploading to a releng dev S3 bucket) we will need to massage a bit beetmoverscript to:

* adapt the payload to cope with the nightly graph kinds
* enrich the templates to more than just fennec (that is firefox/platform/etc)
* adapt its output artifacts to serve them to balrogworker

This bug tracks all that beetmoverscript work in order to have successful beetmover jobs scheduled in the nightly graph (bug 1305096)

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - adjust beetmoverscript to cope with nightly graph constraints

@jlund r?

Milestone - proper bits into place to tweak beetmoverwork to bind nicely with taskgraph as we discussed last week. I thought it'd be useful to land this first to unblock @kmoir while I work on a separate branch to massage the manifest for balrogworker.

Once deployed we should be able to schedule beetmover tasks for signed and unsigned bits, at least for Fennec.

I have few leftovers left to address later:

* add more tests to ensure 100% coverage
* docstrings + docs everywhere
* add Travis support
* make Travis deploy automatically to Pypi every time we tag a new release

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - generate beetmover output to be used by balrogworker. r=jlund

@lundjordan f?

(This is a PR in own fork of beetmoverscript)

Next milestone: prepping the balrogworker manifest. I sort-of finished preparation of the manifest.json. I still have few TODOs to address but wanted you to have a look beforehand when you have a spare minute. I'm not sure yet as to how I'll get my hands on the actual url location but I'll find the answer soon I guess.

Also, while glancing in the funsize-balrogworker code just realized you were right, it looks like there's gonna be a need to refactor some of that to cope with Fennec nightly contraints.

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8805982 [details] [review]
Bug 1313154 - adjust beetmoverscript to cope with nightly graph constraints

see PR

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8806435 [details] [review]
Bug 1313154 - generate beetmover output to be used by balrogworker. r=jlund

see PR

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8805982 [details] [review]
Bug 1313154 - adjust beetmoverscript to cope with nightly graph constraints

https://github.com/mozilla-releng/beetmoverscript/commit/8011b48fc0c468a8d3f6f2cace9cfc945a021d18

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Added all other leftovers that can be postponed for now to the https://github.com/mozilla-releng/beetmoverscript/issues

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Pushed https://pypi.python.org/pypi/beetmoverscript/0.0.2 and to our internal Puppet and also to Pypi.
Github release tag is here - https://github.com/mozilla-releng/beetmoverscript/releases/tag/0.0.2

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Checked-in puppet change as well to bump that version - https://hg.mozilla.org/build/puppet/rev/e05536c4067d

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

* Enriched the mimetypes with JSON as well - PR to beetmoverscript here https://github.com/mozilla-releng/beetmoverscript/commit/4b5e71f74eb152238d29f0584fc0999cbf82ad8d 

* Shipped 0.0.3 version of beetmoverscript
* pushed to puppet to bump it in beetmoverworker too - https://hg.mozilla.org/build/puppet/rev/42640b01fb55

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

We now have fully working multi-locale Fennec nightlies on date - https://tools.taskcluster.net/push-inspector/#/GJMGcUAJQEuMHpGOYM5DNw/J1Qjsr2HTDG-NK7ds5QNNQ?_k=26nos1 \o/

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - bumping beetmoverscript to 0.0.3 in puppet r=jlund

https://hg.mozilla.org/build/puppet/rev/42640b01fb55

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - enrich beetmoverscript mimetypes with json r=jlund

https://github.com/mozilla-releng/beetmoverscript/commit/4b5e71f74eb152238d29f0584fc0999cbf82ad8d

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - adjust beetmoverworker to cope with dekstop nightlies.

Comment 14

[Jordan Lund (:jlund)]

Comment on attachment 8808304 [details] [review]
Bug 1313154 - adjust beetmoverworker to cope with dekstop nightlies.

see PR comments. resetting r? for now.

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8808304 [details] [review]
Bug 1313154 - adjust beetmoverworker to cope with dekstop nightlies.

jlund r+'ed in github.
Merged - https://github.com/mozilla-releng/beetmoverscript/commit/d37855d43a6d1b5fa811097a9efd77da565f9ff1

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - adjust beetmoverworker to cope with dekstop nightlies. r=jlund

https://hg.mozilla.org/build/puppet/rev/4625d85b9c85

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Most of the desktop nightlies got successful beetmoved to S3 buckets but we've hit some issues with gtest.zip, a rather large file (~500MB):

---

2016-11-07T23:26:00     INFO - application/zip
2016-11-07T23:26:00     INFO - 1478561190
2016-11-07T23:26:00     INFO - /net-mozaws-stage-delivery-firefox/pub/firefox/nightly/latest-date/firefox-52.0a1.en-US.linux-i686.gtest.tests.zip
2016-11-07T23:26:00     INFO - 2016-11-07 23:26:00,175 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 1
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,120 - beetmoverscript.script - INFO - 400
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,146 - beetmoverscript.script - INFO - <?xml version="1.0" encoding="UTF-8"?>
2016-11-07T23:26:59     INFO - <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>C22FA4E78B138A23</RequestId><HostId>....+.....=</HostId></Error>
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,152 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,167 - beetmoverscript.script - INFO - 400
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,167 - beetmoverscript.script - INFO - <?xml version="1.0" encoding="UTF-8"?>
2016-11-07T23:26:59     INFO - <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>8D5B1DC5A206467C</RequestId><HostId>.................=</HostId></Error>
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,167 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,168 - beetmoverscript.script - INFO - 400
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,168 - beetmoverscript.script - INFO - <?xml version="1.0" encoding="UTF-8"?><Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>964B2808FEFA1579</RequestId><HostId>..............................</HostId></Error>
2016-11-07T23:26:59     INFO - 2016-11-07 23:26:59,169 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry
2016-11-07T23:27:06     INFO - 2016-11-07 23:27:06,306 - scriptworker.utils - DEBUG - retry_async: <function put at 0x7f420f6189d8>: sleeping before retry
2016-11-07T23:27:06     INFO - 2016-11-07 23:27:06,325 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2
2016-11-07T23:27:06     INFO - 2016-11-07 23:27:06,344 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2
2016-11-07T23:27:06     INFO - 2016-11-07 23:27:06,346 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2
2016-11-07T23:27:06     INFO - 2016-11-07 23:27:06,486 - scriptworker.utils - DEBUG - retry_async: Calling <function put at 0x7f420f6189d8>, attempt 2

---

Pushed a patch to increase the ExpireIn from presigned url validation to 1800 seconds instead of 30m and pushed 0.0.5 beetmoverscript to production. Awaiting to retest the changes. https://github.com/mozilla-releng/beetmoverscript/commit/cbe08d2efe85e384c7692f623c89c76352445112

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Still getting the same error - the culprit is elsewhere. Dump from IRC:

01:11:35 <mtabara> I wish things were that easy. still the same issue, the culprit is elsewhere
01:12:25 <mtabara> something to do with a too large Content-Length we're setting before we send the files to S3, then aws waits for us to send more and timeouts after 20 secs for not receiving anything
01:12:27 <mtabara> https://github.com/aws/aws-sdk-js/issues/281#issuecomment-43798354
01:19:52 <mtabara> I give up, no more brainz. will debug in the morning 
01:20:19 <mtabara> worst case scenario - we can remove the gtest.zip for now as it's not blocking tier-2 and readdress it later on before tier-1
01:20:25 <mtabara> it's the only one causing us troubles
01:20:31 <mtabara> s,one,artifact
mtabara> the rest of them work nicely
01:24:00 <nthomas|away> You may be hitting the size where multi part upload is mandatory 
01:24:42 <mtabara> is there a workaround for that?

So the half-full side of the glass says we're technically done with desktop nightlies in terms of beetmoving. The half-empty says we still need to solve the gtest.zip or remove it for now to make the beetmover tasks green.

Comment 19

[Mihai Tabara [:mtabara]⌚️GMT]

Temporarily will remove gtest.zip and crashtest-report-full.zip from files to be beetmoved since they're 0.5G and 1G respectively and we're hitting some issues. Might need to investigate more on multi-part upload or ditch these two files directly, we'll see. Shipped 0.0.6 to address the temporarily comment of the aforementioned packages.

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

Green tasks in desktop nightlies too! However, we've knocked down though the gtest and crashreport-full-test.zip as they were too large.

Comment 21

[Jordan Lund (:jlund)]

Attachment: Bug 1313154 - generate beetmover output to be used by balrogworker

see PR for context :)

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8808536 [details] [review]
Bug 1313154 - generate beetmover output to be used by balrogworker

Addressed in the PR. Already merged it.
https://github.com/mozilla-releng/beetmoverscript/commit/906fd17d68f7bb508fd5a4043b6a29f7b74e5e36

Comment 23

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Jordan Lund (:jlund) from comment #21)
> Created attachment 8808536 [details] [review]
> Bug 1313154 - generate beetmover output to be used by balrogworker
> 
> see PR for context :)

Merged to production and deployed - manifest.json generation works nicely for Fennec - https://queue.taskcluster.net/v1/task/MN7-qxR-SaaqRxMOxW8YVQ/runs/1/artifacts/public/manifest.json

Comment 24

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - adapt beetmoverscript to cope with desktop nightlies.

Comment 25

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8809971 [details] [review]
Bug 1313154 - adapt beetmoverscript to cope with desktop nightlies.

https://github.com/mozilla-releng/beetmoverscript/commit/1b7075a057452d2eac2c152c2c54ca1592cd9278

r+'ed by Callek.

Comment 26

[Mihai Tabara [:mtabara]⌚️GMT]

Similar to 1315139, oremj granted our beetmover production user premissions to write to "net-mozaws-stage-delivery-{firefox,archive}" which backfills http://archive.mozilla.org/pub/{firefox,mobile}/nightly/. 

Now we're one puppet tweak away to pushing our artifacts in production buckets \o/

Comment 27

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - switch beetmoverworker to S3 production buckets.

Switching the artifacts from "net-mozaws-stage-delivery-{firefox,archive}" to ""net-mozaws-prod-delivery-{firefox,archive}" which backfills http://archive.mozilla.org/pub/{firefox,mobile}/nightly.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8810666 [details] [diff] [review]
Bug 1313154 - switch beetmoverworker to S3 production buckets.

Seems fine to me, based on what I can see on the bucketlister for staging.

Comment 29

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1313154 - switch beetmoverworker to S3 production buckets tight to just nightly subdirs

Comment 30

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8811059 [details] [diff] [review]
Bug 1313154 - switch beetmoverworker to S3 production buckets tight to just nightly subdirs

https://hg.mozilla.org/build/puppet/rev/0563f4230c7f

Comment 31

[Mihai Tabara [:mtabara]⌚️GMT]

Since now we're Fennec/Firefox single-locale Tier-2 ready with beetmoverscript/beetmoverworker working at full capacity, I'm closing this for now and will file a separate bug to track the l10n work for beetmoverworker/beetmoverscript.