Closed
Bug 1274483
Opened 8 years ago
Closed 7 years ago
pulse_actions - Nightlies started with Treeherder don't work well
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: nthomas, Unassigned)
Details
Attachments
(2 files)
Log upload after the job fails because it needs buildid set.
|
Reporter | |
Comment 1•8 years ago
|
||
There's actually two issues. 1, Buildbot sets a buildid property on nightly jobs as they're setup, which is used after the job completes to upload the log (into tinderbox-builds, like the installer/tests did during the job). When a job is started via TH & mozci, the job improvises a buildid using the current time, and tries to set a property, but log_uploader doesn't find it to pass on to post_upload, and it complains 'Error: --buildid must be set'. 2, mozci sets a taskID (bug 1203552), but this interacts badly with Android builds that do the multi-l10n step - the en-US step uses that taskID and then marks the task as completed, which blocks the multilocale build from uploading to the same task. In the buildbot-scheduled nightlies we generate two taskIDs (slugs) at runtime. mshal suggests we could always do that for the multi-l10n case, but is unsure of wider consequences. Armen, I'm not sure how long it will be until we resolve this, or what the best solution is. Can we disable starting nightlies with mozci in the meantime ?
Flags: needinfo?(armenzg)
Summary: Nightlies started with treeherder don't set buildid property → Nightlies started with treeherder work well
|
|
Reporter | |
Updated•8 years ago
|
Summary: Nightlies started with treeherder work well → Nightlies started with don't treeherder work well
Summary: Nightlies started with don't treeherder work well → Nightlies started with treeherder don't work well
|
||
Comment 2•8 years ago
|
||
Removing it from showing up on Treeherder will be something we would need to talk with camd to help us exclude it from the runnable api (which is built from allthethings.json). On the pulse_actions side I can simply drop every request for nightly jobs. Anyone using "add new jobs" would be confused as to why they requested and it did not work. In any case, this is the easiest solution until we have a fix. nthomas: what do you prefer? On another note, would you please point me to some logs to help me figure out I understand completely before I look into solutions? Thanks.
Flags: needinfo?(armenzg)
|
|
Reporter | |
Comment 3•8 years ago
|
||
Turns out it depends on the platform. It looks like a mock issue (eg chroot shadowing the files), except that linux32 worked, and win64 didn't. Anyway, some logs, all for nightly builds. Failing: -------- Linux64 opt - fails to trigger l10n as buildid property not set, failed log upload too IIRC http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/1464276657/mozilla-central-linux64-nightly-bm72-build1-build46.txt.gz Linux64 ASAN opt - build OK, missing buildid so log upload failed http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64-asan/1464276757/mozilla-central-linux64-asan-nightly-bm74-build1-build10.txt.gz Linux64 ASAN debug - build OK, missing buildid so log upload failed http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64-asan-debug/1464276734/mozilla-central-linux64-asan-debug-nightly-bm72-build1-build13.txt.gz Win64 opt - fails to trigger l10n as buildid missing, failed log upload too http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win64/1464276333/mozilla-central-win64-nightly-bm91-build1-build6.txt.gz , Android API 15+ - Build 1, en-US goes OK but multilocale hits point 2 in comment #1, log upload was OK http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1464276631/mozilla-central-android-api-15-nightly-bm91-build1-build36.txt.gz 2, Rebuild fails in en-US taskcluster because the first build marked the upload task as complete; log upload failure http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1464293194/mozilla-central-android-api-15-nightly-bm91-build1-build37.txt.gz Android x86 - same as API 15+ http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1464276631/mozilla-central-android-api-15-nightly-bm91-build1-build36.txt.gz http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1464293194/mozilla-central-android-api-15-nightly-bm91-build1-build37.txt.gz Working: -------- Linux32 http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux/1464276785/mozilla-central-linux-nightly-bm74-build1-build14.txt.gz Mac http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-macosx64/1464276306/mozilla-central-macosx64-nightly-bm86-build1-build11.txt.gz Win32 http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1464276309/mozilla-central-win32-nightly-bm73-build1-build27.txt.gz
Summary: Nightlies started with treeherder don't work well → Nightlies started with don't treeherder work well
|
|
||
Comment 4•8 years ago
|
||
Thank you Nick!
Assignee: nobody → armenzg
Summary: Nightlies started with don't treeherder work well → Nightlies started with Treeherder don't work well
|
|
||
Comment 5•8 years ago
|
||
martianwars is going to give us a hand here!
Assignee: armenzg → kalpeshk2011
|
|
||
Comment 6•8 years ago
|
||
Here's the link to a push with the issues that nthomas raised: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=b0096c5c7277&filter-searchStr=nightly&exclusion_profile=false&selectedJob=3954246
|
||
Comment 7•8 years ago
|
||
Attachment #8759831 -
Flags: review?(armenzg)
|
|
||
Comment 8•8 years ago
|
||
nthomas: do all nightly builds triggered at the same time need to have the exact same buildid? Or is it fine if they're all within few seconds of each other? (since we schedule them sequentially). It seems that a working log will have the buildid defined at the top of the log: > buildid: 20160526083305 Bad logs will also have the buildid being set internally as a property: > 08:30:57 INFO - Setting buildbot property buildid to 20160526083057 > 08:30:57 INFO - Writing buildbot properties ['buildid'] to /builds/slave/m-cen-l64-ntly-000000000000000/properties/buildid Even though it gets set internally and produce a file with the info [1] it does not seem to be eventually be set properly [2]. [1] http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/1464276657/firefox-49.0a1.en-US.linux-x86_64_info.txt [2] In the good log we have this: ========= Finished set props: symbolsUrl packageUrl completeMarFilename appName buildid completeMarUrl got_revision completeMarHash appVersion testPackagesUrl hashType packageFilename testsUrl completeMarSize jsshellUrl sourcestamp sdkUrl stage_platform builduid uploadFiles (results: 0, elapsed: 0 secs) (at 2016-05-26 11:42:58.319457) ========= while on the bad one we have this: ========= Finished set props: symbolsUrl packageUrl completeMarFilename appName packageFilename completeMarHash got_revision completeMarUrl appVersion testPackagesUrl hashType testsUrl completeMarSize sourcestamp sdkUrl jsshellUrl uploadFiles (results: 0, elapsed: 0 secs) (at 2016-05-26 11:22:05.402141) =========
|
|
||
Comment 9•8 years ago
|
||
Comment on attachment 8759831 [details] [review] mozci PR to add extra buildid property Since nthomas is not around until London I will ask Callek for review.
Attachment #8759831 -
Flags: review?(armenzg) → review?(bugspam.Callek)
|
|
||
Comment 10•8 years ago
|
||
martianwars: could you look into what it would take to make the Android nightly builds not to have "upload_to_task_id" property set? I would like to see the Android nightly build at least complete even if the tests might need an extra bit of love. Thanks! > 2, mozci sets a taskID (bug 1203552), but this interacts badly with Android builds that do the multi-l10n step - the en-US > step uses that taskID and then marks the task as completed, which blocks the multilocale build from uploading to the same > task. In the buildbot-scheduled nightlies we generate two taskIDs (slugs) at runtime. mshal suggests we could always do that > for the multi-l10n case, but is unsure of wider consequences. For a bad Android nightly we have (the task IDs are repeated because they're grabbing the property upload_to_task_id): > 09:37:02 INFO - Taskcluster taskId: Mz2NWuOoTLa0kyfKt9WjoA > 09:46:01 INFO - Taskcluster taskId: Mz2NWuOoTLa0kyfKt9WjoA For a good Android nightly we have [1]: > 04:21:57 INFO - Taskcluster taskId: QZ34U4E9TdCZb-CTK69d0g > 04:30:30 INFO - Taskcluster taskId: Po13BeuMSgWTyd_6McbBGA What is happening is that we added some logic on mozci to tell builds jobs the taskID they need to use to upload artifacts to. Unfortunately, this trick did not account for builds that need to upload to two different tasks. Here's a link to an Android nightly and L10n jobs in case I need to review it later. [2] From looking at one of the logs, we don't need to know where the artifacts where uploaded on the build side because the L10n jobs grab the apk from archive instead of TaskCluster [3] However, it seems that we need it to trigger test jobs [4]. In any case, it seems that we could change scheduling of Android nightly builds to not set the "upload_to_task_id" property [5] until we can come up with a working solution (I'm thinking of a list of "upload_task_id" or only caring about setting the task id for only one of the two uploads). mshal, what do you say? [1] http://archive.mozilla.org/pub/mobile/nightly/2016/06/2016-06-02-03-02-20-mozilla-central-android-api-15/mozilla-central-android-api-15-nightly-bm72-build1-build44.txt.gz [2] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=Android%20nightly&revision=e27fe24a746f [3] 05:17:15 INFO - (cd /builds/slave/m-cen-and-api-15-l10n_1-000000/build/mozilla-central/obj-l10n/dist/ && wget --no-cache -nv -N 'http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-15/en-US/fennec-49.0a1.en-US.android-arm.apk') [4] /tools/buildbot/bin/buildbot sendchange --master buildbot-master81.build.mozilla.org:9301 --username sendchange --branch mozilla-central-android-api-15-talos -r e27fe24a746fa839f1cabe198faf1bad42c7dc4b --property buildid:20160603030242 --property pgo_build:False --property builduid:6445c0bda0034d32b6e02a148bc34026 --property nightly_build:True https://queue.taskcluster.net/v1/task/HXcKLrvRT328aXf-7B-W8A/artifacts/public/build/fennec-49.0a1.en-US.android-arm.apk /tools/buildbot/bin/buildbot sendchange --master buildbot-master81.build.mozilla.org:9301 --username sendchange-unittest --branch mozilla-central-android-api-15-opt-unittest -r e27fe24a746fa839f1cabe198faf1bad42c7dc4b --property buildid:20160603030242 --property pgo_build:False --property builduid:6445c0bda0034d32b6e02a148bc34026 --property nightly_build:True https://queue.taskcluster.net/v1/task/HXcKLrvRT328aXf-7B-W8A/artifacts/public/build/fennec-49.0a1.en-US.android-arm.apk https://queue.taskcluster.net/v1/task/HXcKLrvRT328aXf-7B-W8A/artifacts/public/build/fennec-49.0a1.en-US.android-arm.test_packages.json [5] https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/sources/buildbot_bridge.py#L357 https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/taskcluster_helper.py#36 [6] 09:46:02 INFO - ##### 09:46:02 INFO - ##### Finished multi-l10n step (failed) 09:46:02 INFO - ##### 09:46:02 FATAL - Uncaught exception: Traceback (most recent call last): 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/mozharness/base/script.py", line 1768, in run 09:46:02 FATAL - self.run_action(action) 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/mozharness/base/script.py", line 1707, in run_action 09:46:02 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/mozharness/base/script.py", line 1647, in _possibly_run_method 09:46:02 FATAL - return getattr(self, method_name)() 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/mozharness/mozilla/building/buildbase.py", line 1700, in multi_l10n 09:46:02 FATAL - locale='multi') 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/mozharness/mozilla/building/buildbase.py", line 1396, in _taskcluster_upload 09:46:02 FATAL - task = tc.create_task(routes) 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/mozharness/mozilla/taskcluster_helper.py", line 65, in create_task 09:46:02 FATAL - }, taskId=self.task_id) 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/build/venv/lib/python2.7/site-packages/taskcluster/client.py", line 455, in apiCall 09:46:02 FATAL - return self._makeApiCall(e, *args, **kwargs) 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/build/venv/lib/python2.7/site-packages/taskcluster/client.py", line 232, in _makeApiCall 09:46:02 FATAL - return self._makeHttpRequest(entry['method'], route, payload) 09:46:02 FATAL - File "/builds/slave/m-cen-and-api-15-ntly-00000000/build/venv/lib/python2.7/site-packages/taskcluster/client.py", line 424, in _makeHttpRequest 09:46:02 FATAL - superExc=rerr 09:46:02 FATAL - TaskclusterRestFailure: taskId Mz2NWuOoTLa0kyfKt9WjoA already used by another task. 09:46:02 FATAL - This could be the result of faulty idempotency! 09:46:02 FATAL - Existing task definition was: 09:46:02 FATAL - ```js 09:46:02 FATAL - { 09:46:02 FATAL - "provisionerId": "null-provisioner", 09:46:02 FATAL - "workerType": "buildbot", 09:46:02 FATAL - "schedulerId": "-", 09:46:02 FATAL - "taskGroupId": "Mz2NWuOoTLa0kyfKt9WjoA", 09:46:02 FATAL - "dependencies": [], 09:46:02 FATAL - "requires": "all-completed", 09:46:02 FATAL - "routes": [ 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.2016.05.26.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.2016.05.26.latest.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.latest.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile-l10n.android-api-15-opt.en-US", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.pushdate.2016.05.26.20160526152132.mobile-l10n.android-api-15-opt.en-US", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.latest.mobile-l10n.android-api-15-opt.en-US", 09:46:02 FATAL - "index.buildbot.branches.mozilla-central.android-api-15", 09:46:02 FATAL - "index.buildbot.revisions.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mozilla-central.android-api-15" 09:46:02 FATAL - ], 09:46:02 FATAL - "priority": "normal", 09:46:02 FATAL - "retries": 5, 09:46:02 FATAL - "created": "2016-05-26T16:37:02.317Z", 09:46:02 FATAL - "deadline": "2016-05-26T17:37:02.317Z", 09:46:02 FATAL - "expires": "2017-05-26T17:37:02.317Z", 09:46:02 FATAL - "scopes": [], 09:46:02 FATAL - "payload": {}, 09:46:02 FATAL - "metadata": { 09:46:02 FATAL - "owner": "mshal@mozilla.com", 09:46:02 FATAL - "source": "http://hg.mozilla.org/build/mozharness/", 09:46:02 FATAL - "name": "Buildbot/mozharness S3 uploader", 09:46:02 FATAL - "description": "Upload outputs of buildbot/mozharness builds to S3" 09:46:02 FATAL - }, 09:46:02 FATAL - "tags": {}, 09:46:02 FATAL - "extra": { 09:46:02 FATAL - "index": { 09:46:02 FATAL - "rank": 1464276092 09:46:02 FATAL - } 09:46:02 FATAL - } 09:46:02 FATAL - } 09:46:02 FATAL - ``` 09:46:02 FATAL - This request tried to define: 09:46:02 FATAL - ```js 09:46:02 FATAL - { 09:46:02 FATAL - "workerType": "buildbot", 09:46:02 FATAL - "deadline": "2016-05-26T17:46:01.191Z", 09:46:02 FATAL - "created": "2016-05-26T16:46:01.191Z", 09:46:02 FATAL - "routes": [ 09:46:02 FATAL - "index.gecko.v2.mozilla-central.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.pushdate.2016.05.26.20160526152132.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.latest.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.buildbot.branches.mozilla-central.android-api-15", 09:46:02 FATAL - "index.buildbot.revisions.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mozilla-central.android-api-15" 09:46:02 FATAL - ], 09:46:02 FATAL - "extra": { 09:46:02 FATAL - "index": { 09:46:02 FATAL - "rank": 1464276092 09:46:02 FATAL - } 09:46:02 FATAL - }, 09:46:02 FATAL - "payload": {}, 09:46:02 FATAL - "provisionerId": "null-provisioner", 09:46:02 FATAL - "metadata": { 09:46:02 FATAL - "owner": "mshal@mozilla.com", 09:46:02 FATAL - "source": "http://hg.mozilla.org/build/mozharness/", 09:46:02 FATAL - "name": "Buildbot/mozharness S3 uploader", 09:46:02 FATAL - "description": "Upload outputs of buildbot/mozharness builds to S3" 09:46:02 FATAL - }, 09:46:02 FATAL - "schedulerId": "-", 09:46:02 FATAL - "dependencies": [], 09:46:02 FATAL - "requires": "all-completed", 09:46:02 FATAL - "priority": "normal", 09:46:02 FATAL - "retries": 5, 09:46:02 FATAL - "scopes": [], 09:46:02 FATAL - "tags": {}, 09:46:02 FATAL - "taskGroupId": "Mz2NWuOoTLa0kyfKt9WjoA", 09:46:02 FATAL - "expires": "2017-05-26T17:46:01.191Z" 09:46:02 FATAL - } 09:46:02 FATAL - ``` 09:46:02 FATAL - ---- 09:46:02 FATAL - errorCode: RequestConflict 09:46:02 FATAL - statusCode: 409 09:46:02 FATAL - requestInfo: 09:46:02 FATAL - method: createTask 09:46:02 FATAL - params: {"taskId":"Mz2NWuOoTLa0kyfKt9WjoA"} 09:46:02 FATAL - payload: { 09:46:02 FATAL - "workerType": "buildbot", 09:46:02 FATAL - "deadline": "2016-05-26T17:46:01.191Z", 09:46:02 FATAL - "created": "2016-05-26T16:46:01.191Z", 09:46:02 FATAL - "routes": [ 09:46:02 FATAL - "index.gecko.v2.mozilla-central.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.pushdate.2016.05.26.20160526152132.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.latest.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.buildbot.branches.mozilla-central.android-api-15", 09:46:02 FATAL - "index.buildbot.revisions.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mozilla-central.android-api-15" 09:46:02 FATAL - ], 09:46:02 FATAL - "extra": { 09:46:02 FATAL - "index": { 09:46:02 FATAL - "rank": 1464276092 09:46:02 FATAL - } 09:46:02 FATAL - }, 09:46:02 FATAL - "payload": {}, 09:46:02 FATAL - "provisionerId": "null-provisioner", 09:46:02 FATAL - "metadata": { 09:46:02 FATAL - "owner": "mshal@mozilla.com", 09:46:02 FATAL - "source": "http://hg.mozilla.org/build/mozharness/", 09:46:02 FATAL - "name": "Buildbot/mozharness S3 uploader", 09:46:02 FATAL - "description": "Upload outputs of buildbot/mozharness builds to S3" 09:46:02 FATAL - }, 09:46:02 FATAL - "schedulerId": "-", 09:46:02 FATAL - "dependencies": [], 09:46:02 FATAL - "requires": "all-completed", 09:46:02 FATAL - "priority": "normal", 09:46:02 FATAL - "retries": 5, 09:46:02 FATAL - "scopes": [], 09:46:02 FATAL - "tags": {}, 09:46:02 FATAL - "taskGroupId": "Mz2NWuOoTLa0kyfKt9WjoA", 09:46:02 FATAL - "expires": "2017-05-26T17:46:01.191Z" 09:46:02 FATAL - } 09:46:02 FATAL - time: 2016-05-26T16:46:02.167Z 09:46:02 FATAL - details: 09:46:02 FATAL - { 09:46:02 FATAL - "taskId": "Mz2NWuOoTLa0kyfKt9WjoA", 09:46:02 FATAL - "existingTask": { 09:46:02 FATAL - "provisionerId": "null-provisioner", 09:46:02 FATAL - "workerType": "buildbot", 09:46:02 FATAL - "schedulerId": "-", 09:46:02 FATAL - "taskGroupId": "Mz2NWuOoTLa0kyfKt9WjoA", 09:46:02 FATAL - "dependencies": [], 09:46:02 FATAL - "requires": "all-completed", 09:46:02 FATAL - "routes": [ 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.2016.05.26.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.2016.05.26.latest.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.nightly.latest.mobile.android-api-15-opt", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile-l10n.android-api-15-opt.en-US", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.pushdate.2016.05.26.20160526152132.mobile-l10n.android-api-15-opt.en-US", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.latest.mobile-l10n.android-api-15-opt.en-US", 09:46:02 FATAL - "index.buildbot.branches.mozilla-central.android-api-15", 09:46:02 FATAL - "index.buildbot.revisions.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mozilla-central.android-api-15" 09:46:02 FATAL - ], 09:46:02 FATAL - "priority": "normal", 09:46:02 FATAL - "retries": 5, 09:46:02 FATAL - "created": "2016-05-26T16:37:02.317Z", 09:46:02 FATAL - "deadline": "2016-05-26T17:37:02.317Z", 09:46:02 FATAL - "expires": "2017-05-26T17:37:02.317Z", 09:46:02 FATAL - "scopes": [], 09:46:02 FATAL - "payload": {}, 09:46:02 FATAL - "metadata": { 09:46:02 FATAL - "owner": "mshal@mozilla.com", 09:46:02 FATAL - "source": "http://hg.mozilla.org/build/mozharness/", 09:46:02 FATAL - "name": "Buildbot/mozharness S3 uploader", 09:46:02 FATAL - "description": "Upload outputs of buildbot/mozharness builds to S3" 09:46:02 FATAL - }, 09:46:02 FATAL - "tags": {}, 09:46:02 FATAL - "extra": { 09:46:02 FATAL - "index": { 09:46:02 FATAL - "rank": 1464276092 09:46:02 FATAL - } 09:46:02 FATAL - } 09:46:02 FATAL - }, 09:46:02 FATAL - "taskDefinition": { 09:46:02 FATAL - "workerType": "buildbot", 09:46:02 FATAL - "deadline": "2016-05-26T17:46:01.191Z", 09:46:02 FATAL - "created": "2016-05-26T16:46:01.191Z", 09:46:02 FATAL - "routes": [ 09:46:02 FATAL - "index.gecko.v2.mozilla-central.revision.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.pushdate.2016.05.26.20160526152132.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.gecko.v2.mozilla-central.latest.mobile-l10n.android-api-15-opt.multi", 09:46:02 FATAL - "index.buildbot.branches.mozilla-central.android-api-15", 09:46:02 FATAL - "index.buildbot.revisions.b0096c5c727749ad3e79cbdf20d2e96bd179c213.mozilla-central.android-api-15" 09:46:02 FATAL - ], 09:46:02 FATAL - "extra": { 09:46:02 FATAL - "index": { 09:46:02 FATAL - "rank": 1464276092 09:46:02 FATAL - } 09:46:02 FATAL - }, 09:46:02 FATAL - "payload": {}, 09:46:02 FATAL - "provisionerId": "null-provisioner", 09:46:02 FATAL - "metadata": { 09:46:02 FATAL - "owner": "mshal@mozilla.com", 09:46:02 FATAL - "source": "http://hg.mozilla.org/build/mozharness/", 09:46:02 FATAL - "name": "Buildbot/mozharness S3 uploader", 09:46:02 FATAL - "description": "Upload outputs of buildbot/mozharness builds to S3" 09:46:02 FATAL - }, 09:46:02 FATAL - "schedulerId": "-", 09:46:02 FATAL - "dependencies": [], 09:46:02 FATAL - "requires": "all-completed", 09:46:02 FATAL - "priority": "normal", 09:46:02 FATAL - "retries": 5, 09:46:02 FATAL - "scopes": [], 09:46:02 FATAL - "tags": {}, 09:46:02 FATAL - "taskGroupId": "Mz2NWuOoTLa0kyfKt9WjoA", 09:46:02 FATAL - "expires": "2017-05-26T17:46:01.191Z" 09:46:02 FATAL - } 09:46:02 FATAL - } 09:46:02 FATAL - Running post_fatal callback... 09:46:02 ERROR - setting return code to 2 because fatal was called 09:46:02 WARNING - setting return code to 2 09:46:02 FATAL - Exiting -1
Flags: needinfo?(mshal)
|
|
||
Comment 11•8 years ago
|
||
We deployed a new version of mozci/pulse_actions [1] (and few follow up commits) which: 1 - Does not set upload_to_task_id for Android nightly builds (I can see in the output of [2]) From looking at the output [2], I just realized that we added the property 'buildid' to when we schedule via BuildApi rather than via TaskCluster. We will have to follow up on this. martianwars: would you be able to follow up on this? or do you want me to? In pulse_actions we first generate a builders_graph and then we schedule it: https://github.com/mozilla/pulse_actions/blob/master/pulse_actions/handlers/treeherder_runnable.py#L99 https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/sources/buildbot_bridge.py#L59 https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/sources/buildbot_bridge.py#L502 https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/sources/buildbot_bridge.py#L553 Steps to follow: * Can you please move this into _create_task()? https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/sources/buildbot_bridge.py#L553 * Add the logic for 'buildid' under _create_task(): https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/sources/buildbot_bridge.py#L135 I think that must be it. [1] https://github.com/mozilla/mozilla_ci_tools/commit/97d1548a5a39fe835c37f6b196775901dc9ee935 https://github.com/mozilla/mozilla_ci_tools/commits/master [2] Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: New jobs requested by armenzg@mozilla.com for https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4 Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: List of builders: Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - Linux mozilla-central nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - Linux x86-64 mozilla-central nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - Linux x86-64 mozilla-central debug asan nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - Linux x86-64 mozilla-central asan nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - OS X 10.7 mozilla-central nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - WINNT 5.2 mozilla-central nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - WINNT 6.1 x86-64 mozilla-central nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - Android armv7 API 15+ mozilla-central nightly Jun 07 06:45:59 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: - Android 4.2 x86 mozilla-central nightly Jun 07 06:46:16 pulse-actions app/worker2.1: mozci.taskcluster.tc INFO: Outputting the graph (graph id: A6oktmIKTduAEFn_Mx964w): Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [ Jun 07 06:46:16 pulse-actions app/worker2.1: "queue:define-task:buildbot-bridge/buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scheduler:create-task-graph" Jun 07 06:46:16 pulse-actions app/worker2.1: ], Jun 07 06:46:16 pulse-actions app/worker2.1: "tasks": [ Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "b5UDr4l-Q76X27txCnGcSA", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "Uo20DUY3TCqHNAKt94TN4g", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "Linux mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:10.003798Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:10.003765Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:10.003812Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "cBW7DSnHRhWOlIXNJ9c7QA", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "cN1S-7SkRhObe-y5J9wuXg", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "Linux x86-64 mozilla-central asan nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:10.695077Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:10.695036Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:10.695105Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "Th0OXvDURECGbh8IkohEbQ", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "mobile", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "Android 4.2 x86 mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:11.370963Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:11.370923Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:11.370989Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "J0vBI_PoRTCCNkwB7np9uw", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "X3SV1_I-QoeLzmbsvauKQg", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "Linux x86-64 mozilla-central debug asan nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:12.120392Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:12.120367Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:12.120407Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "NT0qaJOeQr6kEO4O2KQVUg", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "XHJ6S9d-TTyo-1x3o-hX9A", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "WINNT 6.1 x86-64 mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:12.862045Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:12.862005Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:12.862074Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "E-1czhYiRzmkh7agkQL3cg", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "mobile", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "Android armv7 API 15+ mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:13.585661Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:13.585621Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:13.585686Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "JWTtO9qpTpqcsutlCJ4EWg", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "UYKtuvvETHa4tnOAWRlLtA", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "OS X 10.7 mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:14.198725Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:14.198699Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:14.198740Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "ZxeZubDGReGNyNfVnXbpXQ", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "XzzJZ-VKQku6fMull18cPg", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "Linux x86-64 mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:14.845637Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:14.845600Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:14.845660Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: { Jun 07 06:46:16 pulse-actions app/worker2.1: "reruns": 0, Jun 07 06:46:16 pulse-actions app/worker2.1: "taskId": "XFlYfq_rSg-cANbAW0tI7A", Jun 07 06:46:16 pulse-actions app/worker2.1: "task": { Jun 07 06:46:16 pulse-actions app/worker2.1: "provisionerId": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "priority": "normal", Jun 07 06:46:16 pulse-actions app/worker2.1: "payload": { Jun 07 06:46:16 pulse-actions app/worker2.1: "sourcestamp": { Jun 07 06:46:16 pulse-actions app/worker2.1: "branch": "mozilla-central", Jun 07 06:46:16 pulse-actions app/worker2.1: "revision": "7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "properties": { Jun 07 06:46:16 pulse-actions app/worker2.1: "upload_to_task_id": "Einz9mPyT6aJJZdWifc7rw", Jun 07 06:46:16 pulse-actions app/worker2.1: "product": "firefox", Jun 07 06:46:16 pulse-actions app/worker2.1: "who": "cbook@mozilla.com" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "buildername": "WINNT 5.2 mozilla-central nightly" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "deadline": "2016-06-08T13:46:15.551369Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: }, Jun 07 06:46:16 pulse-actions app/worker2.1: "workerType": "buildbot-bridge", Jun 07 06:46:16 pulse-actions app/worker2.1: "scopes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "tags": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "routes": [], Jun 07 06:46:16 pulse-actions app/worker2.1: "created": "2016-06-07T13:46:15.551343Z", Jun 07 06:46:16 pulse-actions app/worker2.1: "retries": 5, Jun 07 06:46:16 pulse-actions app/worker2.1: "extra": {}, Jun 07 06:46:16 pulse-actions app/worker2.1: "schedulerId": "task-graph-scheduler", Jun 07 06:46:16 pulse-actions app/worker2.1: "expires": "2016-06-08T13:46:15.551383Z" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: ], Jun 07 06:46:16 pulse-actions app/worker2.1: "metadata": { Jun 07 06:46:16 pulse-actions app/worker2.1: "owner": "armenzg@mozilla.com", Jun 07 06:46:16 pulse-actions app/worker2.1: "description": "Adding new jobs to push via pulse_actions/treeherder for armenzg@mozilla.com.", Jun 07 06:46:16 pulse-actions app/worker2.1: "name": "WINNT 5.2 mozilla-central nightly", Jun 07 06:46:16 pulse-actions app/worker2.1: "source": "https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7c7d24700eb80ce328b05fd260ec58e9725ca4" Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:16 pulse-actions app/worker2.1: } Jun 07 06:46:20 pulse-actions app/worker2.1: mozci.taskcluster.tc INFO: See the graph in https://tools.taskcluster.net/task-graph-inspector/#A6oktmIKTduAEFn_Mx964w Jun 07 06:46:20 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable INFO: We don't have anything to schedule through Buildapi Jun 07 06:46:20 pulse-actions app/worker2.1: root INFO: Message {u'project': u'mozilla-central', u'buildernames': [u'Linux mozilla-central nightly', u'Linux x86-64 mozilla-central nightly', u'Linux x86-64 mozilla-central debug asan nightly', u'Linux x86-64 mozilla-central asan nightly', u'OS X 10.7 mozilla-central nightly', u'WINNT 5.2 mozilla-central nightly', u'WINNT 6.1 x86-64 mozilla-central nightly', u'Android armv7 API 15+ mozilla-central nightly', u'Android 4.2 x86 mozilla-central nightly'], u'requester': u'armenzg@mozilla.com', u'resultset_id': u'5821', u'version': 1}, took 20.1087760925 seconds to execute
|
||
Comment 12•8 years ago
|
||
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #10) > In any case, it seems that we could change scheduling of Android nightly > builds to not set the "upload_to_task_id" property [5] until we can come up > with a working solution (I'm thinking of a list of "upload_task_id" or only > caring about setting the task id for only one of the two uploads). > > mshal, what do you say? Yeah, I think we'll want to add a parameter to _taskcluster_upload() in testing/mozharness/mozharness/mozilla/building/buildbase.py to specify whether or not to use the buildbot taskId or an auto-generated one. The multi-l10n step should then specify to always use an auto-generated one. Something like this (untested): diff --git a/testing/mozharness/mozharness/mozilla/building/buildbase.py b/testing/mozharness/mozharness/mozilla/building/buildbase.py index 34b9501..50ed79a 100755 --- a/testing/mozharness/mozharness/mozilla/building/buildbase.py +++ b/testing/mozharness/mozharness/mozilla/building/buildbase.py @@ -1348,7 +1348,7 @@ or run without that action (ie: --no-{action})" self.routes_json = json.load(f) def _taskcluster_upload(self, files, templates, locale='en-US', - property_conditions=[]): + property_conditions=[], use_buildbot_task_id=True): if not self.client_id or not self.access_token: self.warning('Skipping S3 file upload: No taskcluster credentials.') return @@ -1387,7 +1387,7 @@ or run without that action (ie: --no-{action})" log_obj=self.log_obj, # `upload_to_task_id` is used by mozci to have access to where the artifacts # will be uploaded - task_id=self.buildbot_config['properties'].get('upload_to_task_id'), + task_id=self.buildbot_config['properties'].get('upload_to_task_id') if use_buildbot_task_id else None, ) # TODO: Bug 1165980 - these should be in tree @@ -1699,7 +1699,7 @@ or run without that action (ie: --no-{action})" files = shlex.split(output) abs_files = [os.path.abspath(os.path.join(objdir, f)) for f in files] self._taskcluster_upload(abs_files, self.routes_json['l10n'], - locale='multi') + locale='multi', use_buildbot_task_id=False) def postflight_build(self, console_output=True): """grabs properties from post build and calls ccache -s""" That way the regular en-US upload will use the taskId specified by mozci, and the multi-l10n will get its own taskId like usual.
Flags: needinfo?(mshal)
|
|
||
Comment 13•8 years ago
|
||
mshal: how could we test this change? Push to try, if nothing breaks wait until it merges to central?
|
|
||
Comment 14•8 years ago
|
||
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #13) > mshal: how could we test this change? > > Push to try, if nothing breaks wait until it merges to central? Unfortunately that's the tricky part :(. In the past I've tested in a staging environment, though that can easily take a day or two just to get setup and working in a way to test a change like this. Pushing to try would at least test the en-US part of the build. It might be worth trying to push a test commit that comments out the nightly check in multi_l10n(): if not self.query_is_nightly(): self.info("Not a nightly build, skipping multi l10n.") return Off-hand I don't know if that will just cause other problems, but it should be easy to check. Are you able to use mozci to specify a taskId in a try build? Or would we only be able to test the auto-generated taskId case anyway?
|
||
Comment 15•8 years ago
|
||
Comment on attachment 8759831 [details] [review] mozci PR to add extra buildid property armenzg merged this
Flags: needinfo?(armenzg)
Attachment #8759831 -
Flags: review?(bugspam.Callek)
|
|
||
Comment 16•8 years ago
|
||
I'm sorry about it. I should have been more clear. I wanted to know if the 'buildid' *has* to be the same for all nightlies that start at the same time or if it fine that each have a different one. Do you know? or someone that might know?
Flags: needinfo?(armenzg) → needinfo?(bugspam.Callek)
|
|
||
Comment 17•8 years ago
|
||
martianwars: do you want to try to fix and test both issues? Or would you prefer if I focus on the Android nightly part of this bug? (In reply to Michael Shal [:mshal] from comment #14) > (In reply to Armen Zambrano [:armenzg] - Engineering productivity from > comment #13) > > mshal: how could we test this change? > > > > Push to try, if nothing breaks wait until it merges to central? > > Unfortunately that's the tricky part :(. In the past I've tested in a > staging environment, though that can easily take a day or two just to get > setup and working in a way to test a change like this. Pushing to try would > at least test the en-US part of the build. It might be worth trying to push > a test commit that comments out the nightly check in multi_l10n(): > > if not self.query_is_nightly(): > self.info("Not a nightly build, skipping multi l10n.") > return > > Off-hand I don't know if that will just cause other problems, but it should > be easy to check. > > Are you able to use mozci to specify a taskId in a try build? Or would we > only be able to test the auto-generated taskId case anyway? In general, mozci schedules Buildbot jobs via TaskCluster graphs. In order to schedule a build and its associated test, I need to set upload_to_task_id for the test job to know where to fetch the build from. When scheduling a TaskCluster task which represents a Buildbot job (via Buildbot bridge), the bridge sets 'taskId' for the Buildbot job while 'upload_to_task_id' is set by mozci when scheduling the TaskCluster task. In other words, 'upload_to_task_id' gets set when scheduling via TaskCluster (and 'taskId' being set by BBB) while if I schedule via Buildapi I could also specify 'taskId' as well as 'upload_task_id'. All that being said, I can push to try the changes you mention and use one of my scripts to schedule the jobs we want to test. [1] http://archive.mozilla.org/pub/mobile/try-builds/jandreou@mozilla.com-6c407ce5676639dbd942ec16f333a44a3cdb81b2/try-android-api-15/try-android-api-15-bm75-try1-build1223.txt.gz [2] 17:39:24 INFO - Taskcluster taskId: Gh55eQhkT-izaGZ6FiVUnw
|
|
||
Comment 18•8 years ago
|
||
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #16) > I'm sorry about it. I should have been more clear. > > I wanted to know if the 'buildid' *has* to be the same for all nightlies > that start at the same time or if it fine that each have a different one. Do > you know? or someone that might know? I've talked about this recently within the team, and was told essentially that no, we don't *require* normalized BuildIDs anymore. There are certainly benefits for that being the case, but theres no hard reason they have to be normalized. I'm happy to "eat crow" [1] later on if I am wrong. [1] https://en.wikipedia.org/wiki/Eating_crow
Flags: needinfo?(bugspam.Callek)
|
|
||
Comment 19•8 years ago
|
||
This completely fell off the radar. martianwars: are we done in here? Should we ignore nightly requests for now until we can finish this off? pulse_actions will soon be able to let users know what happened to their requests.
Summary: Nightlies started with Treeherder don't work well → pulse_actions - Nightlies started with Treeherder don't work well
|
|
||
Updated•8 years ago
|
Flags: needinfo?(kalpeshk2011)
|
|
||
Comment 20•8 years ago
|
||
We have merged two PRs on this issue. https://github.com/mozilla/mozilla_ci_tools/pull/484 and https://github.com/mozilla/mozilla_ci_tools/pull/482
Flags: needinfo?(kalpeshk2011)
|
|
||
Comment 21•8 years ago
|
||
Thanks for the reminder! I will check this week where we fail when requesting nightly builds.
Flags: needinfo?(armenzg)
|
|
||
Comment 22•8 years ago
|
||
KWierso: if you schedule a nightly build today, would you be able to use "add new jobs" from Treeherder? Thanks!
Flags: needinfo?(armenzg) → needinfo?(wkocher)
|
|
||
Comment 23•8 years ago
|
||
tl;dr; I believe we should prevent users to schedule nightly jobs through "add new jobs" until it is a polished flow *and* IFFF there are stakeholders that would strongly want it. Could you please confirm that you don't have strong objections to using trigger new nightly builds from Buildapi instead of "adding new jobs"? martianwars: I'm going to grab this bug. It is getting rather complicated. I requested nightly builds and L10n jobs through "add new jobs". https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=214884d507ee369c1cf14edb26527c4f9a97bf48&filter-searchStr=nightly&group_state=expanded I see the main two issues that were filed for this bug fixed (thanks martianwars!) However, I see other issues which I believe should require us to stop supporting this until we can put effort into fixing it properly. Issues (described post main comment): * L10n jobs are not showing up on Treeherder ** Android jobs seem to show up * Setting upload_to_task_id is getting on the way when re-running nightly builds Recommendations: * Stop offering nightly jobs through "add new jobs" * mozci should not set upload_to_task_id for any nightly builds ** The nightly builds will just work (as fixed for Android nightly build) * Fix mozci's dependency relationship between nightly builds and associated L10n jobs * Determine why the L10n jobs did not show up on Treeherder Other info for my own tracking: ############################### I've purposely skipped selecting N5 for one of the Linux nightly builds to see how sendchanges would behave. You can also see the 'Sch' job being scheduled, the log is here: https://tc-gp-public-31d.s3-us-west-2.amazonaws.com/ateam/pulse-action-dev/cce86f25-1553-4174-ba5f-66fcbf27b0c2 A) Aesthetic bug I've noticed an aesthetic bug were we name all tasks the same regardless if the task definition lists a different buildername (i.e. all tasks are name 'Firefox mozilla-central win64 l10n nightly-1') https://tools.taskcluster.net/task-graph-inspector/#OMPV91ZqTRqTTlGlaY5GFg/fS9OmSesTVWQQ-kVDUXOhA/ B) L10n bug TH does not show up the L10n jobs, however, I can see them on the TC side [1][2][3] From looking it seems that the jobs work all the way. For some reason I see all tasks depending to itself [4] > "dependencies": ["A2waP36oR_m0AagyV2-HxQ"], That probably means that the dependency tree is incorrect. I think L10n jobs started at the same time as the nightly builds. [1] https://tools.taskcluster.net/task-graph-inspector/#OMPV91ZqTRqTTlGlaY5GFg/fl0_-r4QTKadmWJxvoGtfA/0 [2] https://public-artifacts.taskcluster.net/fl0_-r4QTKadmWJxvoGtfA/0/public/properties.json [3] http://archive.mozilla.org/pub/firefox/nightly/2016/07/2016-07-11-14-37-43-mozilla-central-l10n/mozilla-central-macosx64-l10n-nightly-2-unknown-bm86-build1-build3.txt.gz [4] https://queue.taskcluster.net/v1/task/A2waP36oR_m0AagyV2-HxQ C) Sharing upload_to_task_id (re-runs and retriggers) There's another issue, there's something rather complicated happening in here. We have three WINNT 5.2 nightly build jobs on Treeherder. 1) The first one (blue) happened when adding new jobs. 2) Since the job turned blue it automatically retried (pureple job) * The builduid property was not kept between automatic reruns (for some odd reason) * You can see two runs for the same task ID in here [1] 3) You can also see the retrigger request (red job) I did * I though the blue job was not being retried automatically [2]. All three requests shared the same upload_to_task_id 'PQSqi6PNRbKYVsJXVfY6iA' This caused the retrigger's upload to TaskCluster issue (see attachment) since the purple job had already uploaded the file and there was a conflict to upload tasks. [1] https://tools.taskcluster.net/task-inspector/#HiwxrFt4Rn2zuIa3FALpsQ/ [2] https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-central/build/117497017 D) The 'Sch' job does not have information about the graph scheduled Mozci pretty prints the graph into stdout, however, that does not make it into the requests' file logging. We should probably upload the graph into S3 and have a link to it.
Assignee: kalpeshk2011 → armenzg
Status: NEW → ASSIGNED
Flags: needinfo?(wkocher)
Flags: needinfo?(sheriffs)
Flags: needinfo?(ryanvm)
|
||
Updated•8 years ago
|
Flags: needinfo?(sheriffs)
Flags: needinfo?(ryanvm)
|
|
||
Updated•8 years ago
|
Assignee: armenzg → nobody
Status: ASSIGNED → NEW
|
|
||
Comment 24•7 years ago
|
||
Shutting pulse_actions off (see bug 1379172).
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
|
Assignee | |
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•