Closed Bug 1238469 Opened 5 years ago Closed 5 years ago

Refactor b2g tests to remove custom mozharness configs from docker images

Categories

(Taskcluster :: Services, defect, P1)

Product:
Taskcluster
Component:
Services
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla46

People

(Reporter: edgar, Assigned: edgar)

References

Details

Attachments

(4 files, 1 obsolete file)

MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal
58 bytes, text/x-review-board-request
ahal
: review+
Details
MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r=ahal
58 bytes, text/x-review-board-request
ahal
: review+
Details
MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal
58 bytes, text/x-review-board-request
ahal
: review+
Details
MozReview Request: Bug 1238469 - Part 4: Remove custom mozharness configs from docker image; r=ahal
58 bytes, text/x-review-board-request
ahal
: review+
Details 
This has the advantage of cleanliness -- that configs are from in-tree, so it's very easy to edit it as needed.
Assignee: nobody → echen
Blocks: 1235734
Attached patch WIP, Patch, v1 (obsolete) — — Splinter Review 
Note: B2G Desktop is going to be disabled in bug 1236835.
Attachment #8706298 - Attachment is obsolete: true
Comment on attachment 8707326 [details]
MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30305/diff/1-2/
Attachment #8707326 - Attachment description: MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs → MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r?ahal
Attachment #8707326 - Flags: review?(ahalberstadt)
Attachment #8707327 - Attachment description: MozReview Request: Bug 1238469 - Part 2: Refactor b2g gaia tests to remove the usage of custom mozharness configs → MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r?ahal
Attachment #8707327 - Flags: review?(ahalberstadt)
Comment on attachment 8707327 [details]
MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30307/diff/1-2/
Comment on attachment 8707328 [details]
MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30309/diff/1-2/
Attachment #8707328 - Attachment description: MozReview Request: Bug 1238469 - Part 3: Remove custom mozharness configs from docker image → MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r?ahal
Attachment #8707328 - Flags: review?(ahalberstadt)
Comment on attachment 8707326 [details]
MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal

https://reviewboard.mozilla.org/r/30305/#review28071

r+ with renaming the file to use underscores.

::: testing/taskcluster/tasks/tests/b2g_emulator_cpp_unit.yml:15
(Diff revision 2)
> -        --config-file ./mozharness/configs/b2g/emulator_automation_config.py
> +        --config-file ./mozharness/configs/b2g/taskcluster-emulator-automation.py

Please use underscores in the filename. In python, using dashes means the file can't be imported!
Attachment #8707326 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8707326 [details]
MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal

https://reviewboard.mozilla.org/r/30305/#review28073

Actually, this is going to break all the ICS tests. They still depend on emulator_automation_config.py being around. So please copy this file instead of moving it.

[1] https://dxr.mozilla.org/build-central/search?q=emulator_automation_config.py&redirect=false&case=true
Attachment #8707326 - Flags: review+
https://reviewboard.mozilla.org/r/30305/#review28073

Ugh, sorry ignore this comment. You deleted emulator_test_config.py, and I misread it as emulator_automation_config.py.
Comment on attachment 8707327 [details]
MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r=ahal

https://reviewboard.mozilla.org/r/30307/#review28077

Looks good, again with the filenames fixed.

::: testing/mozharness/configs/b2g/taskcluster-desktop-automation.py:1
(Diff revision 2)
>  # This is a template config file for b2g desktop unittest production.

nit: put mulet in this comment instead.

::: testing/taskcluster/tasks/tests/b2g_gaia_unit_oop.yml:15
(Diff revision 2)
> -        --no-read-buildbot-config
> +        --config-file ./mozharness/configs/b2g/taskcluster-gaia-unit-production.py

Same comment as before, please use underscores in the filenames.
Attachment #8707327 - Flags: review?(ahalberstadt) → review+
Attachment #8707328 - Flags: review?(ahalberstadt)
Comment on attachment 8707328 [details]
MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal

https://reviewboard.mozilla.org/r/30309/#review28079

::: testing/mozharness/mozharness/mozilla/testing/unittest.py:68
(Diff revision 2)
> -    def evaluate_parser(self):
> +    def evaluate_parser(self, return_code, success_codes=None):
> +        if return_code == 0 and self.passed > 0 and self.failed == 0:
> +            self.tbpl_status = TBPL_SUCCESS
> +        elif return_code == 10 and self.failed > 0:
> +            self.tbpl_status = TBPL_WARNING
> +        else:
> +            self.tbpl_status = TBPL_FAILURE
> +            self.worst_log_level = ERROR
> +
> +        return (self.tbpl_status, self.worst_log_level)

This parser is used in a few places:
https://dxr.mozilla.org/mozilla-central/search?q=TestSummaryOutputParserHelper

Are you sure this doesn't break any of those things? Would be good to at least run a try run with Luciddream (Ld) and talk to :maja_zf about the firefox-media-tests. Those misc gaia jobs might also be impacted.
Comment on attachment 8709034 [details]
MozReview Request: Bug 1238469 - Part 4: Remove custom mozharness configs from docker image; r=ahal

https://reviewboard.mozilla.org/r/31159/#review28081

Thanks for this patch series, this is really good stuff! So glad someone is giving some much needed attention to the b2g emulators.
Attachment #8709034 - Flags: review?(ahalberstadt) → review+
https://reviewboard.mozilla.org/r/30309/#review28079

> This parser is used in a few places:
> https://dxr.mozilla.org/mozilla-central/search?q=TestSummaryOutputParserHelper
> 
> Are you sure this doesn't break any of those things? Would be good to at least run a try run with Luciddream (Ld) and talk to :maja_zf about the firefox-media-tests. Those misc gaia jobs might also be impacted.

Sure, I will review again and run a try to ensure the changes in parser won't break anything, thank you.
Priority: -- → P3
This blocks P1 bug 1235734.
Priority: P3 → P1
https://reviewboard.mozilla.org/r/30309/#review28079

> Sure, I will review again and run a try to ensure the changes in parser won't break anything, thank you.

I don't think these changes will affect media-tests.
https://reviewboard.mozilla.org/r/30309/#review28079

> I don't think these changes will affect media-tests.

Thanks for the comments, Maja.
And I triggerred a try with luciddream, media-tests and media-youtube-tests, the result looks good, https://treeherder.mozilla.org/#/jobs?repo=try&revision=e5fc2e0f0640&group_state=expanded.
I also triggerred a try with all emulator and gaia tests, the result is also positive, https://treeherder.mozilla.org/#/jobs?repo=try&revision=d98480f3c367&group_state=expanded.
I think the changes in parser doesn't bready anything.
Attachment #8707326 - Flags: review?(ahalberstadt)
Comment on attachment 8707326 [details]
MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30305/diff/2-3/
Comment on attachment 8707327 [details]
MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30307/diff/2-3/
Attachment #8707327 - Attachment description: MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r?ahal → MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r=ahal
Comment on attachment 8707328 [details]
MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30309/diff/2-3/
Attachment #8707328 - Flags: review?(ahalberstadt)
Comment on attachment 8709034 [details]
MozReview Request: Bug 1238469 - Part 4: Remove custom mozharness configs from docker image; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/31159/diff/1-2/
Attachment #8709034 - Attachment description: MozReview Request: Bug 1238469 - Part 4: Remove custom mozharness configs from docker image; r?ahal → MozReview Request: Bug 1238469 - Part 4: Remove custom mozharness configs from docker image; r=ahal
Comment on attachment 8707326 [details]
MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal

https://reviewboard.mozilla.org/r/30305/#review28481

Thanks!
Attachment #8707326 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8707328 [details]
MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal

https://reviewboard.mozilla.org/r/30309/#review28483

Great, if this works out on try it works for me.
Attachment #8707328 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8707326 [details]
MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30305/diff/3-4/
Attachment #8707326 - Attachment description: MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r?ahal → MozReview Request: Bug 1238469 - Part 1: Refactor b2g emulator tests to remove the usage of custom mozharness configs; r=ahal
Comment on attachment 8707327 [details]
MozReview Request: Bug 1238469 - Part 2: Refactor b2g mulet tests to remove the usage of custom mozharness configs; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30307/diff/3-4/
Comment on attachment 8707328 [details]
MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30309/diff/3-4/
Attachment #8707328 - Attachment description: MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r?ahal → MozReview Request: Bug 1238469 - Part 3: Use b2g_emulator_unittest.py for b2g marionette tests; r=ahal
Comment on attachment 8709034 [details]
MozReview Request: Bug 1238469 - Part 4: Remove custom mozharness configs from docker image; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/31159/diff/2-3/
Rebase over bug 1237987 only.
Docker image pushed, 0.4.7, https://hub.docker.com/r/taskcluster/tester/tags/.
(In reply to Edgar Chen [:edgar][:echen] from comment #32)
> Docker image pushed, 0.4.7,
> https://hub.docker.com/r/taskcluster/tester/tags/.

After docker image is pushed, I initiated a try run [1] to test the new image. But I got an error in taskcluster-npm-cache-get on Mulet gaia tests.

====
07:35:54     INFO - retry: Calling run_command with args: [['taskcluster-npm-cache-get', '--namespace', 'gaia.npm_cache', 'package.json'], '/home/worker/gaia'], kwargs: {'output_timeout': 300}, attempt #1
07:35:54     INFO - Running command: ['taskcluster-npm-cache-get', '--namespace', 'gaia.npm_cache', 'package.json'] in /home/worker/gaia
07:35:54     INFO - Copy/paste: taskcluster-npm-cache-get --namespace gaia.npm_cache package.json
07:35:54     INFO - Calling ['taskcluster-npm-cache-get', '--namespace', 'gaia.npm_cache', 'package.json'] with output_timeout 300
07:35:57     INFO -  tar (child): /home/worker/gaia/node_modules.tar.gz: Cannot open: No such file or directory
07:35:57     INFO -  tar (child): Error is not recoverable: exiting now
07:35:57     INFO -  tar: Child returned status 2
07:35:57     INFO -  tar: Error is not recoverable: exiting now
07:35:57     INFO -  Something is wrong...
07:35:57     INFO -  /usr/local/lib/node_modules/taskcluster-npm-cache/build/bin/taskcluster-npm-cache-get.js:186
07:35:57     INFO -      throw e;
07:35:57     INFO -      ^
07:35:57     INFO -  Error: Failed running tar zxf /home/worker/gaia/node_modules.tar.gz code: 2
07:35:57     INFO -      at callee$0$0$ (/usr/local/lib/node_modules/taskcluster-npm-cache/build/run.js:46:15)
07:35:57     INFO -      at tryCatch (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
07:35:57     INFO -      at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)
07:35:57     INFO -      at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
07:35:57     INFO -      at GeneratorFunctionPrototype.invoke (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37)
07:35:57     INFO -      at run (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:104:47)
07:35:57     INFO -      at /usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:115:28
07:35:57     INFO -      at flush (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/core-js/modules/$.microtask.js:19:5)
07:35:57     INFO -      at doNTCallback0 (node.js:419:9)
07:35:57     INFO -      at process._tickCallback (node.js:348:13)
====

After digging into the error by running locally, I found that npm-cache was saved as *public%2Fnode_modules.tar.gz*, not node_modules.tar.gz, which is really strange.

Hi :aus, :garndt, could you shed me some light on this? What could be the problem?

Thank you.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6ce5e009ff4&group_state=expanded
[2] https://hub.docker.com/r/edgarchen/tester/tags/
Flags: needinfo?(garndt)
Flags: needinfo?(aus)
I'm not exactly sure what the issue is here.  I attempted to run it locally but it never downloaded that gzip file and ended up failing because of other issues.  Hopefully :aus might know offhand what's wrong with this.
Flags: needinfo?(garndt)
Some more information:

===
root@497f61a29a5c:/home/worker/gaia# DEBUG=npm-cache:get taskcluster-npm-cache-get --namespace gaia.npm_cache package.json
  npm-cache:get Package hash = +0ms e6b0190d9efdfd86736f0ac326374dc8
  npm-cache:get Package namespace = +2ms gaia.npm_cache.node-v4-2.linux-x64.e6b0190d9efdfd86736f0ac326374dc8
     fetch : https://queue.taskcluster.net/v1/task/UZeFNYxuRsek_0j7CVaMUQ/artifacts/public%2Fnode_modules.tar.gz
  progress : [====================] 100% 0.0s

tar (child): /home/worker/gaia/node_modules.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Something is wrong...
/usr/local/lib/node_modules/taskcluster-npm-cache/build/bin/taskcluster-npm-cache-get.js:186
    throw e;
    ^

Error: Failed running tar zxf /home/worker/gaia/node_modules.tar.gz code: 2
    at callee$0$0$ (/usr/local/lib/node_modules/taskcluster-npm-cache/build/run.js:46:15)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
    at GeneratorFunctionPrototype.invoke (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37)
    at run (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:104:47)
    at /usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:115:28
    at flush (/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/babel/node_modules/babel-core/node_modules/core-js/modules/$.microtask.js:19:5)
    at doNTCallback0 (node.js:419:9)
    at process._tickCallback (node.js:348:13)
==
Maybe there is something wrong on taskcluster-client buildUrl API. The version of taskcluster-client is 0.23.8.

===
Step 17 : RUN npm install -g taskcluster-npm-cache@1.3.1
 ---> Running in d2bb561023fb
npm WARN deprecated lodash@1.0.2: lodash@<2.0.0 is no longer maintained. Upgrade to lodash@^3.0.0pm WARN optional dep failed, continuing fsevents@1.0.6                        
/usr/local/bin/taskcluster-npm-cache-get -> /usr/local/lib/node_modules/taskcluster-npm-cache/build/bin/taskcluster-npm-cache-get.js
/usr/local/bin/taskcluster-npm-cache -> /usr/local/lib/node_modules/taskcluster-npm-cache/build/bin/taskcluster-npm-cache.js
taskcluster-npm-cache@1.3.1 /usr/local/lib/node_modules/taskcluster-npm-cache                                                                                             
├── event-to-promise@0.4.0                                                       
├── denodeify@1.2.1                                                             
├── mz@1.3.0 (thenify@3.1.1, native-or-bluebird@1.2.0, thenify-all@1.6.0)  
├── debug@2.2.0 (ms@0.7.1)                                                 
├── bluebird@2.10.2                                                             
├── slugid@1.1.0 (uuid@2.0.1)                                                
├── argparse@0.1.16 (underscore@1.7.0, underscore.string@2.4.0)                         
├── promise@7.1.1 (asap@2.0.3)                                               
├── promised-temp@0.1.0 (q@1.4.1, temp@0.8.3, mkdirp@0.5.1)                   
├── delete@0.1.5 (is-path-cwd@0.1.0, is-path-in-cwd@0.1.1, rimraf@2.5.1)      
├── download@3.3.0 (get-stdin@3.0.2, gulp-rename@1.2.2, url-regex@2.1.3, stream-combiner@0.2.2, vinyl@0.4.6, through2@0.6.5, ware@1.3.0, each-async@1.1.1, concat-stream@1.5.1, decompress-targz@2.1.0, rc@0.5.5, meow@2.1.0, decompress-unzip@2.1.2, decompress-tarbz2@2.0.2, decompress-tar@2.0.2, vinyl-fs@0.3.14, request@2.67.0)
├── download-status@2.2.1 (object-assign@2.1.1, progress@1.1.8, chalk@0.5.1, lpad-align@1.1.0)
├── taskcluster-client@0.23.8 (url-join@0.0.1, superagent-promise@0.2.0, promise@6.1.0, hawk@2.3.1, superagent-hawk@0.0.6, superagent@1.7.1, amqplib@0.4.0, lodash@3.10.1, sockjs-client@1.0.3)                                                         
└── babel@5.8.35 (path-is-absolute@1.0.0, slash@1.0.0, path-exists@1.0.0, fs-readdir-recursive@0.1.2, convert-source-map@1.1.3, commander@2.9.0, source-map@0.5.3, glob@5.0.15, output-file-sync@1.1.1, lodash@3.10.1, chokidar@1.4.2, babel-core@5.8.35)
 ---> 21fe08edf8b3
=====
* Good version is 0.23.5

--
root@edgar-mozilla-pc:/data/docker# docker run taskcluster/tester:0.4.6 npm list taskcluster-client -g
/usr/local/lib
├─┬ taskcluster-npm-cache@1.3.1
│ └── taskcluster-client@0.23.5 
└─┬ taskcluster-vcs@2.3.12
  └── taskcluster-client@0.19.5 


===================

*Bad version is 0.23.8

--
root@edgar-mozilla-pc:/data/docker# docker run taskcluster/tester:0.4.7 npm list taskcluster-client -g
/usr/local/lib
├─┬ taskcluster-npm-cache@1.3.1
│ └── taskcluster-client@0.23.8 
└─┬ taskcluster-vcs@2.3.12
  └── taskcluster-client@0.19.5
Depends on: 1242877
Blocks: 1243085
Moving closed bugs across to new Bugzilla product "TaskCluster".
status-firefox47: fixed → ---
Component: TaskCluster → Integration
Product: Testing → Taskcluster
Target Milestone: mozilla47 → mozilla46
Component: Integration → Services
You need to log in before you can comment on or make changes to this bug.