1347678 - fix AWSY taskcluster nits

Status: RESOLVED FIXED

(Testing :: AWSY, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

from comment 71 in bug 1272113:
> > 
> > ::: taskcluster/ci/test/tests.yml:1255
> > (Diff revisions 4 - 6)
> > > -        no-read-buildbot-config: true
> > >          config:
> > >              by-test-platform:
> > >                  default:
> > >                      - awsy/linux_config.py
> > >                      - remove_executables.py
> > 
> > I am not sure if we need remove_executables.py here, I thought we added that
> > as a way to work around buildbot scripts that still needed to run.

We added this remove_executables.py so jobs that originally ran in buildbot could run in buildbot and taskcluster.  Since this doesn't run in buildbot, I wonder why we need this?  I don't want to just copy everything else.

> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/
> remove_executables.py
> I doubt it's relative for buildbot; it seems to avoid download of
> minidump_stackwalk and nodejs.
> But it looks like a workaround since most of test applied this config. 
> Perhaps we can take it out but definitely need someone to provide more
> information.
>  
> > ::: testing/mozharness/scripts/awsy_script.py:34
> > (Diff revisions 4 - 6)
> > >          [["--e10s"],
> > >          {"action": "store_true",
> > >           "dest": "e10s",
> > >           "default": False,
> > >           "help": "Run tests with multiple processes. (Desktop builds only)",
> > >           }]
> > 
> > as a note, if we are only running in e10s, do we need this cli option? 
> > Possibly we do as this is needed to work smoothly with taskcluster.
> 
> Agreed this option confuses people.  We might need to work on all those
> options and make sure e10s is default.

this doesn't answer my question- do we need this option or not?  Since we are running in e10s only I would believe we do not need this option- do let me know if it is needed.



lets get these resolved soon!

Comment 1

[Paul Yang [:pyang] (away)]

I have finds for this.

remove_executables.py actually purge exec in original config so we can simply take out items in exec and it should works fine.
try result showed in https://treeherder.mozilla.org/#/jobs?repo=try&revision=981199f5283f9b755e1ada99b2e0a403bda040b7

We probably need more explicit definitions for these config and find better ways to fix.

---

For e10s option,
I guess we discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1272113#c45
, and a try run log https://public-artifacts.taskcluster.net/SWfrtsfYRliqvnfTQyZ-6w/0/public/logs/live_backing.log (search "no such option: --e10s") shows it bumped error when we didn't provide e10s option.
Possible solutions mentioned in previous comments.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

can we default e10s=True?  I am glad we can easily delete the remove_executables.py line :)

Comment 3

[Paul Yang [:pyang] (away)]

That is what we have in treeherder when using "-u all"; and e10s=True is certainly included in our last patch.
Furthermore, for taking out e10s options, I can give a poc to show how to work.

Comment 4

[Paul Yang [:pyang] (away)]

Attachment: Bug 1347678 - enabling on linux-opt/linux64-opt

Review commit: https://reviewboard.mozilla.org/r/122640/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/122640/

Comment 5

[Paul Yang [:pyang] (away)]

Attachment: Bug 1347678 - removing dependency for remove_executables.py

Review commit: https://reviewboard.mozilla.org/r/122642/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/122642/

Comment 6

[Paul Yang [:pyang] (away)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

forgot to fix merge failure in bug1272113 so we actually didn't enable awsy on treeherder.
try result look good:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=51b5b6a08d6bb253e873ae1436cdc8b7ed4eec00

Comment 7

[Paul Yang [:pyang] (away)]

Comment on attachment 8849901 [details]
Bug 1347678 - removing dependency for remove_executables.py

remove_executables.py can be eliminated in tc's config.
try result showed in comment 1

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

https://reviewboard.mozilla.org/r/122640/#review124820

just have one question, happy to flip to a r+ if this is expected.

::: taskcluster/ci/test/test-platforms.yml:38
(Diff revision 1)
>      test-sets:
>          -  linux32-tests
>          -  linux32-opt-tests
>          - external-media-tests
>          - external-media-tests-slow
>          - awsy

since we are enabling per push, is there value to get out of keeping this on the nightly scheduler?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8849901 [details]
Bug 1347678 - removing dependency for remove_executables.py

https://reviewboard.mozilla.org/r/122642/#review124824

looks good

Comment 10

[Paul Yang [:pyang] (away)]

(In reply to Joel Maher ( :jmaher) from comment #8)
> Comment on attachment 8849900 [details]
> Bug 1347678 - enabling on linux-opt/linux64-opt
> 
> https://reviewboard.mozilla.org/r/122640/#review124820
> 
> just have one question, happy to flip to a r+ if this is expected.
> 
> ::: taskcluster/ci/test/test-platforms.yml:38
> (Diff revision 1)
> >      test-sets:
> >          -  linux32-tests
> >          -  linux32-opt-tests
> >          - external-media-tests
> >          - external-media-tests-slow
> >          - awsy
> 
> since we are enabling per push, is there value to get out of keeping this on
> the nightly scheduler?

I am not quite sure if it's redundant, and removing awsy in nightly test-set is ok for me.
Joel, if that makes sense I can update the patch.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

I think just scheduling it per push makes the most sense.  Ensure that is the case for linux32 and linux64.

Comment 12

[Paul Yang [:pyang] (away)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/122640/diff/1-2/

Comment 13

[Paul Yang [:pyang] (away)]

Attachment: Bug 1347678 - removing dependency for remove_executables.py

Review commit: https://reviewboard.mozilla.org/r/122662/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/122662/

Comment 14

[Paul Yang [:pyang] (away)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

- remove awsy in linux-night test-set

Comment 15

[Paul Yang [:pyang] (away)]

Comment on attachment 8849918 [details]
Bug 1347678 - removing dependency for remove_executables.py

carrying r+

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

https://reviewboard.mozilla.org/r/122640/#review124848

thanks for the update

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8849918 [details]
Bug 1347678 - removing dependency for remove_executables.py

https://reviewboard.mozilla.org/r/122662/#review124852

Comment 18

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 620e90f6b818 -d adc2959fab79: rebasing 383201:620e90f6b818 "Bug 1347678 - enabling on linux-opt/linux64-opt r=jmaher"
merging taskcluster/ci/test/test-platforms.yml
warning: conflicts while merging taskcluster/ci/test/test-platforms.yml! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)

Comment 19

[Paul Yang [:pyang] (away)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/122640/diff/2-3/

Comment 20

[Paul Yang [:pyang] (away)]

Comment on attachment 8849918 [details]
Bug 1347678 - removing dependency for remove_executables.py

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/122662/diff/1-2/

Comment 21

[Paul Yang [:pyang] (away)]

Comment on attachment 8849900 [details]
Bug 1347678 - enabling on linux-opt/linux64-opt

Commit rebased.
Result looks ok.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1565644a43a8432b4e2a10bb75e95d723803075d

Carry review result

Comment 22

[Paul Yang [:pyang] (away)]

Comment on attachment 8849918 [details]
Bug 1347678 - removing dependency for remove_executables.py

Commit rebased.
Result looks ok.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1565644a43a8432b4e2a10bb75e95d723803075d

Carry review result in comment 17

Comment 23

[Paul Yang [:pyang] (away)]

r+ from comment 16 and comment 17

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f54f6455a648
enabling on linux-opt/linux64-opt r=jmaher
https://hg.mozilla.org/integration/autoland/rev/78e8bd1ed7a3
removing dependency for remove_executables.py r=jmaher

Comment 25

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/f54f6455a648
https://hg.mozilla.org/mozilla-central/rev/78e8bd1ed7a3

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7ac409daeb94
https://hg.mozilla.org/releases/mozilla-beta/rev/0139170a19a7