941659 - adjust android automation to run tsvgx with fewer cycles

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

there are two ways to solve this problem:
1) add a --tppagecycles 10 (vs the default of 25) to remote-tsvgx jobs
2) add a new test suite in talos (android-tsvgx) which remote-tsvgx will call and that will have the appropriate default

As we don't have mozharness, we will need to adjust the releng scripts for both solutions.  The only advantage of #2 is if we want to change the number of page cycles in the future (not sure why we would), we could do it with a talos.zip update, not a buildbot-custom patch.

:Callek- which would you prefer?

Comment 1

[Justin Wood (:Callek)]

(In reply to Joel Maher (:jmaher) from comment #0)
> there are two ways to solve this problem:
> 1) add a --tppagecycles 10 (vs the default of 25) to remote-tsvgx jobs
> 2) add a new test suite in talos (android-tsvgx) which remote-tsvgx will
> call and that will have the appropriate default
> 
> As we don't have mozharness, we will need to adjust the releng scripts for
> both solutions.  The only advantage of #2 is if we want to change the number
> of page cycles in the future (not sure why we would), we could do it with a
> talos.zip update, not a buildbot-custom patch.
> 
> :Callek- which would you prefer?

I prefer adding --tppagecycles 10 over adding yet a new suite. IFF we are ok with doing this across all trees at once, rather than have it ride the trains.

I however don't have time to write the patch at this moment, and likely won't until after the december vacation, so needinfo to jordan to see if he feels comfortable doing this, and needinfo to joel to get an answer on my IFF.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

I am ok with it being changed across all the trees.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Attachment: set --tppagecycles 10 for remote-tsvgx (1.0)

Comment 4

[Jordan Lund (:jlund)]

I started looking at this on Friday. I'm just on my way back to Vancouver now so I will finish this on Monday.

AIUI, our tegra (android 2.2) tests do not use Mozharness (as mentioned) but I think our panda (android 4.0) does avail of it via 'scripts/android_panda_talos.py':
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/mobile_config.py#90

So if I am right and we wish to change tppagecycles for all versions of android, I think we will need to also patch talos.json here:
http://mxr.mozilla.org/mozilla-central/source/testing/talos/talos.json#85

But I'll try and test this out on Monday. :)

Comment 5

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

you are right, we use mozharness for panda, but I don't believe we use talos.json.  I know we had worked on that prior to :kmoir's leave, but we had to back it out due to a random issue on mozilla-beta.  We should still update talos.json to match this.

Comment 6

[Jordan Lund (:jlund)]

ah ok, I'm coming to grasp with this now.

yes, I found: Bug 932640 - tpn on Pandas on mozilla-beta broken

So now IIUC, looking at this patch to revert talos.json:
https://bug920757.bugzilla.mozilla.org/attachment.cgi?id=824711

we would need to change: 
http://mxr.mozilla.org/build/source/mozharness/configs/android/android_panda_talos_releng.py#46

to:
"remote-tsvgx":  ["--activeTests=tsvgx", "--noChrome", '--tppagecycles', '10'],

so that it's picked up here in preflight_talos():
http://mxr.mozilla.org/build/source/mozharness/scripts/android_panda_talos.py#157

And, as per comment 5, I'll also patch talos.json so it mirrors behavior.

Comment 7

[Jordan Lund (:jlund)]

Attachment: bug_941659_tppages_mh.diff

Sorry Joel, it's my first time looking at mozharn talos panda. Is my logic right or am I once again not 100% correct and we don't need this patch?

Comment 8

[Jordan Lund (:jlund)]

Attachment: bug_941659_tppages_inbound.diff

this is a no-op patch as we do not use talos.json for mozharness panda talos tests. However we plan to do so again in the future so let's keep this in sync with current args

Comment 9

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

landed the buildbot patch:
https://hg.mozilla.org/build/buildbot-configs/rev/9b0db4adf953

Comment 10

[Jordan Lund (:jlund)]

landed mozharness patch to default:
 https://hg.mozilla.org/build/mozharness/rev/1503129e0bef

Comment 11

[Jordan Lund (:jlund)]

landed inbound patch:
 https://hg.mozilla.org/integration/mozilla-inbound/rev/8efca7a458ce

I made a mistake, I put this under tcanvasmark where it should be tsvgx. It was a no-op move so should be safe, creating a patch now to fix.

Comment 12

[Jordan Lund (:jlund)]

Attachment: bug_941659_tppages_inbound.261113.diff

Comment 13

[John Hopkins (:jhopkins)]

http://hg.mozilla.org/build/buildbot-configs/rev/9b0db4adf953 is in production

Comment 14

[Jordan Lund (:jlund)]

fix pushed to inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/614bb2bcee10

Comment 15

[John Hopkins (:jhopkins)]

https://hg.mozilla.org/build/mozharness/rev/1503129e0bef is in production

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/8efca7a458ce
https://hg.mozilla.org/mozilla-central/rev/614bb2bcee10