629503 - copy tp4.manifest to mobile_tp4.manifest, cut pages from mobile_tp4.manifest until n900s go green, then put live

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

Per DougT.

Higher priority than release work per Stuart.

Comment 1

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: mobile_tp4.manifest (try #1)

This manifest is a cut down version of the full tp4 manifest. It has 25 sites, instead of the 100 sites in the full manifest. The 25 sites are a mix of languages, I didn't try to focus on English only.

How I choose the sites is arbitrary, but we don't have a real mobile-specific pageset yet.

Comment 2

[Aki Sasaki (not active)]

Attachment: new mobile_tp4.manifest

Also update mobile.config to point to it.

N900 talos jobs download talos from a tarball.
I'll update the staging tarball with this patch and let it run tp4 for a while.
It should report to the MobileTest tinderbox tree:

http://tinderbox.mozilla.org/showbuilds.cgi?tree=MobileTest

Comment 3

[Aki Sasaki (not active)]

Running in staging. MobileTest may be behind since tinderbox.m.o was having issues earlier, or I may have something to fix.

I'll keep these running and report on status; hopefully they'll show up on tinderbox soon.

Comment 4

[Aki Sasaki (not active)]

Attachment: n900 qt m-c tp4 nochrome failures

Comment 5

[Aki Sasaki (not active)]

Attachment: n900 gtk m-c tp4 (chrome)

The chrome-enabled tp4 runs don't even list the pages they're loading.

Comment 6

[Aki Sasaki (not active)]

Attachment: n900-qt m-c tp4 (chrome)

Same with the qt non-nochrome (chrome) run -- no pages listed.
I'm going to guess this is a side effect of pageloader in e10s?

Comment 7

[Aki Sasaki (not active)]

Attachment: n900-gtk m-c tp4 nochrome

this looks similar to the n900-qt tp4 nochrome run.

Comment 8

[Aki Sasaki (not active)]

We're orange across the board with the smaller pageset.  We're not finishing cycle 1, let alone 3 cycles of 25 pages.

Comment 9

[Aki Sasaki (not active)]

Ok, these are starting to show up on MobileTest so I'll stop pasting logs. (Ignore the reds; these are orange).

8/8 runs are orange so far.

Comment 10

[Mark Finkle (:mfinkle) (use needinfo?)]

Something is busted trying to start the test. We don't even make it to the real test. Looks like a e10s issue, but it should work since we use the pageloader for other tests.

I can try this on my N900

Comment 11

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: mobile_tp4.manifest (try #2)

This smaller manifest removes a few pages that were crashing Fennec running Tp with chrome (e10s mode). I can successfully run Tp4 with this manifest on my N900.

I have a patch to pageloader to allow it to run using chrome (e10s mode). The nochrome mode should work OK with this smaller manifest too.

Comment 12

[Mark Finkle (:mfinkle) (use needinfo?)]

Needs patched pageloader (bug 629691) and a "-tpdelay 1000" in the config. I can pass 3 cycles on my N900

Comment 13

[Aki Sasaki (not active)]

Attachment: talos patch with mfinkle's changes

With this patch, and the pageloader patch from bug 629691, the staging n900s running m-c tp4 (gtk, gtk nochrome, qt, qt nochrome) all went green.

Comment 14

[Aki Sasaki (not active)]

Attachment: enable running tp4 in production on n900s

Also remove the obsolete non-n900 configs.

Comment 15

[Mark Finkle (:mfinkle) (use needinfo?)]

Great stuff! After watching MobileTest for a bit, I see some orange, but I'm not sure how much is OOM and how much is crashes in X. In any case, we are doing much better now.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 508076 [details] [diff] [review]
talos patch with mfinkle's changes

we should make the same change to remote.config as well.  Also we will be testing a much smaller set of web pages on each build, so our 'coverage' is reduced.  We should consider running a full cycle for functionality testing instead of performance testing.  All of this feedback belongs in other bugs.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

one question I had about this is I don't see -tpdelay on the tdhtml, tsspider, tgfx, and tsvg tests?  I think we should be consistent and have all TP style tests act the same.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: update all tp tests and remote.config with tpdelay changes (1.0)

this patch supercedes the 'talos patch with mfinkle's changes' such that it adds the -tpdelay to all TP style tests and replicates all these changes from mobile.config into remote.config as well.

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 508427 [details] [diff] [review]
update all tp tests and remote.config with tpdelay changes (1.0)

I've got this running on n900s in staging.

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 508427 [details] [diff] [review]
update all tp tests and remote.config with tpdelay changes (1.0)

I'm ok with this, as long as the mobile team knows that their numbers may shift when this lands.

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

http://hg.mozilla.org/build/talos/pushloghtml?changeset=7e3bd009ed05

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 508077 [details] [diff] [review]
enable running tp4 in production on n900s

http://hg.mozilla.org/build/buildbot-configs/rev/87d7d09620cd

Comment 23

[Aki Sasaki (not active)]

production-mobile-master reconfiged. I think we're done here.