723667 - turn on robocop talos tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

We need to replace tpan with a robocop test.  We also have tcheckerboard as well.

These will run just like other tests by doing the PerfConfigurator.py step with --activeTests=trobocop or --activeTests trobocheck.

We need to ensure that robocop.apk is installed properly first in bug 715215.  The other issue is figuring out where fennec_ids.txt will come from.  Right now this lives in tests.zip, but we don't download that for each talos run.  Would it be better to have this in the upload directory as we do fennec.apk and robocop.apk?

Comment 1

[Aki Sasaki (not active)]

We need to stop proceeding any further with robocop until l10n packaging is fixed.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Aki Sasaki [:aki] from comment #1)
> We need to stop proceeding any further with robocop until l10n packaging is
> fixed.

Aki: can you add the dependency here then, please?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the core of trobopan is landed, it is ready to start running.

note to releng: this requires us to have a fennec_ids.txt file somewhere in the food chain that we can point talos at.  We need to discuss where you might already have this file lying around or the best place to get it.

I could upload the fennec_ids.txt to the upload directory, or we could get it from the tests.zip.

Comment 4

[Hal Wine [:hwine] use NI!]

(In reply to Joel Maher (:jmaher) from comment #3)
> note to releng: this requires us to have a fennec_ids.txt file somewhere in
> the food chain that we can point talos at.  We need to discuss where you
> might already have this file lying around or the best place to get it.
> 
> I could upload the fennec_ids.txt to the upload directory, or we could get
> it from the tests.zip.

We need more info to answer the question - how often does it change? Where is it built? is it big enough that copy is an issue? etc.

If it's something you generate, then likely tests.zip is the best place.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

the problem is tests.zip is massive (50mb) and we would have to unpack a bit of it to get a small <1k text file.  We do this already for all other test types, so if we want to do this for talos, then we can and call it good.

Alternatively I could upload fennec_ids.txt to the upload directory and it would live next to the .apk files.  This is sort of hacky, but it would make it easier in some sense.  

we generate fennec_ids.txt with each build as it is a map of the UI elements that are defined in fennec (these change quite frequently now, and probably on a regular basis going forward).

I am not sure of the exact target, but if it goes into m-c now, it would be fennec-13.  We might want to downport a bunch of tests to fennec-12 (currently m-a) so we could get good test coverage on that release.  I don't believe we would consider fennec-11.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Here is an example commandline that I run locally:
python remotePerfConfigurator.py -v -e org.mozilla.fennec_jmaher -a tcheckerboard --output test.yml --develop --remoteDevice 192.168.1.103 --fennecIDs=fennec_ids.txt

Here is what it might look like adapted to a commandline that I see in the tbpl logs:
python remotePerfConfigurator.py -v -e org.mozilla.fennec -t tegra-074 --branchName Mozilla-Inbound --resultsServer graphs-old.mozilla.org --resultsLink /server/collect.cgi --symbolsPath ../symbols --remoteDevice 10.250.49.62 --sampleConfig remote.config --output local.yml --webServer bm-remote.build.mozilla.org --browserWait 60 --activeTests tcheckerboard --fennecIDs=fennec_ids.txt

The changes are just the values we pass to --activeTests and the --fennecIDs which is what we have been discussing in this bug to date.

the new values for --activeTests are:
 * --activeTests tcheckerboard
 * --activeTests trobopan

In my example above I am assuming we have a file called fennec_ids.txt in the same folder as run_tests.py.  You can have this point to anywhere on the local file system that the python script has access to.

Comment 7

[Armen [:armenzg]]

I will be taking this. jmaher do you have some time tomorrow in the morning to help me understand this better? I will be looking at how talos for the tegras work this afternoon.

Comment 8

[Armen [:armenzg]]

https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-1beafade1870/try-android/

Comment 9

[Armen [:armenzg]]

I have a patch on staging that I hope will give results soon after I managed to keep my tegra connected.
I should have a patch for review tomorrow (depending on buildduty it could be today).

Comment 10

[Armen [:armenzg]]

I have a patch to download the fennec_ids.txt.

I hope on Tuesday the boards actually cooperate with me.

jmaher: is it enough to have the fennec_ids.txt in /builds/tegra-XXX/talos-data/talos?

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

yes, that is the ideal location.

Comment 12

[Armen [:armenzg]]

I am having trouble getting this to work:
http://tinderbox.mozilla.org/showlog.cgi?log=MobileTest/1330440249.1330442219.12671.gz&fulltext=1
http://tinderbox.mozilla.org/showlog.cgi?log=MobileTest/1330440249.1330441363.10021.gz&fulltext=1

jmaher is helping with this.

> getting files in '/mnt/sdcard/tests/profile/minidumps/'
> Failed trobopan: 
> 		Stopped Tue, 28 Feb 2012 06:57:27
> FAIL: Busted: trobopan
> FAIL: unrecognized output format
> Traceback (most recent call last):
>   File "run_tests.py", line 659, in <module>
>     main()
>   File "run_tests.py", line 656, in main
>     test_file(arg, options, parser.parsed)
>   File "run_tests.py", line 597, in test_file
>     raise e
> utils.talosError: 'unrecognized output format'
> program finished with exit code 1
> elapsedTime=161.549437
> TinderboxPrint:<a href = "http://hg.mozilla.org/mozilla-central/rev/6989376471f7">rev:6989376471f7</a>
> 
> TinderboxPrint:FAIL: Busted: trobopan
> 
> TinderboxPrint:FAIL: unrecognized output format

Comment 13

[Armen [:armenzg]]

After I changed it to download to "build" I am actually sure that robocop.apk got installed:
http://dev-master01.build.scl1.mozilla.com:8043/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocheck/builds/34
http://tinderbox.mozilla.org/showlog.cgi?log=MobileTest/1330440249.1330441363.10021.gz&fulltext=1
but we are still having the "unrecognized output format".

> devroot /mnt/sdcard/tests
> Installing /mnt/sdcard/tests/robocop.apk
> in push file with: build/robocop.apk, and: /mnt/sdcard/tests/robocop.apk
> local hash returned: '2294cdf71a71eac8d7f4611d2aa0cff4'
> sending: push /mnt/sdcard/tests/robocop.apk
> push returned: 2294cdf71a71eac8d7f4611d2aa0cff4

Comment 14

[Armen [:armenzg]]

Attachment: [wip] patch for robocop and talos.zip

This is so you can have some visibility on what the patch is looking like.
I need to adjust it so it only adds the robocop step for the robocop jobs.

Comment 15

[Armen [:armenzg]]

jmaher told me to pass --fennecIDs=<path>/fennec_ids.txt which you can see in the step [1] as --fennecIDs ../fennec_ids.txt since the pwd is /builds/tegra-026/test/../talos-data/talos

fenned_ids.txt is in 'talos-data'.

It now simply stops without much explanation [2].
> FIRE PROC: 'org.mozilla.fennec  -profile /mnt/sdcard/tests/profile http://bm-remote.build.mozilla.org/getInfo.html'
> process killed by signal 15

[1] http://dev-master01.build.scl1.mozilla.com:8043/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocheck/builds/37/steps/Update%20config/logs/stdio
[2] http://dev-master01.build.scl1.mozilla.com:8043/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocheck/builds/37/steps/Run%20performance%20tests/logs/stdio

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

the talos changes are landed and here is a talos.zip bundle:
http://people.mozilla.org/~jmaher/taloszips/zips/talos.mobile.zip

We still need to resolve the OOM, but with this talos bundle you will get a lot further along than you were getting before.

Shall we land this bundle somewhere?  Could we just add this to talos.json?

Comment 17

[Armen [:armenzg]]

I have the bundle on build.m.o. For now nothing else to be done.

I am getting this with robocop:
http://tinderbox.mozilla.org/showlog.cgi?log=MobileTest/1330618380.1330619117.30870.gz&fulltext=1#err0

error returned from pull: could not find metadata
error returned from pull: could not find metadata
reconnecting socket
DeviceManager: error pulling file '/mnt/sdcard/tests/browser_output.txt': No such file or directory
...
DeviceManager: error pulling file '/mnt/sdcard/tests/browser_output.txt': No such file or directory
getting files in '/mnt/sdcard/tests/profile/minidumps/'
Failed tcheckerboard: 
		Stopped Thu, 01 Mar 2012 08:25:17
FAIL: Busted: tcheckerboard
FAIL: no output from browser
Traceback (most recent call last):
  File "run_tests.py", line 666, in <module>
    main()
  File "run_tests.py", line 663, in main
    test_file(arg, options, parser.parsed)
  File "run_tests.py", line 604, in test_file
    raise e
utils.talosError: 'no output from browser'
program finished with exit code 1
elapsedTime=419.815983

Comment 18

[Armen [:armenzg]]

I also got this on another suite:
removing file: /mnt/sdcard/tests/browser_output.txt
NOISE: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileInputStream.init]"  nsresult: "0x80520012 (NS_ERROR_FILE_NOT_FOUND)"  location: "JS frame :: chrome://pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 634"  data: no]
reconnecting socket
FIRE PROC: 'org.mozilla.fennec  -profile /mnt/sdcard/tests/profile -tp page_load_test/tp4m.manifest -tpnoisy -tpformat tinderbox -tpcycles 2 -rss -tpdelay 1000'
NOISE: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileInputStream.init]"  nsresult: "0x80520012 (NS_ERROR_FILE_NOT_FOUND)"  location: "JS frame :: chrome://pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 634"  data: no]
getting files in '/mnt/sdcard/tests/profile/minidumps/'
Failed tp4m: 
		Stopped Wed, 29 Feb 2012 14:17:16
FAIL: Busted: tp4m
FAIL: unrecognized output format
Traceback (most recent call last):
  File "run_tests.py", line 659, in <module>
    main()
  File "run_tests.py", line 656, in main
    test_file(arg, options, parser.parsed)
  File "run_tests.py", line 597, in test_file
    raise e
utils.talosError: 'unrecognized output format'
program finished with exit code 1
elapsedTime=1324.439399

Comment 19

[Armen [:armenzg]]

and this:
NOISE: _x_x_mozilla_cycle_collect,1027
NOISE: __end_cc_report
NOISE: __startTimestamp1330591597557__endTimestamp
NOISE: 
getting files in '/mnt/sdcard/tests/profile/minidumps/'
Failed tsvg: 
		Stopped Thu, 01 Mar 2012 09:02:55
FAIL: Busted: tsvg
FAIL: unrecognized output format
Traceback (most recent call last):
  File "run_tests.py", line 666, in <module>
    main()
  File "run_tests.py", line 663, in main
    test_file(arg, options, parser.parsed)
  File "run_tests.py", line 604, in test_file
    raise e
utils.talosError: 'unrecognized output format'
program finished with exit code 1
elapsedTime=1310.658404

and this one:
DeviceManager: error pulling file '/mnt/sdcard/tests/browser_output.txt': No such file or directory
getting files in '/mnt/sdcard/tests/profile/minidumps/'
Failed trobopan: 
		Stopped Thu, 01 Mar 2012 07:01:27
FAIL: Busted: trobopan
FAIL: no output from browser
Traceback (most recent call last):
  File "run_tests.py", line 666, in <module>
    main()
  File "run_tests.py", line 663, in main
    test_file(arg, options, parser.parsed)
  File "run_tests.py", line 604, in test_file
    raise e
utils.talosError: 'no output from browser'
program finished with exit code 1
elapsedTime=425.284433

Comment 20

[Armen [:armenzg]]

http://dev-master01.build.scl1.mozilla.com:8043/buildslaves/tegra-025
http://dev-master01.build.scl1.mozilla.com:8043/buildslaves/tegra-026

Comment 21

[Armen [:armenzg]]

Updated summary:
* bug 731637 - we hit a permanent OOM issue with robocop
* I will use this bug to switch over to talos.zip model while bug 731637 gets figured it out

Comment 22

[Armen [:armenzg]]

Attachment: move to talos.zip and prepare for robocop

I am doing a last run with tegra-025 (since it is the one behaving).

You can wait until tomorrow for me to post back on the results but it is looking good so far.

I am very inclined to switch all DownloadFile steps to be retry capable. What do you think?

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 603431 [details] [diff] [review]
move to talos.zip and prepare for robocop

Review of attachment 603431 [details] [diff] [review]:
-----------------------------------------------------------------

overall, I like it.  I am r-'ing this due to the self.suies.find('robocop').

::: process/factory.py
@@ +7441,5 @@
>           haltOnFailure=True,
>           ignore_certs=self.ignoreCerts,
>           name="Download build",
>          ))
> +        if self.OS == "android" and self.suites.find('robocop') != -1:

this won't work because tcheckerboard uses robocop.  Can we key off --fennecIDs being present instead?

@@ +7444,5 @@
>          ))
> +        if self.OS == "android" and self.suites.find('robocop') != -1:
> +            def get_fennec_ids_url(build):
> +                url = build.source.changes[-1].files[0]
> +                return url[:-len('fennec-XX.0XX.en-US.android-arm.apk')] + "fennec_ids.txt" 

this seems fragile, but will work for now, maybe a comment?

Comment 24

[Mike Taylor [:bear]]

Comment on attachment 603431 [details] [diff] [review]
move to talos.zip and prepare for robocop

cannot wait for mozharness so this kind of code can be forgotten!

is robocop an unique enough string?  like Joel mentioned in his review I would be worried that it may cause more test suites than you expect to add these steps

Comment 25

[Armen [:armenzg]]

Attachment: move to talos.zip and prepare for robocop (take 2)

Comment 26

[Armen [:armenzg]]

Attachment: add robocop tests disabled

Let's get it in until we figure out the last issues for robocop (I got unrecognized output on my last run).

Comment 27

[Armen [:armenzg]]

ftr robocop jobs failed without much in the logs:
reconnecting socket
RETURN:<a href = "http://hg.mozilla.org/mozilla-central/rev/3dcb40ebd487">rev:3dcb40ebd487</a>
tegra-025: 
		Started Wed, 07 Mar 2012 13:15:20
Running test trobopan: 
		Started Wed, 07 Mar 2012 13:15:20
reconnecting socket
FIRE PROC: 'org.mozilla.fennec  -profile /mnt/sdcard/tests/profile http://bm-remote.build.mozilla.org/getInfo.html'
process killed by signal 15
program finished with exit code -1
elapsedTime=426.407035

[1]
http://dev-master01.build.scl1.mozilla.com:8043/builders/Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobocop/builds/41/steps/Run%20performance%20tests/logs/stdio

Comment 28

[Phil Ringnalda (:philor)]

You might have noticed by the 938 emails that I've sent you (except you're too cunning to watch all of releng, aren't you?) about it through bug comments that bug 660480 is not uncommon, though ;)

If you're using production slaves, I'd start getting interested in killed by signal 15 after it happens 3 times in a row on 2 slaves; if you're using staging slaves, which are the most vile and uncooperative of a population that's nothing but vile and uncooperative, I'd rerun something more like 10 times on at least 3 slaves before getting interested in a lack of successes.

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 603820 [details] [diff] [review]
add robocop tests disabled

Review of attachment 603820 [details] [diff] [review]:
-----------------------------------------------------------------

one small change.

::: mozilla-tests/config.py
@@ +308,5 @@
> +        'options': (TALOS_REMOTE_FENNEC_OPTS, ANDROID),
> +    },
> +    'remote-trobocheck': {
> +        'enable_by_default': False,
> +        'suites': GRAPH_CONFIG + ['--activeTests', 'trobopan', '--noChrome', '--fennecIDs', '../fennec_ids.txt'],

s/trobopan/tcheckerboard/

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 603819 [details] [diff] [review]
move to talos.zip and prepare for robocop (take 2)

Review of attachment 603819 [details] [diff] [review]:
-----------------------------------------------------------------

this looks right.  I am not a factory.py expert, but nothing stands out as questionable or incorrect.

Comment 31

[Armen [:armenzg]]

Comment on attachment 603819 [details] [diff] [review]
move to talos.zip and prepare for robocop (take 2)

r+ from IRC.

Comment 32

[Armen [:armenzg]]

Comment on attachment 603819 [details] [diff] [review]
move to talos.zip and prepare for robocop (take 2)

http://hg.mozilla.org/build/buildbotcustom/rev/d65002e32e87
http://hg.mozilla.org/build/buildbot-configs/rev/2f62125bc466

Comment 33

[Armen [:armenzg]]

These changes went live in a reconfiguration around 9AM.

Comment 34

[Armen [:armenzg]]

Comment on attachment 603820 [details] [diff] [review]
add robocop tests disabled

http://hg.mozilla.org/build/buildbot-configs/rev/2f62125bc466

Comment 35

[Armen [:armenzg]]

While I am still working on the sut agent upgrade I have noticed that I get this cannot remove browser_output.txt.[1]

In fact, when I list the directory I don't see it there.
$>ls /mnt/sdcard/tests
profile
robocop.apk
fennec-13.0a1.en-US.android-arm.apk

[1]
reconnecting socket
RETURN:<a href = "http://hg.mozilla.org/mozilla-central/rev/3dcb40ebd487">rev:3dcb40ebd487</a>
tegra-022: 
		Started Mon, 12 Mar 2012 14:35:37
Running test trobopan: 
		Started Mon, 12 Mar 2012 14:35:37
reconnecting socket
FIRE PROC: 'org.mozilla.fennec  -profile /mnt/sdcard/tests/profile http://bm-remote.build.mozilla.org/getInfo.html'
reconnecting socket
FIRE PROC: 'am instrument -w -e class org.mozilla.fennec.tests.testPan org.mozilla.roboexample.test/android.test.InstrumentationTestRunner'
DeviceManager: error pulling file '/mnt/sdcard/tests/browser_output.txt': No such file or directory
reconnecting socket
pushing directory: /tmp/tmp79VX21/profile to /mnt/sdcard/tests/profile
	Screen width/height:1024/768
	colorDepth:24
	Browser inner width/height: 980/671

removing file: /sdcard/fennec_ids.txt
removing file: /sdcard/robotium.config
removing file: /mnt/sdcard/tests/browser_output.txt

Comment 36

[Armen [:armenzg]]

Right now, jmaher is dealing with one of the boards to determine why comment 35 happens.

On my side, I have the patches for the SUT Agent almost to be ready to be deployed.

I have also been running unit tests and talos jobs to determine what happens after the SUT upgrade.

So far everything is running well except for two suites:
* for jsreftest-2 I keep on hitting bug 719377 (I have submitted a patch).
* for robocop unit tests I have hit twice:
> 11 INFO TEST-UNEXPECTED-FAIL | testWebContentContextMenu | Bookmark removal verified - got false, expected true

I am running three more jobs of each in hopes is transient.

Comment 37

[Armen [:armenzg]]

I noticed that I should also give Android XUL a run.

Comment 38

[Armen [:armenzg]]

Attachment: [buildbot-configs] only run robocop talos for native and enable it for pine

Comment 39

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 606585 [details] [diff] [review]
[buildbot-configs] only run robocop talos for native and enable it for pine

does PROJECT_BRANCHES == 'pine' or all project branches?

Comment 40

[Armen [:armenzg]]

Only "pine" [1]

BTW do you want this enabled today? or can it wait until Monday?
I just did a reconfig but I could do another one if wanted.


[1]
diff --git a/mozilla/project_branches.py b/mozilla/project_branches.py
--- a/mozilla/project_branches.py
+++ b/mozilla/project_branches.py
@@ -255,37 +255,39 @@ PROJECT_BRANCHES = {
     'pine': {
         'enable_unittests': False,
         'enabled_products': ['firefox'],
         'talos_suites': {
             'tp': [1, {'suites': ['--sampleConfig', 'cycles.config']}],
             'chrome': [1, {'suites': ['--sampleConfig', 'cycles.config']}],
             'chrome_mac': [1, {'suites': ['--sampleConfig', 'cycles.config']}],
             'nochrome': [1, {'suites': ['--sampleConfig', 'cycles.config']}],
             'dirty': 0,
             'svg': [1, {'suites': ['--sampleConfig', 'cycles.config']}],
             'dromaeo': 0,
             # hate that remote talos still show up when mobile product is not being requested
             # but that's not part of the bug i'm doing this for, so leaving for now
             'remote-ts': 0,
             'remote-tdhtml': 0,
             'remote-tsvg': 0,
             'remote-tsspider': 0,
             'remote-tpan': 0,
             'remote-tp4m': 0,
             'remote-tp4m_nochrome': 0,
             'remote-twinopen': 0,
             'remote-tzoom': 0,
+            'remote-trobocop': 1,
+            'remote-trobocheck': 1,

Comment 41

[Armen [:armenzg]]

Comment on attachment 606585 [details] [diff] [review]
[buildbot-configs] only run robocop talos for native and enable it for pine

http://hg.mozilla.org/build/buildbot-configs/rev/59b0ab57d21b

Landed on "default".

Comment 42

[Armen [:armenzg]]

Passing it to Joel Maher to figure out some fennec issues.
We will enable this as soon as it starts working.

I will meanwhile work on bug 736587 to have robocop enabled there and then we can enable it everywhere.

Comment 43

[Armen [:armenzg]]

jmaher, does bug 736587 really block us?
Is it so we have more memory for robocop? (since we don't allocate it for the higher resolution?)

Comment 44

[Joel Maher ( :jmaher ) (UTC -8)]

it doesn't block us.  more memory for robocop doesn't help as we run robocop right now with 1024x768.  If we left the resolution at 1600x1200 (the current reftest resolution), then it would matter a lot.

Comment 45

[Chris AtLee [:catlee]]

deployed as part of this morning's reconfig

Comment 46

[Joel Maher ( :jmaher ) (UTC -8)]

with my local testing of a patch to fix bug 737411, I am able to get robocop tests working again.

Comment 47

[Armen [:armenzg]]

I have triggered a job on staging based on this change:
https://tbpl.mozilla.org/?tree=Try&rev=b12a718cb1f1

Comment 48

[Armen [:armenzg]]

jmaher, you should be able to see the results in here:
* http://dev-master01.build.scl1.mozilla.com:8043/builders/Android%20Tegra%20250%20try%20talos%20remote-trobocheck
* http://dev-master01.build.scl1.mozilla.com:8043/builders/Android%20Tegra%20250%20try%20talos%20remote-trobocop

So far the first one fails with:
> TypeError: 'NoneType' object is not iterable
and the second is still running.

Comment 49

[Joel Maher ( :jmaher ) (UTC -8)]

it looks like trobopan worked great (although the reporting failed), but tcheckerboard is failing...getting close!

Comment 50

[Joel Maher ( :jmaher ) (UTC -8)]

s/trobocheck/tcheckerboard/

Comment 51

[Armen [:armenzg]]

Attachment: add trobopan and trobocheck to mozilla-central and mozilla-inbound

Comment 52

[Armen [:armenzg]]

Comment on attachment 609327 [details] [diff] [review]
add trobopan and trobocheck to mozilla-central and mozilla-inbound

checked-in 35addb0fc98d

This should go in with today's scheduled reconfig or I can do one tomorrow morning.

Comment 53

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #52)
> Comment on attachment 609327 [details] [diff] [review]
> add trobopan and trobocheck to mozilla-central and mozilla-inbound
> 
> checked-in 35addb0fc98d

That is a production landing which should have been gone on default, given the test masters weren't reconfiged (they're on 36ce86337cb2 right now). I've merged it to default and backed it out of production, because I need to reconfig for bug 739505.

Comment 54

[Armen [:armenzg]]

nthomas, somehow, hg merge default would not add my changes on default back to production.

I had to merge from production to default. Re-add my changes. Push back to production.

I am waiting for jhford's reconfig to happen so I can do mine.

Comment 55

[Armen [:armenzg]]

Attachment: really add robocop

r+ from jmaher in IRC.

Deploying.

Comment 56

[Armen [:armenzg]]

Attachment: add trobopan and trobocheck (for real) to m-i

Because nothing gets fixed in two places the same way *sigh*

I wish that patches this simple would not need this disparate ways to make them work.

Comment 57

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 609856 [details] [diff] [review]
add trobopan and trobocheck (for real) to m-i

Review of attachment 609856 [details] [diff] [review]:
-----------------------------------------------------------------

I could be missing the obvious...

::: mozilla/project_branches.py
@@ +109,5 @@
>          },
>          'talos_suites': {
>              'v8': 1,
> +            'remote-trobopan': 1,
> +            'remote-trobocheck': 1,

I don't understand why these are here?  Shouldn't they be with the other tests like tp4m or remote_ts?

Comment 58

[Armen [:armenzg]]

(In reply to Joel Maher (:jmaher) from comment #57)
> Comment on attachment 609856 [details] [diff] [review]
> add trobopan and trobocheck (for real) to m-i
> 
> Review of attachment 609856 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I could be missing the obvious...
> 
> ::: mozilla/project_branches.py
> @@ +109,5 @@
> >          },
> >          'talos_suites': {
> >              'v8': 1,
> > +            'remote-trobopan': 1,
> > +            'remote-trobocheck': 1,
> 
> I don't understand why these are here?  Shouldn't they be with the other
> tests like tp4m or remote_ts?

These are here because they are not enabled by default like the other ones you mention.

jmaher, trobopan is working. Should we just enable it everywhere?

I filed bug 739977 for trobocheck failures.

Comment 59

[Joel Maher ( :jmaher ) (UTC -8)]

we can't run trobopan on aurora/beta/release/esr10/1.9.2.  I am fine enabling it everwhere.

Comment 60

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: run ts_paint and tcheckerboard

Comment 61

[Armen [:armenzg]]

Comment on attachment 610142 [details] [diff] [review]
run ts_paint and tcheckerboard

Review of attachment 610142 [details] [diff] [review]:
-----------------------------------------------------------------

Sweet :)

Comment 62

[Joel Maher ( :jmaher ) (UTC -8)]

pushed tcheckerboard and ts_paint:
http://hg.mozilla.org/build/buildbot-configs/rev/40b64f4b4bb3

Comment 63

[Armen [:armenzg]]

Attachment: enable trobopan and tcheckerboard by default (not for m-a/m-b/m-r)

There is a little bit of re-ordering that I believe makes things clearer.

This patch does not enable these tests for m-a/m-b/m-r.

Comment 64

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 610189 [details] [diff] [review]
enable trobopan and tcheckerboard by default (not for m-a/m-b/m-r)

Review of attachment 610189 [details] [diff] [review]:
-----------------------------------------------------------------

looks good

Comment 65

[Armen [:armenzg]]

Comment on attachment 610189 [details] [diff] [review]
enable trobopan and tcheckerboard by default (not for m-a/m-b/m-r)

I landed it on "default" and disable it for 1.9.2 as well:
http://hg.mozilla.org/build/buildbot-configs/rev/b16777a3388d

Comment 66

[Armen [:armenzg]]

Comment on attachment 610142 [details] [diff] [review]
run ts_paint and tcheckerboard

Landed on "default":
http://hg.mozilla.org/build/buildbot-configs/rev/40b64f4b4bb3

Comment 67

[Armen [:armenzg]]

These changes went live around 11:55AM PDT.

Comment 68

[Aki Sasaki (not active)]

Backed out ts_paint in bug 740196.

Comment 69

[Armen [:armenzg]]

This bug is done! yay!
https://tbpl.mozilla.org/?jobname=Android%20Tegra%20250%20mozilla-central%20talos%20remote-trobo

bug 739974 will fix the robocop jobs show up as a "T" symbol on tbpl but it is not a blocker for having trobopan and trobocheck enabled.

Comment 70

[Aki Sasaki (not active)]

[16:47]	<philor>	hmm, https://tbpl.mozilla.org/php/getParsedLog.php?id=12365739&tree=Mozilla-Beta - "wget --progress=dot:mega -N http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-beta-android/133884418fennec_ids.txt" isn't going to go well
[16:48]	<philor>	aki|brb: ^?
[16:49]	<philor>	seems to be post-clobber, post-reconfig, and there's rather a lot of them lacking that slash
[16:52]	<philor>	oh, I should probably yell at the numbskull who blindly turned robopan on on beta without checking whether it works, shouldn't I?
[16:53]	<catlee-away>	the trains must go on!
[16:53]	<philor>	or asking one of multiple |afks who know about android talos what would be missing the trailing slash
[16:54]	<philor>	cunning armen, not even being afk
[16:54]	<catlee-away>	WTH
[16:55]	<catlee-away>	splitting based on / seems safer that fixed length string manipulation
[16:55]	<catlee-away>	hehehe
[16:56]	<catlee-away>	because we lost the a2 from 14.0a2
[16:56]	<catlee-away>	http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l7541
[16:56]	<philor>	yeah, I just got to there :)
[16:57]	<philor>	and said... WTH

Should we back out or fix?

Comment 71

[Phil Ringnalda (:philor)]

They're hidden on beta (with rather an excess of glee), so there's no real need to back out, they fail fast.

Comment 72

[Phil Ringnalda (:philor)]

Attachment: fix beta bustage

Really, who *doesn't* think a bit of my untested unidiomatic Python is exactly what we need for major, critical, right up against the releasing edge bustage?

Comment 73

[Armen [:armenzg]]

Comment on attachment 630428 [details] [diff] [review]
fix beta bustage

You are right. Thanks philor.

>>> url = 'http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-beta-android/1338958999/fennec-14.0.en-US.android-arm.apk'
>>> url[:-len('fennec-XX.0XX.en-US.android-arm.apk')] + "fennec_ids.txt"
'http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-beta-android/133895899fennec_ids.txt'
>>> url.rsplit("/", 1)[0] + "/fennec_ids.txt"
'http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-beta-android/1338958999/fennec_ids.txt'

Comment 74

[Armen [:armenzg]]

Comment on attachment 630428 [details] [diff] [review]
fix beta bustage

http://hg.mozilla.org/build/buildbotcustom/rev/92eb97dd7247

Comment 75

[Aki Sasaki (not active)]

This made it to production today.