Closed Bug 733583 Opened 12 years ago Closed 9 years ago

run_tests.py should wipe the /mnt/sdcard/tests directory on android

Categories

(Testing :: Talos, defect)

Product:
Testing
Component:
Talos
Platform:
All
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: k0scist, Unassigned)

References

Details

I have seen this before so this is nothing new.  We *probably* avoid this in production since IIRC we flash the device for each run.

In any case, I make a ts testing file like::

python remotePerfConfigurator.py -v -e org.mozilla.fennec --develop --activeTests ts --output ts.yml --results_url file://${HOME}/ts.txt --noChrome --remoteDevice 10.251.28.71

The resulting file::

# Sample Talos configuration file

# The title of the report
title: qm-pxp01

test_name_extension: _nochrome

#*** output options ****
#uncomment to turn on dump to csv option
#csv_dir: 'output'
#comment out next line to disable send to graph server
results_url: file:///home/jhammel/ts.txt
develop: True
fennecIDs: 

# data filters
# filters: [median]

# browser info
process: fennec
browser_path: org.mozilla.fennec
browser_log: browser_output.txt
# arguments to pass to browser
extra_args: ''
#how long the browser takes to open/close
browser_wait: 5
test_timeout: 1200
bcontroller_config: ${talos}/bcontroller.yml

branch: mobile

remote: True
deviceip: 10.251.28.71
deviceport: 20701
webserver: 10.251.27.101:15707
deviceroot: /mnt/sdcard/tests

buildid: 2012030603

init_url: http://10.251.27.101:15707/getInfo.html

# Preferences to set in the test (use "preferences : {}" for no prefs)
preferences:
  extensions.enabledScopes: 5
  extensions.autoDisableScopes: 10
  browser.EULA.override : true
  security.fileuri.strict_origin_policy : false
  browser.shell.checkDefaultBrowser : false
  browser.warnOnQuit : false
  browser.link.open_newwindow : 2
  dom.allow_scripts_to_close_windows : true
  dom.disable_open_during_load: false
  dom.max_script_run_time : 0
  hangmonitor.timeout : 0
  browser.dom.window.dump.enabled: true
  dom.disable_window_flip : true
  dom.disable_window_move_resize : true
  security.enable_java : false
  extensions.checkCompatibility : false
  extensions.update.notifyUser: false
  talos.logfile: /mnt/sdcard/tests/browser_output.txt
  browser.cache.disk.smart_size.enabled: false
  browser.cache.disk.smart_size.firstrun: false
  network.manage-offline-status: false

extensions:
 - ${talos}/pageloader
extensions:
 - ${talos}/pageloader

#any directories whose contents need to be installed in the browser before running the tests
# this assumes that the directories themselves already exist in the firefox path
dirs:
  chrome : ${talos}/page_load_test/chrome
  components : ${talos}/page_load_test/components

# Environment variables to set during test (use env: {} for none)
env :
  NO_EM_RESTART : 1
  MOZ_CRASHREPORTER_NO_REPORT : 1
  MOZ_CRASHREPORTER_SHUTDOWN : 1
# Tests to run
#  url         : (REQUIRED) url to load into the given firefox browser
#  url_mod     : (OPTIONAL) a bit of code to be evaled and added to the given url during each cycle of the test
#  resolution  : (REQUIRED) how long (in seconds) to pause between counter sampling
#  cycles      : (REQUIRED) how many times to run the test
#  timeout     : (OPTIONAL) how many seconds the test can run before we consider it failed and quit (default 8 hours)
#  pagetimeout : (OPTIONAL) how many seconds each page is allowed to take before considered to be frozen (default 8 hours)
#            Must be used in conjuction with the pageloader with the -tpnoisy option
#  counters    : (REQUIRED) types of system activity to monitor during test run, can be empty
#            For possible values of counters argument on Windows, see
#            http://technet2.microsoft.com/WindowsServer/en/Library/86b5d116-6fb3-427b-af8c-9077162125fe1033.mspx?mfr=true
#            Possible values on Mac:
#            counters : ['Private Bytes', 'RSS']
#            Possible values on Linux:
#            counters : ['Private Bytes', 'RSS', 'XRes']
#            Standard windows values:
#            counters : ['Working Set', 'Private Bytes', '% Processor Time']

# to set up a new test it must have the correct configuration options and drop information in a standard format
# the format is seen in the regular expressions in ttest.py
# to see how the data passed from the browser is processed see send_to_graph and send_to_csv in run_tests.py

basetest:
  url :
  url_mod :
  resolution : 1
  cycles : 1
  timeout : 3600
  win_counters : []
  w7_counters : []
  linux_counters : []
  mac_counters : []
  remote_counters : []
  shutdown : False
  responsiveness : False
  profile_path : ${talos}/mobile_profile
# pageloader options
  tpmanifest: 
  tpchrome: True
  tpmozafterpaint: False
  tpnoisy: True
  tpformat: tinderbox
  tpcycles: 3
  tppagecycles: 1
  rss: False
  tprender: False
  tpdelay: 1000

tests :
- name: ts
  tpchrome: False
  url : http://10.251.27.101:15707/startup_test/startup_test.html?begin=
  url_mod : str(int(time.time()*1000))
  cycles : 10
  timeout: 300
  shutdown : True

Then I run it like::

python run_tests.py -n -d ts.yml

However, I get output like tsvg, which was the previous test I ran::

<snip/>
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-001.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml)
NOISE: RSS: Main: 83058688
NOISE: 
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml)
NOISE: RSS: Main: 81244160
NOISE: 
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml)
NOISE: RSS: Main: 81510400
NOISE: 
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml)
NOISE: RSS: Main: 79773696
NOISE: 
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml)
NOISE: RSS: Main: 80232448
NOISE: 
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml)
NOISE: RSS: Main: 77053952
NOISE: 
NOISE: Cycle 2(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml (next: http://10.251.27.101:15707/page_load_test/svg/gearflowers.svg)
NOISE: RSS: Main: 78024704
NOISE: 
NOISE: RSS: Main: 78012416
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/gearflowers.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale.svg)
NOISE: RSS: Main: 78225408
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale-opacity.svg)
NOISE: RSS: Main: 80527360
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale-opacity.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate.svg)
NOISE: RSS: Main: 78508032
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate.svg (next: http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate-opacity.svg)
NOISE: RSS: Main: 78635008
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/composite-scale-rotate-opacity.svg (next: http://10.251.27.101:15707/page_load_test/svg/hixie-001.xml)
NOISE: RSS: Main: 78962688
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-001.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml)
NOISE: RSS: Main: 79032320
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml)
NOISE: RSS: Main: 73363456
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-003.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml)
NOISE: RSS: Main: 79794176
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-004.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml)
NOISE: RSS: Main: 76701696
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-005.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml)
NOISE: RSS: Main: 77381632
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-006.xml (next: http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml)
NOISE: RSS: Main: 77598720
NOISE: 
NOISE: Cycle 3(1): loaded http://10.251.27.101:15707/page_load_test/svg/hixie-007.xml (next: http://10.251.27.101:15707/page_load_test/svg/gearflowers.svg)
NOISE: RSS: Main: 79126528
NOISE: 
NOISE: __start_tp_report
NOISE: _x_x_mozilla_page_load,3444.9545454545455,NaN,NaN
NOISE: _x_x_mozilla_page_load_details,avgmedian|3444.9545454545455|average|3444.95|minimum|NaN|maximum|NaN|stddev|NaN
NOISE: |i|pagename|median|mean|min|max|runs|
NOISE: |0;gearflowers.svg;664;664;541;853;787;853;541
NOISE: |1;composite-scale.svg;206;206;201;283;211;283;201
NOISE: |2;composite-scale-opacity.svg;49.5;49.5;45;58;58;54;45
NOISE: |3;composite-scale-rotate.svg;40;40;38;49;49;38;42
NOISE: |4;composite-scale-rotate-opacity.svg;50;50;50;81;50;81;50
NOISE: |5;hixie-001.xml;15137.5;15137.5;15122;15182;15182;15153;15122
NOISE: |6;hixie-002.xml;15123.5;15123.5;15113;15166;15134;15113;15166
NOISE: |7;hixie-003.xml;5061.5;5061.5;5055;5083;5055;5083;5068
NOISE: |8;hixie-004.xml;5159;5159;5158;5167;5160;5158;5167
NOISE: |9;hixie-005.xml;5124.5;5124.5;5124;5145;5125;5145;5124
NOISE: |10;hixie-006.xml;5109.5;5109.5;5097;5139;5097;5139;5122
NOISE: |11;hixie-007.xml;1307;1307;1281;1368;1333;1368;1281
NOISE: __end_tp_report
NOISE: __start_cc_report
NOISE: _x_x_mozilla_cycle_collect,1493
NOISE: __end_cc_report
NOISE: __startTimestamp1331072013846__endTimestamp
NOISE: 
NOISE: __metrics	Screen width/height:0/0
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 980/800
NOISE: __metrics__start_report2643__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072590349 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072588244" line: 0}]
NOISE: __startTimestamp1331072591155__endTimestamp
NOISE: __start_report2652__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072636105 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072633893" line: 0}]
NOISE: __startTimestamp1331072636774__endTimestamp
NOISE: __start_report3304__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072682424 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072679556" line: 0}]
NOISE: __startTimestamp1331072682943__endTimestamp
NOISE: __start_report2755__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072727961 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072725674" line: 0}]
NOISE: __startTimestamp1331072728626__endTimestamp
NOISE: __start_report2902__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072781122 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072778674" line: 0}]
NOISE: __startTimestamp1331072781701__endTimestamp
NOISE: __start_report2749__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072835213 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072833082" line: 0}]
NOISE: __startTimestamp1331072835884__endTimestamp
NOISE: __start_report2951__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072880816 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072878647" line: 0}]
NOISE: __startTimestamp1331072881640__endTimestamp
NOISE: __start_report2954__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072927455 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072925258" line: 0}]
NOISE: __startTimestamp1331072928419__endTimestamp
NOISE: __start_report3094__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331072974147 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331072971733" line: 0}]
NOISE: __startTimestamp1331072974860__endTimestamp
NOISE: __start_report2719__end_report
NOISE: 
NOISE: Could not read chrome manifest 'file:///data/data/org.mozilla.fennec/chrome.manifest'.
NOISE: zerdatime 1331073019954 - browser chrome startup finished.
NOISE: [JavaScript Warning: "Use of enablePrivilege is deprecated.  Please use code that runs with the system principal (e.g. an extension) instead." {file: "http://10.251.27.101:15707/startup_test/startup_test.html?begin=1331073017687" line: 0}]
NOISE: __startTimestamp1331073020499__endTimestamp
getting files in '/mnt/sdcard/tests/profile/minidumps/'
DEBUG: Received test results: 2215 2215 2215 2215 2215 2215 2215 2215 2215 2215
Completed test ts: 
		Stopped Tue, 06 Mar 2012 14:33:24
RETURN: cycle time: 00:08:44<br>
qm-pxp01: 
		Stopped Tue, 06 Mar 2012 14:33:24
Sending results: 
		Started Tue, 06 Mar 2012 14:33:24
DEBUG: Working with test: ts
DEBUG: Sending results: 2215 2215 2215 2215 2215 2215 2215 2215 2215 2215
Generating results file: ts: 
		Started Tue, 06 Mar 2012 14:33:24
Generating results file: ts: 
		Stopped Tue, 06 Mar 2012 14:33:24
Generating results file: ts_shutdown: 
		Started Tue, 06 Mar 2012 14:33:24
Generating results file: ts_shutdown: 
		Stopped Tue, 06 Mar 2012 14:33:24
Transmitting test: ts: 
		Started Tue, 06 Mar 2012 14:33:24
Transmitting test: ts: 
		Stopped Tue, 06 Mar 2012 14:33:24
Transmitting test: ts_shutdown: 
		Started Tue, 06 Mar 2012 14:33:24
Transmitting test: ts_shutdown: 
		Stopped Tue, 06 Mar 2012 14:33:24
RETURN:<br>
RETURN:<p style="font-size:smaller;">Details:<br>|</p>
Completed sending results: 
		Stopped Tue, 06 Mar 2012 14:33:24

There is some test_startup.html in there but there is also some tsvg pages like http://10.251.27.101:15707/page_load_test/svg/hixie-002.xml .

I don't know what's going on here, but needless to say it worries me.
Running it twice seems to "clear" whatever is happening that has leftover tsvg stuff
Beh, I lied, it does not
Neither does resetting the device
looking in /mnt/sdcard/tests svg.manifest does linger after the tsvg run.  I haven't tracked how it is being picked up by talos yet.  Maybe the thing to do is nuke it
clearing out /mnt/sdcard/{svg.manifest,browser.log} seems to fix this problem.  Maybe we should clear out this directory before testing as well as afterwards?
actually you only need to clear out browser.log and all of this should live in the /mnt/sdcard/tests folder.  Anyway, this should take place in run_tests.py or something like that.
Summary: weird residuals on android talos testing? → run_tests.py should wipe the /mnt/sdcard/tests directory on android
Depends on: 735500
Depends on: 735502
we  can probably fix this bug now without much trouble.
moving the remaining android talos tests to autophone this quarter, autophone is more robust in device management and retrying, most likely we will not see this issue there.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.