622311 - Socorro - make new submitter for load testing

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[K Lars Lohn [:lars] [:klohn]]

Created a more sophisticated submitter.py.  This one is in scripts and will eventually replace the one in .../socorro/collector.  For now, I'm leaving
that one in place.

  New options:
        -s, --delay
                pause between submission queing in milliseconds (default: 1)
        -D, --dryrun
                don't actually submit, just print product/version
        -N, --numberOfSubmissions
                the number items to submit (all, forever, 1...) (default: all)
        -n, --numberOfThreads
                the number of threads (default: 4)

the --numberOfSubmissions option allows the submitter to get a specific number of crashes from the source (specified by --search or -s), or loop through a search over and over.  The default behavior is to just submit all the crashes found in a search (this mimics the behavior of the old submitter).

Also includes some rudimentary statistics.  After one minute, it starts to print the average number of submissions attempted per minute over the last
five minutes.  

This app is BETA.  There are known defects:
  * the --numberOfSubmissions option: all and forever work fine, integers do not.
  * khan seems to have a defective pycurl module for python26.  Actual
    submissions are not tested.
  * stats don't settle down until after five minutes of running.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

examples of use:

python26 .../scripts/submitter.py -N forever -s ~lars/breakpad17/testData/500

this will loop through the 500 crashes in my testData directory over and over forever.  It will submit as fast as it can.  It will only stop with ^C, SIGTERM or SIGKILL

python26 .../scripts/submitter.py --delay=200 -N all -s ~lars/breakpad17/testData/1013

this will loop through the 1013 crashes in one of my testData directories.  It will submit only 5 per second.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

enhancements and corrections are now checked in.
  * minor modification to the command line parameters
  * all features of --numberOfSubmissions now work
  * integrated into the syslog and config system - submitterconfig.py.dist
  * working components moved into the .../socorro/collector/submitter module
  * original submitter still intact

** pycurl for python26 is still broken on khan.  It works on my workstation at home.

we'll modify this app on the fly as our actual needs become more known.

(py26)[lars@khan scripts]$ python submitter.py --help
submitter 2.0
	    --delay
		pause between submission queing in milliseconds (default: 0)
	-D, --dryrun
		don't actually submit, just print product/version 
                (default: False)
	-d, --dumpfile
		the pathname of a dumpfile to submit (default: None)
	-?, --help
		print this list
	-j, --jsonfile
		the pathname of a json file to submit (default: None)
	-n, --numberOfSubmissions
		the number of crashes to submit (all, forever, 1...) 
                (default: all)
	    --numberOfThreads
		the number of threads to use (default: 4)
	-s, --searchRoot
		a filesystem location to begin a search for json/dump pairs 
                (default: None)
	    --stderrErrorLoggingLevel
		logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 
                30 - WARNING, 40 - ERROR, 50 - CRITICAL) (default: 10)
	    --stderrLineFormatString
		python logging system format for logging to stderr (default:
                %(asctime)s %(levelname)s - %(threadName)s - %(message)s)
	    --syslogErrorLoggingLevel
		logging level for the log file (10 - DEBUG, 20 - INFO, 
                30 - WARNING, 40 - ERROR, 50 - CRITICAL) (default: 10)
	    --syslogFacilityString
		syslog facility string ("user", "local0", etc) (default: user)
	    --syslogHost
		syslog hostname (default: localhost)
	    --syslogLineFormatString
		python logging system format for syslog entries (default: 
                Socorro Submitter (pid %(process)d): %(asctime)s %(levelname)s
                - %(threadName)s - %(message)s)
	    --syslogPort
		syslog port (default: 514)
	    --uniqueHangId
		cache and uniquify hangids (default: True)
	-u, --url
		The url of the server to load test (default: 
                https://crash-reports.stage.mozilla.com/submit)