1093939 - [weekend data] - build a list of all the talos tests and platforms and whether or not we see less noise on weekends

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

as documented in a blog post:
http://elvis314.wordpress.com/2014/10/30/a-case-of-the-weekends/

we have less data points on weekends, but also less noise in our tests.

There are many ways to figure out what is going on, but first we need to understand what our problem is before measuring.

In this bug we will have a spreadsheet (table) which lists all our platforms as columns and tests as rows.

Each cell of the table will have a yes if we have a change in our numbers or variability in them on weekends (keep in mind we have shifted timezones, so weekends could be defined as saturday midday-sunday evening)

getting a list of tests and platforms might be interesting.  Here is a link that shows all the current platforms and tests:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=talos&rev=5ccd5638a7b2

if you click on each green letter, a pane will pop up below such that a set of tests and results will show up.  You can click on the links and see graph server.  This would be one way to collect this data, click on the link for each test result, view the graph server and switch the view to 30 days.

Comment 1

[Alice Scarpa [:adusca]]

Attachment: Generate report of average test result variance per weekday

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8545386 [details] [diff] [review]
Generate report of average test result variance per weekday

Review of attachment 8545386 [details] [diff] [review]:
-----------------------------------------------------------------

I really don't like giving a r- for nice code, but this is really close.  A few small cleanup bits and we should be good!  Thanks for hacking this together and making it useful:)

::: talos/scripts/report.py
@@ +5,5 @@
> +import collections
> +import argparse
> +
> +sys.path.insert(1, os.path.join(sys.path[0], '..'))
> +from .. import compare

Is there a cleaner way to do this?  Maybe if we do the sys.path.insert we could get the abspath and find the parent dir and just insert that.  Then we could just import compare

@@ +10,5 @@
> +
> +def get_branch(platform):
> +    if platform == 'Android' or platform.startswith('OSX'):
> +        return 63
> +    return 131

please add a comment here or better yet use specific values from compare.py

@@ +74,5 @@
> +            out = csv.writer(report, quoting=csv.QUOTE_ALL)
> +            out.writerow(line)
> +
> +
> +def is_normal(y):

please add a comment here that this is sort of hacky, but gets us started!

@@ +103,5 @@
> +    test or platform - report of all the average variances for all relevant tests")
> +    args = parser.parse_args()
> +    if args.name == 'all':
> +        tests = get_all_test_tuples()
> +        generate_report(tests, mode='a')

do we have a method for mode in generate_report?  I don't see it in the argument list.

@@ +113,5 @@
> +            for test in compare.test_map:
> +                tuple_list.extend(get_tuple(test, name))
> +        elif name in compare.test_map:
> +            for platform in compare.platform_map:
> +                tuple_list.extend(get_tuple(name, platform))

I don't understand the if/else condition here.  Can you pass in -name Linux64 and it will show all results for Linux64?  Or can you pass in 'tp5o' and it will show tp5o for all platforms?  If my understanding is true, then could we just have some different args:
-platform
-test

then we could filter on both and it would be a bit clearer.

Comment 3

[Alice Scarpa [:adusca]]

Attachment: Generate report of average test result variance per weekday

Comment 4

[Geoff Brown [:gbrown]]

Comment on attachment 8545464 [details] [diff] [review]
Generate report of average test result variance per weekday

Review of attachment 8545464 [details] [diff] [review]:
-----------------------------------------------------------------

::: talos/scripts/report.py
@@ +51,5 @@
> +                runs = len(time_dict[time])
> +                days[weekday] = days.get(weekday, []) + [variance]
> +
> +            line = ["-".join(test[3:])]
> +            for day in ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']:

The calendar module provides day names; you could use those:
>>> from calendar import day_name
>>> print day_name[0]
Monday

Comment 5

[Alice Scarpa [:adusca]]

Attachment: Generate report of average test result variance per weekday

I updated again to follow Geoff's suggestion.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8545989 [details] [diff] [review]
Generate report of average test result variance per weekday

Review of attachment 8545989 [details] [diff] [review]:
-----------------------------------------------------------------

thank you for this.  I have a couple questions and a couple of nits, please address them and upload a final patch for landing.  r+ as this is pretty much done.

::: talos/scripts/report.py
@@ +48,5 @@
> +
> +            for time in time_dict:
> +                weekday = datetime.strptime(time, '%Y-%m-%d').strftime('%A')
> +                variance = numpy.var(time_dict[time])
> +                runs = len(time_dict[time])

what do we use runs for?

@@ +53,5 @@
> +                days[weekday] = days.get(weekday, []) + [variance]
> +
> +            line = ["-".join(test[3:])]
> +            for day in day_name:
> +                average = numpy.average(sorted(days[day])[5:45])

why [5:45]?

@@ +106,5 @@
> +    f = 'report'
> +    if args.platform:
> +        tuple_list = filter(lambda x: x[4] == args.platform, tuple_list)
> +        f += '-%s' % args.platform
> + 

nit: blank line with whitespace, make it a blank line

@@ +110,5 @@
> + 
> +    if args.test:
> +        tuple_list = filter(lambda x: x[3] == args.test, tuple_list)
> +        f += '-%s' % args.test
> +   

nit: blank line with whitespace, make it a blank line

Comment 7

[Alice Scarpa [:adusca]]

Attachment: Generate report of average test result variance per weekday

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/build/talos/rev/8ed7a2f27b3d