[weekend data] - build a list of all the talos tests and platforms and whether or not we see less noise on weekends

RESOLVED FIXED

Status

Testing
Talos
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jmaher, Assigned: adusca, Mentored)

Tracking

(Blocks: 1 bug)

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [good next bug][project_talos_weekends])

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

3 years ago
as documented in a blog post:
http://elvis314.wordpress.com/2014/10/30/a-case-of-the-weekends/

we have less data points on weekends, but also less noise in our tests.

There are many ways to figure out what is going on, but first we need to understand what our problem is before measuring.

In this bug we will have a spreadsheet (table) which lists all our platforms as columns and tests as rows.

Each cell of the table will have a yes if we have a change in our numbers or variability in them on weekends (keep in mind we have shifted timezones, so weekends could be defined as saturday midday-sunday evening)

getting a list of tests and platforms might be interesting.  Here is a link that shows all the current platforms and tests:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=talos&rev=5ccd5638a7b2

if you click on each green letter, a pane will pop up below such that a set of tests and results will show up.  You can click on the links and see graph server.  This would be one way to collect this data, click on the link for each test result, view the graph server and switch the view to 30 days.
(Assignee)

Comment 1

3 years ago
Created attachment 8545386 [details] [diff] [review]
Generate report of average test result variance per weekday
Attachment #8545386 - Flags: review?(jmaher)
(Reporter)

Comment 2

3 years ago
Comment on attachment 8545386 [details] [diff] [review]
Generate report of average test result variance per weekday

Review of attachment 8545386 [details] [diff] [review]:
-----------------------------------------------------------------

I really don't like giving a r- for nice code, but this is really close.  A few small cleanup bits and we should be good!  Thanks for hacking this together and making it useful:)

::: talos/scripts/report.py
@@ +5,5 @@
> +import collections
> +import argparse
> +
> +sys.path.insert(1, os.path.join(sys.path[0], '..'))
> +from .. import compare

Is there a cleaner way to do this?  Maybe if we do the sys.path.insert we could get the abspath and find the parent dir and just insert that.  Then we could just import compare

@@ +10,5 @@
> +
> +def get_branch(platform):
> +    if platform == 'Android' or platform.startswith('OSX'):
> +        return 63
> +    return 131

please add a comment here or better yet use specific values from compare.py

@@ +74,5 @@
> +            out = csv.writer(report, quoting=csv.QUOTE_ALL)
> +            out.writerow(line)
> +
> +
> +def is_normal(y):

please add a comment here that this is sort of hacky, but gets us started!

@@ +103,5 @@
> +    test or platform - report of all the average variances for all relevant tests")
> +    args = parser.parse_args()
> +    if args.name == 'all':
> +        tests = get_all_test_tuples()
> +        generate_report(tests, mode='a')

do we have a method for mode in generate_report?  I don't see it in the argument list.

@@ +113,5 @@
> +            for test in compare.test_map:
> +                tuple_list.extend(get_tuple(test, name))
> +        elif name in compare.test_map:
> +            for platform in compare.platform_map:
> +                tuple_list.extend(get_tuple(name, platform))

I don't understand the if/else condition here.  Can you pass in -name Linux64 and it will show all results for Linux64?  Or can you pass in 'tp5o' and it will show tp5o for all platforms?  If my understanding is true, then could we just have some different args:
-platform
-test

then we could filter on both and it would be a bit clearer.
Attachment #8545386 - Flags: review?(jmaher) → review-
(Assignee)

Comment 3

3 years ago
Created attachment 8545464 [details] [diff] [review]
Generate report of average test result variance per weekday
Attachment #8545386 - Attachment is obsolete: true
Attachment #8545464 - Flags: review?(jmaher)
Comment on attachment 8545464 [details] [diff] [review]
Generate report of average test result variance per weekday

Review of attachment 8545464 [details] [diff] [review]:
-----------------------------------------------------------------

::: talos/scripts/report.py
@@ +51,5 @@
> +                runs = len(time_dict[time])
> +                days[weekday] = days.get(weekday, []) + [variance]
> +
> +            line = ["-".join(test[3:])]
> +            for day in ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']:

The calendar module provides day names; you could use those:
>>> from calendar import day_name
>>> print day_name[0]
Monday
(Assignee)

Comment 5

3 years ago
Created attachment 8545989 [details] [diff] [review]
Generate report of average test result variance per weekday

I updated again to follow Geoff's suggestion.
Attachment #8545464 - Attachment is obsolete: true
Attachment #8545464 - Flags: review?(jmaher)
Attachment #8545989 - Flags: review?(jmaher)
(Reporter)

Comment 6

3 years ago
Comment on attachment 8545989 [details] [diff] [review]
Generate report of average test result variance per weekday

Review of attachment 8545989 [details] [diff] [review]:
-----------------------------------------------------------------

thank you for this.  I have a couple questions and a couple of nits, please address them and upload a final patch for landing.  r+ as this is pretty much done.

::: talos/scripts/report.py
@@ +48,5 @@
> +
> +            for time in time_dict:
> +                weekday = datetime.strptime(time, '%Y-%m-%d').strftime('%A')
> +                variance = numpy.var(time_dict[time])
> +                runs = len(time_dict[time])

what do we use runs for?

@@ +53,5 @@
> +                days[weekday] = days.get(weekday, []) + [variance]
> +
> +            line = ["-".join(test[3:])]
> +            for day in day_name:
> +                average = numpy.average(sorted(days[day])[5:45])

why [5:45]?

@@ +106,5 @@
> +    f = 'report'
> +    if args.platform:
> +        tuple_list = filter(lambda x: x[4] == args.platform, tuple_list)
> +        f += '-%s' % args.platform
> + 

nit: blank line with whitespace, make it a blank line

@@ +110,5 @@
> + 
> +    if args.test:
> +        tuple_list = filter(lambda x: x[3] == args.test, tuple_list)
> +        f += '-%s' % args.test
> +   

nit: blank line with whitespace, make it a blank line
Attachment #8545989 - Flags: review?(jmaher) → review+
(Reporter)

Updated

3 years ago
Assignee: nobody → alicescarpa
(Assignee)

Comment 7

3 years ago
Created attachment 8546085 [details] [diff] [review]
Generate report of average test result variance per weekday
Attachment #8545989 - Attachment is obsolete: true
(Reporter)

Comment 8

3 years ago
https://hg.mozilla.org/build/talos/rev/8ed7a2f27b3d
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.