Closed Bug 1375039 Opened 7 years ago Closed 7 years ago

determine if AWSY on mozilla-beta is running properly

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(firefox56 verified, firefox57 verified)

VERIFIED FIXED
mozilla57
Tracking Status
firefox56 --- verified
firefox57 --- verified

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

(Whiteboard: [PI:August])

Attachments

(2 files)

I noticed on a AWSY regression on mozilla beta that we are:
* running on a taskcluster build (we don't ship this)
* running on opt (typically perf is done on pgo)
* not run on pgo

are these valid things to test with?
Whiteboard: [PI:June]
That's what we do with Nightly as well, right? The original work there was either directly uplifted to Aurora/Beta or is just riding the trains. What is the risk of a difference between the taskcluster build and a buildbot build showing significant differences that would justify the work of dealing with buildbot?

I'll leave the pgo decision to you. Talos does pgo builds only? I would think both would be a requirement as pgo might hide stuff?
I think using the TC builds is probably safe- it has given us reliable results on trunk.

ss for PGO, we have historically only run talos on pgo for mozilla-beta- I believe we do PGO for every commit in beta.

There is probably no danger is running on opt and pgo, but that is duplicative IMO.  :erahm, do you have thoughts on how we should be running AWSY on mozilla-beta?
Flags: needinfo?(erahm)
On beta running on PGO seems reasonable, that's effectively what we ship right? I don't think we need to do both as long as every beta commit has a PGO build. On the other hand if it's an undue burden I'm fine with just sticking with plain old opt builds.
Flags: needinfo?(erahm)
ok, we have PGO for every commit- lets start with that.

I have this on my list for next week!
Whiteboard: [PI:June] → [PI:July]
ok, back into looking at this- we were running win7/10 "opt", and now we are running osx opt, win7 opt/pgo/devedition/nightly, win10 opt/pgo/nightly.  Note, there is no linux AWSY running.

I would propose only running on one configuration per platform- so we should see one run for linux, osx, win7, win10.  

When bug 1386776 is resolved, I will circle back here and try to get this cleaned up.
Depends on: 1386776
Whiteboard: [PI:July] → [PI:August]
osx/linux there is no awsy on mozilla-beta, but win7/10 for nightly and for devedition- we should fix this problem this week.
when bug 1387377 is resolved and uplifted to beta, lets check on what is running.

in trying to figure out why we are not running on osx and linux64, I am at a loss.  I looked in the config and there is nothing that restricts us from running on beta:
http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#8

and why on mozilla-beta are we running win7/win10?

:dustin, could you help me figure out why awsy is not running on mozilla-beta for linux and osx?
Flags: needinfo?(dustin)
The tasks are in the full-task-graph and not in the target tasks, so I'd look at the target task method.

        if platform in (
                'linux', 'linux64',
                'macosx64',
                'win32', 'win64',
                ):  
            if task.attributes['build_type'] == 'opt' and \
               task.attributes.get('unittest_suite') != 'talos':
                return False

looks like we don't run opt on beta?
Flags: needinfo?(dustin)
mozilla-beta is confusion, we do run talos on linux64 opt (which is really a pgo build), and osx (which we do not have pgo builds for).

My knowledge is based on what I see on treeherder:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=02852b5a5b3c6b13029da12b9da5060f88ee2222&filter-searchStr=awsy

on windows 7, this is run on nightly opt- but we have these nightly opt builds for linux32/64 as well.

In summary I see similar builds for windows and linux (and mostly osx)- I see talos running on all platforms correctly, and the 'gtest' job runs correctly on nightly and debug builds; but the awsy job doesn't seem to run except for on windows 7/10.

:dustin, do you have other thoughts on how to debug this?
Flags: needinfo?(dustin)
Assignee: nobody → jmaher
I'm sorry, I don't really understand what you're asking.
Flags: needinfo?(dustin)
Basically I cannot figure out why these tests are not running on mozilla-beta and I don't understand taskcluster well enough to debug it.  If you are not the right person to help out, could you help me find someone else on the team to help figure this out?
Flags: needinfo?(dustin)
I think I answered it in comment 8.  Looking at https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml?q=file%3Atest-platforms.yml&redirect_type=single I only see awsy set to run on */opt platforms.  And in the mozilla-beta target task function, unittests (of which awsy is one) on opt platforms appear to be filtered out.

I don't know why either of those things are the case, but those are the assumptions to question.  I'm not sure who to ask, either -- I don't know who is in charge of AWSY.  I suppose someone from releng may know the background on the beta config.  And there's always 'hg blame'.
Flags: needinfo?(dustin)
this will land on nightly, then we need to uplift to beta.
Attachment #8902494 - Flags: review?(bob)
Comment on attachment 8902494 [details] [diff] [review]
run awsy tests on linux64/osx nightly test-sets

looks ok. let's see how it does.
Attachment #8902494 - Flags: review?(bob) → review+
Attached file diff-tasks.txt
diff of before and after output of ./mach taskgraph tasks --json
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/bcf177055b33
Enable AWSY for linux64/osx nightly builds. r=bc
:ryanvm, what is needed to uplift this to mozilla-beta? do I need to write a beta specific patch?
Flags: needinfo?(ryanvm)
https://hg.mozilla.org/mozilla-central/rev/bcf177055b33
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Flags: needinfo?(ryanvm)
thanks everyone!
You need to log in before you can comment on or make changes to this bug.