674350 - Start sending traffic to the add-on perf system

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P2)

Description

[Wil Clouser [:clouserw]]

Alice is ready for traffic to head her way.  You wrote some methods in bug 660331, we should hook those up to send a payload to Alice:

- Through a button in the dev tools would be cool but not required.  Would need to have some rate limiting

- On add-on upload/approval.  I'm not sure what is ideal there, I kind of like the idea of approval to cut down on potential automated compromises.  CCing jorgev if he has opinions.

All of this should be behind a waffle flag, as usual.

Comment 1

[Jorge Villalobos [:jorgev] (he/him)]

(In reply to comment #0)
> - On add-on upload/approval.  I'm not sure what is ideal there, I kind of
> like the idea of approval to cut down on potential automated compromises. 
> CCing jorgev if he has opinions.

I assume you're asking what level of approval the tested add-ons should have. Any file that has received at least Preliminary Approval should be OK to test. Unreviewed add-ons should never be tested like this.

Comment 2

[Wil Clouser [:clouserw]]

Great, let's trigger it for add-ons when they are approved to prelim or full approval.

Comment 3

[alice nodelman [:alice] [:anode]]

Can I get some controlled pushes of test requests into the system so that I can confirm that staging work correctly?  We're going to want to work in concert here so that I can shake out bugs on my end.

Comment 4

[Wil Clouser [:clouserw]]

I think Kumar's last patch let us run these off the CLI - kumar, can you confirm that command?

If that's the case, Alice, what data do you want me to send your way?

Comment 5

[alice nodelman [:alice] [:anode]]

I'd like to see a test for a linux, macosx and win32 addon.  It would also be good to send through some specifically broken requests to ensure that they are handled correctly.

Comment 6

[Kumar McMillan [:kumar]]

(In reply to comment #4)
> I think Kumar's last patch let us run these off the CLI - kumar, can you
> confirm that command?

./manage.py perftest --os={os} --firefox={firefox} --file-id={addon-file-id}

But the output won't be very useful. I'll fix it up when I start on this for 6.1.9

Comment 7

[alice nodelman [:alice] [:anode]]

Wil - with that information would you be able to push through some test runs?

Comment 8

[Wil Clouser [:clouserw]]

Alice and I played with this a little on IRC.  We had great success until her machine disappeared and I haven't talked to her since.  Alice, do you know what happened there?  Should we hook up our preview server to start sending data to you regularly?

Comment 9

[alice nodelman [:alice] [:anode]]

I filed for the timouts we were seeing in bug 679465.  It appears to be a known infrastructure issue that IT is already investigating - so it wasn't connected to our testing or code.

I will be doing some final work this week to ensure that graph server recognizes the new addon perf testing machines and deals with them correctly.

Once that it complete, I would like to get AMO sending data regularly, but it will have to be with the understanding that we will continue to experience timeouts so the system will be *ahem* less than reliable.

Comment 10

[alice nodelman [:alice] [:anode]]

Graph server updates are complete.

I've hooked the trigger script up to the production master.

We are now ready for a Beta launch of the system.

Comment 11

[Wil Clouser [:clouserw]]

Let's put this behind a waffle flag on AMO (defaulting to off) so we can turn it off if it starts breaking.

Also, as with all our remote calls, have a reasonable timeout on it.

Comment 12

[Kumar McMillan [:kumar]]

Alrighty, we can now start performance tests from the version management page on the AMO devhub, e.g. https://addons.allizom.org/en-US/developers/addon/firebug-kumar-test/versions/

Note that this just hooks up the sender. We don't do anything with the response except log it right now.

Alice, does it look like we're sending the right request?  Each one of the requests ended in a line like this:
ERROR: buildUrl construction:  urllib2.HTTPError HTTP Error 404: Not Found 4.0 linux#012

I'm attaching a full log.  The lines prefixed with 'DEBUG PERF TEST line' are output received from trigger.cgi.  Here is an example of a request we are making from the dev site: http://talos-addon-master1.amotest.scl1.mozilla.com/trigger/trigger.cgi?url=https%3A%2F%2Faddons.allizom.org%2Fdownloads%2Ffile%2F128527%2Ffirebug_kumar_test-1.8.1-Kumar-fx.xpi%3Fsrc%3Dperftest&firefox=firefox4.0&os=macosx64

I'm going to close this bug. We can open new bugs if we need to adjust the way AMO makes trigger requests and also new bugs for displaying results.

Comment 13

[Kumar McMillan [:kumar]]

Attachment: log of response after triggering perf tests

Revision to introduce the devhub trigger button https://github.com/jbalogh/zamboni/commit/f436699db2c693905c9851db34a4bed1de1007f0

Comment 14

[alice nodelman [:alice] [:anode]]

Looking at where we pull builds from:

http://releases.mozilla.org/pub/mozilla.org/firefox/releases/

There is currently only latest-3.6 and latest-6.0 in there - so those are the only versions of firefox that can currently be sent through the system, explaining the error that you seeing.

Comment 15

[Kumar McMillan [:kumar]]

Attachment: log of response after triggering perf tests

Comment 16

[Kumar McMillan [:kumar]]

(In reply to alice nodelman [:alice] [:anode] from comment #14)
> There is currently only latest-3.6 and latest-6.0 in there - so those are

Fixed: https://github.com/jbalogh/zamboni/commit/828d32b47317e9a8d21ce5ffaccf1c6ba2a71ab7

(I'll kick off another set of tests once I figure out unrelated permission issues on the staging server)

Comment 17

[Kumar McMillan [:kumar]]

re-opening so I don't lose track of this

Comment 18

[Kumar McMillan [:kumar]]

(In reply to Kumar McMillan [:kumar] from comment #16)
> (In reply to alice nodelman [:alice] [:anode] from comment #14)
> > There is currently only latest-3.6 and latest-6.0 in there - so those are
> 
> Fixed:
> https://github.com/jbalogh/zamboni/commit/
> 828d32b47317e9a8d21ce5ffaccf1c6ba2a71ab7

I got access to the logs.  So, after this change every request to trigger.cgi now times out.  Our timeout is set to one second like all other services.  Is that too low?  Here is an example request: http://talos-addon-master1.amotest.scl1.mozilla.com/trigger/trigger.cgi?url=https%3A%2F%2Faddons-dev.allizom.org%2Fdownloads%2Ffile%2F111865%2Fstatus_watch-0.2.rev11-tb%2Bfx%2Bfn.xpi%3Fsrc%3Dperftest&firefox=firefox4.0&os=win64

Comment 19

[Kumar McMillan [:kumar]]

Oh, btw, the requests that were timing out were getting sent from the phx data center (which we recently moved to).  Does that need to be white listed?

Comment 20

[alice nodelman [:alice] [:anode]]

Since the link is up and accessible for me, then yes I would assume that you are outside the whitelist.  You should file a bug with IT.  They maintain a mana page detailing the network flow for the addon master vm:

https://mana.mozilla.org/wiki/display/websites/talos-addon-master1.amotest.scl1.mozilla.com#

Comment 21

[Kumar McMillan [:kumar]]

Hi. AMO in PHX got whitelisted and we are able to connect to Talos and start the performance tests.  Yay.

Alice, I'm still seeing some errors in the service response from a test run I did on 2011-09-07.  Is there anything that we need to change on the AMO side to correct this?  I'll cc the log.

Comment 22

[Kumar McMillan [:kumar]]

Attachment: log #2 of response after triggering perf tests

Lines prefixed with 'DEBUG PERF TEST line' were what the server responded with.

I see some URLError timeouts (in AMO) and also 'ERROR: pingUrl, server error: 404' in Talos.

Comment 23

[alice nodelman [:alice] [:anode]]

I pushed through an example link from your log:

http://talos-addon-master1.amotest.scl1.mozilla.com/trigger/trigger.cgi?url=https%3A%2F%2Faddons-dev.allizom.org%2Fdownloads%2Ffile%2F129535%2Ffirebug_kumar_test-1.8.1-Kumar-fx.xpi%3Fsrc%3Dperftest&firefox=firefox6.0&os=win64

And it completed with:

"SENDCHANGE: change sent successfully"

Comment 24

[alice nodelman [:alice] [:anode]]

Maybe you should try just hitting the links from the command line to see if you can isolate something in your script that is interfering?  Because the links seem good and working on my end.

Comment 25

[alice nodelman [:alice] [:anode]]

Also, please limit your testing to 'macosx64', 'win32' and 'linux'.  These are the only platforms that we have test slaves for.  It you attempt to test other platforms the sendchanges are simply ignored.

Comment 26

[Wil Clouser [:clouserw]]

We should file a "have your way with" bug for this with steps on how to test it so QA isn't digging through this bug

Comment 27

[alice nodelman [:alice] [:anode]]

I've updated

https://wiki.mozilla.org/Buildbot/Talos/AddonTrigger#Variables

It now indicates the limitation as to supported platforms and the limitation on the versions of firefox that can be tested.

Comment 28

[Kumar McMillan [:kumar]]

Here is a bug to explore and test this further: bug 689187

I'm closing this one since we have an initial connection working. Thanks for your help Alice.