Closed Bug 635310 Opened 13 years ago Closed 11 years ago

Mozmill Endurance test for flash video performance

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Product:
Mozilla QA Graveyard
Component:
Mozmill Tests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: u279076, Unassigned)

References

Details

(Whiteboard: [endurance]) 

Create an endurance test to check for performance regressions related to flash video.

Steps:
*** begin iteration ***
1. Load a local flash video file in a new tab
*** end iteration ***
A dependency for this test is to get a local flash video into the mozmill-tests test-files repository.  I'm working with Rainer to get one of the Mozilla-owned OGG videos converted.
Why do we wanna limit us to local Flash files for endurance tests? I think it's more realistic to load an example from the web, i.e. from Youtube.
(In reply to comment #2)
> Why do we wanna limit us to local Flash files for endurance tests? I think it's
> more realistic to load an example from the web, i.e. from Youtube.

This was a big concern in our Endurance meeting this week.  Running iterative tests on a site like YouTube might be detected as an attack.
Ok, haven't had checked the notes. That makes sense. If those files are larger can we land those on our new shadow server, instead of having them in the Mozmill repository? The latter repo we should try to keep as small as possible.
(In reply to comment #4)
> Ok, haven't had checked the notes. That makes sense. If those files are larger
> can we land those on our new shadow server, instead of having them in the
> Mozmill repository? The latter repo we should try to keep as small as possible.

If that's the case, I'll add this blocked by bug 621794.
Henrik has recently brought to my attention that Flash videos use Core:Animation and Flash ads use Core:Graphics.  I'd like to add to this test the testing of Flash ads.  This way we will have more complete coverage.

Naturally, this means we now have 2 dependencies:
1) A flash video
2) A flash ad

Hosting-wise, we've agreed to host content on people.mo until a shadow server can be made available.
(In reply to comment #6)
> Henrik has recently brought to my attention that Flash videos use
> Core:Animation and Flash ads use Core:Graphics.  I'd like to add to this test
> the testing of Flash ads.  This way we will have more complete coverage.

On Windows that correlates to Direct 3D and Direct 2D. That means we cover multiple platforms with that. I'm not sure about Linux and hardware accelerated OpenGL layers.
I tried creating a patch for the test video -- it exceeds limitations.  We'll have to wait for a shadow server for this test.
A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/13350385
Now that we have the shadow server, what's the status of this test?
Blocked on getting flash videos to test. I still need to talk to Rainer and file a bug -- it's on my to do list.
I now have two flash videos from Rainer; one is .flv, the other is .swf. He assures me there is a difference between the two technologies but they are both "flash". I think it would be useful to use both in a single "flash video endurance" test.

Question is, where do we want these hosted? I assume I can just put this somewhere on mozqa.com, but where?
CCing Geo and Al on this bug as they might have an opinion about comment 12.
Why does it need to be on mozqa.com separately instead of just checking it into the QA hg repo for tests (which is mirrored up there)?
(In reply to comment #14)
> Why does it need to be on mozqa.com separately instead of just checking it
> into the QA hg repo for tests (which is mirrored up there)?

Only reason I haven't done that is because Henrik expressed reservations in the past checking in large files to the repos.  Each video is about 5MB.
Ah.

Well, as long as it isn't continually refreshing and reloading the video file (and thus using bandwidth constantly), I see no reason not to have it on mozqa.com.
(In reply to comment #16)
> Ah.
> 
> Well, as long as it isn't continually refreshing and reloading the video
> file (and thus using bandwidth constantly), I see no reason not to have it
> on mozqa.com.

Is there anywhere on mozqa.com that I should put the file? We should probably have a global "test file" staging area...

mozqa.com/test-files/plugins/flash
 Well we do. The problem is that it expects test files to come from the HG repo, which we copy out every 15 minutes.

See, for example, http://mozqa.com/data/firefox/content_handling/test-mailto-peoplemail.html.
Ok, sounds like we might have a larger discussion here. I'll wait for feedback from Geo and Henrik before proceeding. The following questions need answering:

1) Can I put the flash video files into HG (~10MB combined)?
2) If not, where on mozqa.com can they live?

My suggestion to #2 is to create a staging area for test files which are "too big" for the repo. Something like:
mozqa.com/test-files/plugins/flash
Re: file on hg:

The general rule on version control is that largish binaries shouldn't be checked in; depending on the system implementation they can really chew significant resources. I have no idea if hg is friendly towards binaries or not. 

But if we have a limitation on ours, my guess is that it's technical and getting it lifted for a single scenario won't happen. Doesn't mean we can't still try, but I'd prefer it wasn't our first option. 

I'm also not sure the crowd extension users would appreciate having to download too many large binaries, so this might be something we'd exclude from them anyway somehow. AFAIK, that would require at least a secondary repo, since they'll get everything in our primary.

Re: stuff on mozqa.com, I'd really like to be able to move forward without figuring out the version control thing yet.

Al, can we expose sftp or similar so we can stage files without jumping through the HG hoop? Or can we just scp it into somewhere? We need to be able to do this for test creation purposes anyway, if nothing else.

Limited user accounts could be created for this purpose so you're not giving out the master PW everywhere and for auditing reasons (probably a good idea anyway).

What are the other technical issues we'll deal with doing this (for example, does the refresh nuke all existing files first, or do a pull in place?)
Thanks Geo. I completely agree with your assumptions on HG. I think hosting unchanging binaries to a staging area on mozqa.com is the way to go. I would like to agree on a hierarchical structure for organizing these binaries. As suggested before:

mozqa.com/test-files/plugins/flash

Assuming we can agree to this hierarchy, what's the best way to upload my files?
I'm fine with whatever structure makes sense. If it's possible to separate them, putting them in a directory not controlled by hg would be good. If not, then we'll just have to be real careful with filenames/paths.
I'll have to set up ssh to people and make a world writable area on the box under the web directory.

The nice thing about the HG way was that it happened automagically. :-)
(In reply to comment #23)
> I'll have to set up ssh to people and make a world writable area on the box
> under the web directory.

Thanks Al. Did you want me to file a bug for that or will you just comment here when it's ready?
Please file a bug and assign it to me.
Depends on: 662640
(In reply to comment #25)
> Please file a bug and assign it to me.

See bug 662640
I don't see a problem in putting those two 5MB files into the hg repository. That's not that much as I have had expected earlier. This repository is for test data and we should avoid to have a parallel handling of test files as much as possible.
Landed flash videos to litmus-data:
http://hg.mozilla.org/qa/litmus-data/rev/308b1bab1148

No review due to binary nature of the patch.
No attachment due to 12MB patch size.

Videos tested as playing fine directly from http://hg.mozilla.org
Those are now up on mozqa.com:
http://www.mozqa.com/data/firefox/plugins/flash/

We should also create two web pages which can handle those files and which we can use for automated tests.
I'm converting this bug into a tracking bug for overall Flash endurance test development. The following bugs will be branched off this one:

* swf URL
* swf embed
* swf object
* flv URL
* flv embed
* flv object

Each of these will inherently need a test page checked into litmus-data and mozmill-tests as well.
Adding 2 more tests here:

* swf iframe
* flv iframe
Actually, some preliminary testing indicates that Firefox tries to download FLV files (at least here on my Mac). However, Firefox does allow loading and playing of SWF files in the browser -- we will just focus on SWF for the time being.
Depends on: 671476
Depends on: 671477
Depends on: 671478
Depends on: 671479
vlad.maniac changed story state to started in Pivotal Tracker
Depends on: 717640
We already have flash video tests. Dave, anything this bug would help us with?
Assignee: anthony.s.hughes → nobody
Flags: needinfo?(dave.hunt)
Whiteboard: [endurance]
This depends on bug 717640, which is still open.
Flags: needinfo?(dave.hunt)
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.