1362554 - mitmproxy replay files for additional sites

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Armen [:armenzg]]

As per conversation with bsmedberg we will need some cookies to make this work properly.

jmaher: our goal is to get started with Amazon, Google Search and YouTube. Are you trying to re-adjust the requirements document? Or have a plan on how to get the last two sites working properly?


9:25 AM B<bsmedberg> the recording has you already logged-in to facebook
9:25 AM (login is not part of the process)
9:26 AM B<bsmedberg> so in order for a recording to work, you have to send the cookies the same as if you were logged in
9:33 AM A<@armenzg> Armen Zambrano Gasparnian bsmedberg: does mitmproxy record the cookies? or do we have to store them and reimport it into the player host?
9:34 AM B<bsmedberg> That question could mean several things, let me unpack a bit: 9:34 AM 1) does it record the set-cookie headers in general? yes.
9:35 AM 2) does a replay of loading the facebook homepage etc have set-cookie headers? I don't think so, because those are set at login time not load time.
9:36 AM B<bsmedberg> 3) Does mitmproxy itself need the cookies to replay the load? No, by default and in our configuration it doesn't do cookie header matching.
9:36 AM 4) Does the facebook web app itself change behavior based on cookies in ways that destroy replay if you don't have them? Yes I think so.
9:37 AM bsmedberg: what options do we have to have a similar behaviour as the user would have? do we need to shove the cookies into the profile before we replay?
9:37 AM B<bsmedberg> probably, yes
9:39 AM A<@armenzg> Armen Zambrano Gasparnian bsmedberg: do you have instructions for that? or do you want us to figure it out? or should we push this for the next iteration?
9:41 AM B<bsmedberg> My current instructions are written entirely in terms of a manual testing procedure at https://docs.google.com/a/mozilla.com/document/d/1ViLJw-ZH9K8ecr5itZaXcnyxEDDYtJcv1KOSXMO5dVA/edit?usp=sharing
9:42 AM I need you and the team to help translate that into talos in the best way possible.
9:42 AM but
9:42 AM I'd say that we should start out with the sites that don't need a login: google search, amazon, youtube
9:42 AM B<bsmedberg> let's get this working end-to-end with the simple cases first, and then go back and start picking off the harder things
...

3:55 PM J<jmaher> ok, it seems that we should have a valid pageset by now, it has been a couple weeks; maybe there are other reasons why they pages are not finalized
3:56 PM A<@armenzg> Armen Zambrano Gasparnian jmaher: what are you trying to figure out?
3:57 PM this morning bsmedberg mentioned that we should use those 3 sites w/o logging since the other two require login + cookies
3:57 PM B<bsmedberg> login
3:57 PM J<jmaher> armenzg: check off some of the original requirements; if we don't have a pageset, then I think the original requirements need some adjusting
3:57 PM A<@armenzg> Armen Zambrano Gasparnian the matter of the cookies he said we could punt for now

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to have a list of pages we want.  I know for the manual work to take a video and get measurements at certain points there is recordings of some basic browsing- we should be getting full recordings so we have options in the future.

there is a pending question on youtube to consider using the new site, but that isn't documented in this scenario document:
https://docs.google.com/document/d/1ViLJw-ZH9K8ecr5itZaXcnyxEDDYtJcv1KOSXMO5dVA/edit?ts=59088023#

Also it sounds like gmail is not realistic for now, so possibly this bug is:
facebook.com
amazon.com
google search

and when we get confirmation, youtube.com.

:cpeterson- can you help get to a resolution on youtube.com?
:bsmedberg- can you confirm we should be using the same recordings for the manual video analysis as well as in automation?

Comment 2

[Benjamin Smedberg]

> :cpeterson- can you help get to a resolution on youtube.com?

For now, please use the currently-deployed youtube site. We will probably want to switch to the new site at some point later, but for now we want to stick to what users actually see.

> :bsmedberg- can you confirm we should be using the same recordings for the
> manual video analysis as well as in automation?

This is incorrect. The video analysis will be recreating their recording new each week. This project wants recording to laster for longer periods (a few months at a time probably). So you should probably create your own base recordings.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

thanks :bsmedberg- we will make our own recordings.

Cristian- when you are doing this locally, do the test account logons that you have work for supporting the scenarios?  I want to make sure there are no issues with the test accounts.

Comment 4

[Cristian Chiorean]

Joel - we use our personal accounts in order to run those manual scenarios. We are not aware of any specific tests accounts we should use, nor the reference documents don't specify to use the test accounts when running the scenarios. But if there are any accounts we should use we are glad to have them.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks for the information everyone.  Ideally we use the same accounts, but if not it is ok.  Since we are ok with updating the recordings, I believe we can use the test accounts and update later if needed.

:rwood, can you pick this up next week?

Comment 6

[Robert Wood [:rwood]]

Yes, this will also require some additional talos work to support mitproxy playback of sites that require login. I'll file more specific bugs against talos for that work as required.

Comment 7

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) from comment #1)

> Also it sounds like gmail is not realistic for now, so possibly this bug is:
> facebook.com
> amazon.com
> google search
> 
> and when we get confirmation, youtube.com.
> 

To clarify, do we need logins for each of those? For facebook.com it's clear we do. It's not clear in the 'topsites' document though, if for amazon.com, youtube.com (and google searches) if the intention is to have a user logged in (?) while visiting those URLs. It's not really  necessary to be logged in for those, and I would prefer to reduce complexity where we can.

Comment 8

[Benjamin Smedberg]

We don't log in to amazon.

For youtube, I don't *think* it's necessary to be logged in. It is important that we have realistic results for "related" and so forth.

google search doesn't need to be logged in.

Comment 9

[Robert Wood [:rwood]]

Hey Ben,

Recording a mitmproxy archive that includes the actual facebook login itself, and then playing that back using your custom playback script [1] works even without copying the Firefox cookies. When navigating to facebook it automatically goes to the test account timeline appearing to be all logged in.  When I playback the same file but NOT using the custom playback script, when navigating to facebook it just stops at the facebook login screen.

Does this seem legit? And if so, is it ok to use the mitmproxy playback archive and custom script without the cookies, since that works, but is that still legit from a 'user simulated' or perf point of view?

Thanks!

[1] https://github.com/bsmedberg/mitmdump-alternate-server-replay

Comment 10

[Benjamin Smedberg]

Yeah, that seems legit (the custom script is going to be always necessary). As long as we don't actually have to log in during the test it should be ok.

Comment 11

[Robert Wood [:rwood]]

New mitmproxy recording archive created on win10, which contains pagesets for the 'top 5 sites' in the top sites document (minus gmail): a google search, an amazon search, youtube homepage, and a logged-in facebook test account timeline. The mitmproxy archive will be uploaded to tooltool in Bug 1370982. New talos quantum-pageload-e10s tests using these new archives will be delivered in Bug 1370607.