841566 - Turn on testserver logging for WebRTC-related systems

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Adam Roach [:abr]]

Currently, progress on intermittent bug 841496, bug 841150, and bug 839677
has proven difficult; these bugs are difficult to reproduce, and TBPL logs
provide only the roughest estimation of where something may have gone wrong.
For as long as these (and similar) intermittant bugs exist, we would benefit
greatly from having relevant logging turned on.

Comment 1

[Adam Roach [:abr]]

Attachment: Turn on testserver logging for WebRTC-related systems

This patch should activate logging on the test servers for the signaling,
webrtc, ice, and transport subsystems.

Comment 2

[Adam Roach [:abr]]

If all goes well, our additional debugging should show up in the logs for this push:

https://tbpl.mozilla.org/?tree=Try&rev=4247a24bb8e5

Comment 3

[Adam Roach [:abr]]

Re-try after fixing the patch for bug 841457:
https://tbpl.mozilla.org/?tree=Try&rev=a437dbcdc5f7

Comment 4

[Adam Roach [:abr]]

Okay, that that looks like it actually broke automation. Third time's a charm...
https://tbpl.mozilla.org/?tree=Try&rev=8f874b4ae911

Comment 5

[(not currently active) Ted Mielczarek]

I told Adam that the sheriffs were the people most likely to care about stuffing extra debug logging into the tinderbox logs.

Comment 6

[Adam Roach [:abr]]

Looks like Ethan just took care of some of the FORCE_PR_LOG stuff that I was going to have to work with -- I'm going to depend this bug on that one, since I've just rebased my (local, work-in-progress) patch on top of that one.

Comment 7

[Adam Roach [:abr]]

Okay, this did manage to turn on ICE debugging, so it's now clear that the environment variables are being propagated to the tests:

https://tbpl.mozilla.org/?tree=Try&rev=3e2ffe882d36

Comment 8

[Adam Roach [:abr]]

Attachment: Turn on testserver logging for WebRTC-related systems

Comment 9

[Adam Roach [:abr]]

I think this most recent patch probably does what we want. It uses a reasonable log level for ICE, signaling, and mtransport. I've also demoted a couple of relatively noisy log messages that weren't adding any value at this level, as well as lengthened a buffer that was too short (and generating a metric ton of error logs as a consequence).

Proof's in the pudding -- the peerconnection mochi tests should contain both ICE logging and "cpr" logging in both opt and debug builds:

https://tbpl.mozilla.org/?tree=Try&rev=25e59929e596

Comment 10

[Ed Morley [:emorley]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> I told Adam that the sheriffs were the people most likely to care about
> stuffing extra debug logging into the tinderbox logs.

Roughly how many additional lines of output are we talking about per mochitest chunk)? (However, if this is just temporary then I don't mind regardless of length) :-)

Comment 11

[Adam Roach [:abr]]

Ed -- here's the resulting output. All three of these should look roughly the same, and gives you a good feel for how much extra output we'll be introducing.

And yes, the intention is very much that this is temporary.

Example linux opt log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19974947&tree=Try&full=1

Example windows opt log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19979225&tree=Try&full=1

Example linux debug log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19975741&tree=Try&full=1

Comment 12

[Adam Roach [:abr]]

Ryan -- can you take a look at the logs in comment 11 and let me know if you have any objection to my turning on this level of logging temporarily until we nail down the root cause of some intermittent bugs?

Comment 13

[Adam Roach [:abr]]

Phil -- same question as I have for Ryan above.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Looks fine to me.

Comment 15

[Adam Roach [:abr]]

Ed -- given that the plan is for this to be temporary, I'm basing your position on the statement "I don't mind regardless of length".

Comment 16

[Ed Morley [:emorley]]

Yup, no qualms with this at all; but thank you for checking :-)

Comment 17

[Phil Ringnalda (:philor)]

Yeah, mochitest-3 logs are half or a third the size of things like mochitest-1 and browser-chrome, you'd have to get stuck spewing in a loop for a while to run into any problems.

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 716926 [details] [diff] [review]
Turn on testserver logging for WebRTC-related systems

Review of attachment 716926 [details] [diff] [review]:
-----------------------------------------------------------------

r+, though a build peer should look at automation.py.in

Comment 19

[(not currently active) Ted Mielczarek]

Comment on attachment 716926 [details] [diff] [review]
Turn on testserver logging for WebRTC-related systems

Review of attachment 716926 [details] [diff] [review]:
-----------------------------------------------------------------

Technically you need a test harness peer, not a build peer, but conveniently I own both of those modules anyway.

Comment 20

[Adam Roach [:abr]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/86c4d6a9775a

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/86c4d6a9775a