1239989 - Autophone - PROCESS-CRASH | autophone-s1s2 | application crashed [@ nsSocketTransport::InitiateSocket]

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

I've started seeing PROCESS-CRASH | autophone-s1s2 | application crashed [@ nsSocketTransport::InitiateSocket] and possibly related PROCESS-CRASH | autophone-s1s2 | application crashed [unknown top frame] on remote nytimes.

There must be a new pref we need to twiddle to turn off some background network access.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I want to be sure this is unrelated to the new servers at all.  I would be surprised if it was related, but want to get all out there.

Comment 2

[Bob Clary [:bc] (inactive)]

Could be related. I haven't seen this on the old mac mini, but I can't think of what would be the cause. Just wanted it on file so we could track it.

Comment 3

[Bob Clary [:bc] (inactive)]

mining logcat:

01-15 19:06:18.922 D/SwitchBoard( 2575): Read from server URL: https://switchboard-server.dev.mozaws.net/v1?uuid=d126444f-0b9e-4816-b5e9-1e3ff9b3ff25&device=flounder&lang=eng&country=USA&manufacturer=HTC&appId=org.mozilla.fennec&version=46.0a1

01-15 19:06:59.323 I/Gecko   ( 3531): FATAL ERROR: Non-local network connections are disabled and a connection attempt to nytimes.com (170.149.161.130) was made.
01-15 19:06:59.323 I/Gecko   ( 3531): You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.

mining ep1/nytimes

nytimes.com/index.html:    <link rel="stylesheet" type="text/css" href="http://css.nyt.com/css/0.1/screen/build/homepage/ie.css?v=012611">
nytimes.com/index.html:    <link rel="stylesheet" type="text/css" href="http://css.nyt.com/css/0.1/screen/build/homepage/ie6.css">

This is inside of an html comment:

<!--[if IE]>
    <link rel="stylesheet" type="text/css" href="http://css.nyt.com/css/0.1/screen/build/homepage/ie.css?v=012611">
<![endif]-->
<!--[if IE 6]>
    <link rel="stylesheet" type="text/css" href="http://css.nyt.com/css/0.1/screen/build/homepage/ie6.css">
<![endif]-->

Did the parser change recently?

nytimes.com/index.html:<!--<h4 class="sectionHeaderHome"><a><img src="http://graphics8.nytimes.com/images/opinion/sundayreview/sundayreview163sub.jpg"

ditto

<!--<h4 class="sectionHeaderHome"><a><img src="http://graphics8.nytimes.com/images/opinion/sundayreview/sundayreview163sub.jpg" width="163" height="15" alt="Sunday Review" /></a></h4>
-->

<!--<h4 class="sectionHeaderHome"><a><img src="http://graphics8.nytimes.com/images/opinion/sundayreview/sundayreview163sub.jpg" width="163" height="15" alt="Sunday Review" /></a></h4>
-->

nytimes.com/index.html:<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/fashion/05MOTH_RUBE/05MOTH_RUBE-moth.jpg" alt="Tiffany and Co.&rs

<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/fashion/05MOTH_RUBE/05MOTH_RUBE-moth.jpg" alt="Tiffany and Co.&rsquo;s Not-So-New Metal" width="151" height="151" /></a>

nytimes.com/index.html:<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/arts/05moth_comedy/05moth_comedy-moth.jpg" alt="Telling Tales Wit

<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/arts/05moth_comedy/05moth_comedy-moth.jpg" alt="Telling Tales With a Tear and a Smile" width="151" height="151" /></a>

nytimes.com/index.html:<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/garden/05MOTH_DYE/05MOTH_DYE-moth.jpg" alt="Colors of the Caldron
nytimes.com/index.html:<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/greathomesanddestinations/05MOTH_LOCATION/05MOTH_LOCATION-moth-v2
nytimes.com/index.html:<a><span class="img" src="http://i1.nyt.com/images/2012/04/05/opinion/05moth_opart/05moth_opart-moth.jpg" alt="Op-Ed: Down the 
nyt

etc. We should have already fixed these?

when I pulled the files from ep1, I did

git submodule update --init --remote

Wlach: Did I do something wrong in pulling these files from ep1?

Actually, looking at view-source:https://git.mozilla.org/?p=automation/ep1.git;a=blob_plain;f=nytimes/nytimes.com/index.html;hb=HEAD it appears these references are still there. I think the parser changed and we are now seeing some elements we didn't before?

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: bug-1239989-v1.patch

Not sure why I don't see this problem locally or on the original mac mini. Could be due to the lame way we copy the files to the phonedash server's html directory but that is a different issue. I checked phonedash's output using desktop Firefox and used wireshark and didn't see extraneous network accesses. /me cross fingers.

Comment 5

[Geoff Brown [:gbrown]]

Comment on attachment 8709111 [details] [diff] [review]
bug-1239989-v1.patch

Review of attachment 8709111 [details] [diff] [review]:
-----------------------------------------------------------------

These changes seem sensible to me, but I don't have experience with ep1 sanitization...might be a good idea to run this by someone like jmaher.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8709111 [details] [diff] [review]
bug-1239989-v1.patch

Review of attachment 8709111 [details] [diff] [review]:
-----------------------------------------------------------------

overall this looks right- hard to tell on this minified .js files.  If this doesn't work, we can dive in more- I have done this a lot, so let me know if I can help.

Comment 7

[Bob Clary [:bc] (inactive)]

https://git.mozilla.org/?p=automation/ep1.git;a=commit;h=249850d28ad39af0b1cdf26aa6d49c38bfd2d755

Comment 8

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/5e802b0b5206cd357562c1c2b8055b7c9fd7ce68

This hasn't fixed the entire issue. I'm going to do an interative approach and come back with the changeset when I have one that works.

Comment 9

[Bob Clary [:bc] (inactive)]

I can not reproduce this locally due to I believe the way my local instance of ep1 was copied into the phonedash/html directory.

We have lived with this mess for a long time and we have all wanted to change the way the html is copied into the phonedash for some time.

If it is ok with you, I would like to sync the server's phonedash html directories with mine for the time being in order to fix these crashes.

Then I will fix the real issue by making the phonedash html directory contain an exact copy of the autophone files directory and adjusting the paths in the test configs appropriately.

Comment 10

[Geoff Brown [:gbrown]]

OK with me.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

this sounds good- i recall pulling what little hair I had in my head trying to get this working originally with the talos tests- lets just get it working.

keep in mind the webpage could have location specific code which could cause mountain view ip addresses to query network resources differently or different files.

Comment 12

[Bob Clary [:bc] (inactive)]

Yeah, this will take some testing to be sure to get it right. But I'm tired of this mess and this will fix it once and for all for a minor investement in time.

Thanks, I'll rsync my local copy and hopefully silence the network access crashes for the moment.

Comment 13

[Bob Clary [:bc] (inactive)]

Attachment: bug-1239989-autophone-v1.patch

test run:
https://treeherder.allizom.org/#/jobs?filter-searchStr=autophone&exclusion_profile=false&repo=mozilla-inbound&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=2f8be7ffe2df&tochange=22e333b0a82c

Note that this no longer pushes the full talos directories to the device since they aren't needed. We just push the new manifests. I tested with remote but not local.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8709961 [details] [diff] [review]
bug-1239989-autophone-v1.patch

Review of attachment 8709961 [details] [diff] [review]:
-----------------------------------------------------------------

thanks, I had to do a double take on the .py changes, but they seem just fine!

Comment 15

[Bob Clary [:bc] (inactive)]

Attachment: bug-1239989-ep1-additional-v1.patch

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8710454 [details] [diff] [review]
bug-1239989-ep1-additional-v1.patch

Review of attachment 8710454 [details] [diff] [review]:
-----------------------------------------------------------------

nearly impossible to review the minified js stuff, but the rest was easy to follow along.

Comment 17

[Bob Clary [:bc] (inactive)]

https://git.mozilla.org/?p=automation/ep1.git;a=commit;h=249850d28ad39af0b1cdf26aa6d49c38bfd2d755

https://git.mozilla.org/?p=automation/ep1.git;a=commit;h=54a091054f5979099e0e9bda6513efb799744929

https://git.mozilla.org/?p=automation/ep1.git;a=commit;h=0333484d1d9aae0db0d2b1bd9582114851cccdc4

Comment 18

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/5e802b0b5206cd357562c1c2b8055b7c9fd7ce68
https://github.com/mozilla/autophone/commit/dcd0c8afad196dfbfa2a89500fbdfbf7c8bfdbb2
https://github.com/mozilla/autophone/commit/2d237a9c94b25461478b604102953c885016f46d

Comment 19

[Bob Clary [:bc] (inactive)]

marking fixed. if i still have non local network accesses i'll file a new bug.