602312 - post-image setup on talos-r3-w7-024

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Josh Matthews [:jdm]]

Rev3 WINNT 6.1 mozilla-central opt test xpcshell on 2010/10/06 12:05:07
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1286391907.1286396191.13321.gz

s: talos-r3-w7-024
TEST-UNEXPECTED-FAIL | c:\talos-slave\mozilla-central_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_canceled.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Should have failed to load URL (status is 0) - See following stack:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in onStopRequest: 2147500036 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\mozilla-central_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_passing.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/test_proxy-failover_passing.js | <html>
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in closure function: 2147500036 - See following stack:

Comment 1

[Honza Bambas (:mayhemer)]

This was probably caused by some misconfiguration on the box (?).

Both the tests are trying to use 'a_non_existent_domain_x7x6c572v:80' as a proxy (to check we fail to proxy and try another one in the list).  This first proxy connection is expected to fail - no domain of that name should be found.  

The second test is expected to get some content finally, exactly expects 'response body' as a payload.  But it gets the following:


<html>



<head>

<title>wsearch.net</title>

<script type="text/javascript" src="/js/general.js"></script>

<script type="text/javascript">

ChkRequestEnc('YToyMTp7aTowO3M6MTk6IjIwMTAtMTAtMDcgMDY6MjM6NDEiO2k6MTtzOjU6Ijg4OTQ3IjtpOjI7TjtpOjM7czo1ODoiTW96aWxsYS81LjAgKFdpbmRvd3MgTlQgNi4xOyBydjoyLjBiN3ByZSkgR2Vja28vIC80LjBiN3ByZSI7aTo0O3M6Mjg6Imh0dHA6Ly93d3cud3NlYXJjaC5uZXQvP19pbnYiO2k6NTtzOjEzOiI2My4yNDUuMjIyLjY2IjtpOjY7czoxOiIyIjtpOjc7czowOiIiO2k6ODtzOjA6IiI7aTo5O3M6MjoiVVMiO2k6MTA7czoxMDoiQ0FMSUZPUk5JQSI7aToxMTtzOjEzOiJNT1VOVEFJTiBWSUVXIjtpOjEyO3M6MjoiMTUiO2k6MTM7czoxMToid3NlYXJjaC5uZXQiO2k6MTQ7czo3MDoiaHR0cDovL3NlYXJjaHBvcnRhbC5pbmZvcm1hdGlvbi5jb20vP29faWQ9NjUwMTQmZG9tYWlubmFtZT13c2VhcmNoLm5ldCI7aToxNTtOO2k6MTY7TjtpOjE3O047aToxODtOO2k6MTk7TjtpOjIwO047fQ==');

</script>

<script type="text/javascript">



</script>

</head>



<frameset rows="100%,*" frameborder="no" border="0" framespacing="0">

  <!-- SCC a2 -->

  <frame src="http://searchportal.information.com/?o_id=65014&domainname=wsearch.net">



<noframes>

<body bgcolor="#ffffff" text="#000000">

  <a href="http://searchportal.information.com/?o_id=65014&domainname=wsearch.net">Click here to enter</a>.

</body>

</noframes>

</frameset>

</html>


The base64 string decoded:

a:21:{i:0;s:19:"2010-10-07 06:23:41";i:1;s:5:"88947";i:2;N;i:3;s:58:"Mozilla/5.0 (Windows NT 6.1; rv:2.0b7pre) Gecko/ /4.0b7pre";i:4;s:28:"http://www.wsearch.net/?_inv";i:5;s:13:"63.245.222.66";i:6;s:1:"2";i:7;s:0:"";i:8;s:0:"";i:9;s:2:"US";i:10;s:10:"CALIFORNIA";i:11;s:13:"MOUNTAIN VIEW";i:12;s:2:"15";i:13;s:11:"wsearch.net";i:14;s:70:"http://searchportal.information.com/?o_id=65014&domainname=wsearch.net";i:15;N;i:16;N;i:17;N;i:18;N;i:19;N;i:20;N;}

Something interacted with the requested.

Comment 2

[Legacy TBPL/Treeherder Robot]

philringnalda%gmail.com
http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1286636374.1286640469.27490.gz
Rev3 WINNT 6.1 tracemonkey opt test xpcshell on 2010/10/09 07:59:34

s: talos-r3-w7-024
TEST-UNEXPECTED-FAIL | c:\talos-slave\tracemonkey_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_canceled.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/tracemonkey_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Should have failed to load URL (status is 0) - See following stack:
TEST-UNEXPECTED-FAIL | c:/talos-slave/tracemonkey_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in onStopRequest: 2147500036 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\tracemonkey_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_passing.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/tracemonkey_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/test_proxy-failover_passing.js | <html>
TEST-UNEXPECTED-FAIL | c:/talos-slave/tracemonkey_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in closure function: 2147500036 - See following stack:

Comment 3

[Legacy TBPL/Treeherder Robot]

ehsan%mozilla.com
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1286915637.1286919761.11584.gz
Rev3 WINNT 6.1 mozilla-central opt test xpcshell on 2010/10/12 13:33:57

s: talos-r3-w7-024
TEST-UNEXPECTED-FAIL | c:\talos-slave\mozilla-central_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_canceled.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Should have failed to load URL (status is 0) - See following stack:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in onStopRequest: 2147500036 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\mozilla-central_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_passing.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/test_proxy-failover_passing.js | <html>
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in closure function: 2147500036 - See following stack:

Comment 4

[Legacy TBPL/Treeherder Robot]

adw%mozilla.com
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1286915637.1286919761.11584.gz
Rev3 WINNT 6.1 mozilla-central opt test xpcshell on 2010/10/12 13:33:57

s: talos-r3-w7-024
TEST-UNEXPECTED-FAIL | c:\talos-slave\mozilla-central_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_canceled.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Should have failed to load URL (status is 0) - See following stack:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in onStopRequest: 2147500036 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\mozilla-central_win7_test-xpcshell\build\xpcshell\tests\netwerk\test\unit\test_proxy-failover_passing.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/test_proxy-failover_passing.js | <html>
TEST-UNEXPECTED-FAIL | c:/talos-slave/mozilla-central_win7_test-xpcshell/build/xpcshell/tests/netwerk/test/unit/head_channels.js | Error in closure function: 2147500036 - See following stack:

Comment 5

[Honza Bambas (:mayhemer)]

Ted, please check comment 1.  I need advise here what happens on the testing box.  The issue seems to happen only on s: talos-r3-w7-024.

Comment 6

[(not currently active) Ted Mielczarek]

You need to ask Release Engineering. I'm moving this over to the RelEng component.

The only hits I find on Google for wsearch.net are about a Trojan, or this:
http://www.dslreports.com/forum/r20689069-FIX-for-wsearchnet-redirects-503s-on-SMC-wireless-modems

This machine doesn't have a trojan on it, does it? It's very suspect that only this box is failing.

Comment 7

[[e-mail] (:bhearsum)]

I'll pull this out for investigation.

Comment 8

[[e-mail] (:bhearsum)]

I started a scan with BitDefender which is estimated to take 8h or so to complete. Lukas is on buildduty, so I'll let her pick this up.

I also manually checked for the signs of infection listed on these two pages:
http://www.f-secure.com/v-descs/dnschang.shtml
http://www.symantec.com/security_response/writeup.jsp?docid=2005-042316-2917-99&tabid=2

As well, I reproduced manually by typing 'abc' into Firefox's or IE's address bar, which ended up redirecting me to wsearch.net. I could not reproduce this behaviour on another Windows 7 test machine.

Comment 9

[Chris Cooper [:coop] (he/him)]

Re-assigning to buildduty (lsblakk) based on comment #8. Probably need to be re-imaged if it is infected.

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: BitDefender results on talos-r3-w7-024

BitDefender says it's clean, but since the manual test (I also did the 'abc' in IE and got the same results) shows infection I'm filing an IT bug to get this machine re-imaged.

Comment 11

[[e-mail] (:bhearsum)]

Please don't re-image this yet....we really need to confirm if indeed it has a trojan and if so, how it got on our machine.

Comment 12

[[e-mail] (:bhearsum)]

(I'm happy to do this, if you've prefer)

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Haven't done bug 607140 on this, leaving that for the re-image.

Comment 14

[[e-mail] (:bhearsum)]

After this has sat for 3 months I'm pretty sure I'm never going to find time to properly investigate this. I've moved the dependent bug back to Server Ops, and this one can track post-image setup.

Comment 15

[Zandr Milewski [:zandr]]

I want to at least get an ack/nack out of infrasec before I wipe this box.

Comment 16

[Chris Lyon [:clyon]]

(In reply to comment #15)
> I want to at least get an ack/nack out of infrasec before I wipe this box.

We need to take a look at this box here in MV. please pull.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Chris, I'm sure you guys can figure this out more quickly than you can read this comment, but just running an nslookup at the command prompt will give bogus results.  ipconfig /all shows correct DNS servers configured.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

I've shut the machine down at joduinn's request.

Comment 19

[Zandr Milewski [:zandr]]

(In reply to comment #18)
> I've shut the machine down at joduinn's request.

Just for completeness, how did you do the shutdown? I was going to do a clean shutdown from physical console.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Via VNC (the shutdown button on the start menu)

Comment 21

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #15)
> I want to at least get an ack/nack out of infrasec before I wipe this box.

Do *not* wipe the box, but we need it physically handed to clyon asap.

Comment 22

[Zandr Milewski [:zandr]]

I've synced up with clyon, and will fetch it from scl1 today or tonight.

Comment 23

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

after talking with clyon, reassigning to infrasec for investigation. zandr will fetch from colo tonight/tmrw morning.

Comment 24

[Zandr Milewski [:zandr]]

Retrieved from scl1.
I'm PTO tomorrow but rmilewski will drop off the machine in the morning.

Comment 25

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #24)
> Retrieved from scl1.
> I'm PTO tomorrow but rmilewski will drop off the machine in the morning.

At 2:15pm today, clyon confirmed he had received the machine.

Comment 26

[Chris Lyon [:clyon]]

After looking at this for a bit, it looks like a few drivers were downloaded. I take it the issues started after this?

This is one example
Visited: cltbld@http://dl2.paretologic.com/downloads/drivercure/revenuewire/1.5/DriverCure%20Installer.exe

Then they was NVIDIA DRIVERS 195.62 WHQL download and the 9400 driver. 

I do question the DriverCure Installer.exe as there are a number of hits that have malware associated with it. 


Why are we downloading stuff to these machines and why do they have access to the internet?

Comment 27

[[e-mail] (:bhearsum)]

(In reply to comment #26)
> After looking at this for a bit, it looks like a few drivers were downloaded. I
> take it the issues started after this?
> 
> This is one example
> Visited:
> cltbld@http://dl2.paretologic.com/downloads/drivercure/revenuewire/1.5/DriverCure%20Installer.exe
> 
> Then they was NVIDIA DRIVERS 195.62 WHQL download and the 9400 driver. 
> 
> I do question the DriverCure Installer.exe as there are a number of hits that
> have malware associated with it. 
> 
> 
> Why are we downloading stuff to these machines and

We regularly have to make updates to the machines -- new SDKs, drivers, upgraded buildbot, other software. I can't find any references in bugzilla to NVIDIA driver changes around that time. We've looked at them *very* recently in bug 624044, but that's well after this bug was filed.

This bug has a reference to the NVIDIA control panel, but nothing that indicates someone was looking at changing the driver: https://bugzilla.mozilla.org/show_bug.cgi?id=589258

I can't find any other bugs of note.

> why do they have access to
> the internet?

They've got Internet access because it can't been turned off yet, but we're working in that direction in https://bugzilla.mozilla.org/show_bug.cgi?id=617414.

Comment 28

[Chris Lyon [:clyon]]

But this download is still of concern:

cltbld@http://dl2.paretologic.com/downloads/drivercure/revenuewire/1.5/DriverCure%20Installer.exe


Do you guys know what this is? Based on my searches, this could have been the entrance of Malware.

Comment 29

[[e-mail] (:bhearsum)]

I've never heard of it, but *if* somebody was looking at potentially upgrading the drivers on these machines it seems like something someone may have installed. It claims to be able to find compatible drivers for your hardware. I have no evidence that anyone installed it, though.

Comment 30

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #29)
> I've never heard of it, but *if* somebody was looking at potentially upgrading
> the drivers on these machines it seems like something someone may have
> installed. It claims to be able to find compatible drivers for your hardware. I
> have no evidence that anyone installed it, though.

1) If this is something we did during setup, I'd expect this same "DriverCure" to be on other win7 machines. Can you verify if this is on another win7 machine too?

2) If this program is *not* found on another machine - is it possible that this machine was loaned out to a developer for experimenting, and accidentally never reimaged after it was returned?

Comment 31

[[e-mail] (:bhearsum)]

I doubt this was something we did at initial setup, I was suggesting that someone may have been investigating a potential driver change by testing it on this *one* machine.

I couldn't find any record of this machine being loaned out in bugzilla or e-mail when i searched earlier, fwiw.

Comment 32

[Chris Lyon [:clyon]]

machine handed off to releng for re-image.

moving back over to RelEng, we are good with the system being wiped and re-imaged.

Comment 33

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #32)
> machine handed off to releng for re-image.
> 
> moving back over to RelEng, we are good with the system being wiped and
> re-imaged.

Filed bug#602312 to get machine reimaged.

Comment 34

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #33)
> (In reply to comment #32)
> > machine handed off to releng for re-image.
> > 
> > moving back over to RelEng, we are good with the system being wiped and
> > re-imaged.
> 
> Filed bug#602312 to get machine reimaged.

Linking a bug to itself is typically not useful, sorry. Of course, this should read:

Filed bug#604486 to get machine reimaged.

Comment 35

[Chris Cooper [:coop] (he/him)]

This slave has been running in production now for a few days without issue.