1155242 - community bugzilla systems talking to a known malicious IP

Status: RESOLVED FIXED

(Bugzilla :: bugzilla.org, defect)

Description

[Michal Purzynski [:michal`] (use NEEDINFO)]

Hello.

Our network security monitoring system has detected connections from

bugzilla2.community.scl3.mozilla.com
landfill.bugzilla.org

to a DNS server 195.22.26.248 which was recently used for distributing malware. Traffic like this can be a sign of a former infection, where malware tries to connect to one of the command and control servers to receive instructions what to do next.

This 195.22.26.248 has been sinkholed by a security firm and that's never without a reason.

https://www.virustotal.com/en/ip-address/195.22.26.248/information/

Shows the (bad) history of this IP address.

Who is the best person to investigate why two community bugzilla systems might be connecting to a known bad malicious IP?

Comment 1

[Dave Miller [:justdave]]

Both of these boxes NAT other hosts, so they may or may not be infected themselves.  bugzilla2 has almost nothing running on it except KVM, so it's likely one of the VMs running on it in its case. (not to rule it out, but there's almost nothing to infect there, so it's more likely one of its VMs is infected).

landfill is the default gateway for anything running on Bugzilla's internal vlan, so it could be any host behind it.  On the other hand. landfill has a crapton of stuff running on it, so an infection of itself is still a likely option.

I'm open for help if you'd like me to give you access to these boxes.

Comment 2

[Dave Miller [:justdave]]

One of the VMs on bugzilla2 is infra.bugzilla.lan, which along with landfill.bugzilla.lan (internal interface to the internal vlan) are the recursive DNS servers for all of the machines on the internal vlan.  Anyone doing a DNS lookup of a domain which is served by that rogue DNS server from anywhere in the vlan would have likely triggered a hit from both of those servers.

Comment 3

[Teemu Mannermaa (:wicked)]

Both of these DNS servers have this IP-address in their lame server delegation logs as 71.221.73.178.in-addr.arpa reverse domain seems to be delegated to ns1.transitionalprotocol.net and ns2.transitionalprotocol.net DNS-servers that are pointing to 195.22.26.248 IP.

This log is from landfill (latest lame-server log):
06-Apr-2015 04:09:59.188 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
07-Apr-2015 04:09:16.519 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
08-Apr-2015 04:12:44.378 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
09-Apr-2015 04:09:36.865 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
10-Apr-2015 04:11:16.186 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
11-Apr-2015 04:13:26.208 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
12-Apr-2015 04:10:15.231 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
13-Apr-2015 04:10:05.655 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
14-Apr-2015 04:12:11.579 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
15-Apr-2015 04:11:38.371 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
16-Apr-2015 04:10:54.144 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
16-Apr-2015 10:17:37.704 connection refused resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53

And this is from infra.bugzilla.lan latest lame-server log (which will be seen as bugzilla2.community.scl3.mozilla.com in the outside world as it has no public IP but is NATted behind bugzilla2):
05-Apr-2015 04:12:36.170 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
06-Apr-2015 04:09:54.175 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
07-Apr-2015 04:09:12.946 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
08-Apr-2015 04:12:39.371 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
09-Apr-2015 04:09:32.631 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
10-Apr-2015 04:11:10.889 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
11-Apr-2015 04:13:21.571 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
12-Apr-2015 04:10:20.218 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
13-Apr-2015 04:09:59.642 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
14-Apr-2015 04:12:16.717 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
15-Apr-2015 04:11:41.621 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
16-Apr-2015 04:10:47.636 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53
16-Apr-2015 10:17:34.397 error (connection refused) resolving '71.221.73.178.in-addr.arpa/PTR/IN': 195.22.26.248#53

Comment 4

[Teemu Mannermaa (:wicked)]

Looks like we got two requests to our landfill web server from the IP that gets reverse DNS queries every night:
 httpd/ssl_access_log:178.73.221.71 - - [18/Mar/2015:03:57:59 -0700] "GET / HTTP/1.1" 200 4453 "-" "Xenu Link Sleuth/1.3.8" TLSv1 AES128-SHA
 httpd/access_log:178.73.221.71 - - [18/Mar/2015:00:29:08 -0700] "GET / HTTP/1.1" 200 4453 "-" "Xenu Link Sleuth/1.3.8"

We actually do have webalizer running on landfill that will go through /var/log/httpd/access_log log and tries to reverse resolve all the IPs. This would explain the lame server entries. Webalizer is set to run daily by cron and the daily run usually starts around 4:02 on landfill. So depending on how much is there is to process and load on landfill, the offending DNS resolve attempt can easily happen around 4:10 every night.

Comment 5

[Michal Purzynski [:michal`] (use NEEDINFO)]

Confirmed!! That would be it. Thanks a lot.

Comment 6

[Frédéric Buclin]

Any reason why this bug is kept private?

Comment 7

[Frédéric Buclin]

(In reply to Frédéric Buclin from comment #6)
> Any reason why this bug is kept private?

6 months later, still no reply. Removing sec flag.