895958 - buildbot master release e-mail doesn't contain hostname anymore

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

Used to look like:
Return-Path: [e-mail]
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO
    zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by
    zmmbox6.mail.corp.phx1.mozilla.com with LMTP; Tue, 26 Mar 2013 04:52:10
    -0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
    by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 4EF7F1063E6;
    Tue, 26 Mar 2013 04:52:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from buildbot-master30.srv.releng.scl3.mozilla.com (v-1030.fw1.releng.scl3.mozilla.net [63.245.214.82])
    by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id 1F67CF2262;
    Tue, 26 Mar 2013 04:52:10 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf8"
Date: Tue, 26 Mar 2013 04:52:05 -0700
Subject: [release] Firefox 20.0b7 build1: completed repack_4/6 on win32
From: [e-mail]
References: <[e-mail]>
In-Reply-To: <[e-mail]>
To: [e-mail], [e-mail]
Message-Id: <[e-mail]>

repack_4/6 complete for Firefox 20.0b7 build1 on win32

-buildbot

Now looks like:
Return-Path: [e-mail]
Received: from zmmta2.mail.corp.phx1.mozilla.com (LHLO
    zmmta2.mail.corp.phx1.mozilla.com) (10.20.77.30) by
    zmmbox5.mail.corp.phx1.mozilla.com with LMTP; Thu, 18 Jul 2013 20:22:26
    -0700 (PDT)
Received: from zmmta2.mail.corp.phx1.mozilla.com (localhost6.localdomain [127.0.0.1])
    by zmmta2.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 52C0A1021A3;
    Thu, 18 Jul 2013 20:22:26 -0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
    by zmmta2.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 46E26102019;
    Thu, 18 Jul 2013 20:22:26 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from localhost.localdomain (v-1030.fw1.releng.scl3.mozilla.net [63.245.214.82])
    by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id CC792F2051;
    Thu, 18 Jul 2013 20:22:25 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Date: Thu, 18 Jul 2013 20:22:17 -0700
Subject: [release] Firefox 23.0b7 build1: completed repack_6/10 on linux
From: [e-mail]
References: <[e-mail]>
In-Reply-To: <[e-mail]>
To: [e-mail], [e-mail]
Message-Id: <[e-mail]>

repack_6/10 complete for Firefox 23.0b7 build1 on linux

-buildbot


Note the localhost.localdomain vs. buildbot-masterXX in the last "Received" header.

Comment 1

[Rail Aliiev [:rail]]

I think it's related to missing FQDN entry in /etc/hosts

Comment 2

[Rail Aliiev [:rail]]

http://hg.python.org/cpython/file/f8df7c50132f/Lib/smtplib.py#l261

smtplib uses socket.getfqdn() which returns localhost.localdomain (the first entry with '.' in /etc/hosts).

I fixed /etc/hosts entries on all masters manually, but it would be great to manage them by puppet.

http://docs.puppetlabs.com/references/latest/type.html#host doesn't provide a method to explicitly manage entries (you can't specify their order for example). 

We can manage /etc/hosts like a regular file, but in this case we'd need to be careful for AWS hosts - they need entries for "repos" because there is no way to  specify the "search" option in DHCP.

Dustin, ho w do you feel about managing /etc/hosts as a regular file (using concat, maybe)?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Is there anything accessing 'repos' other than mozharness?  That's a serious reliability flaw anyway, so I'd rather just fix mozharness to use pip.conf, and manage all /etc/hosts files identically.

Comment 4

[Rail Aliiev [:rail]]

A quick searchreturns the puppet repo as well:

http://hg.mozilla.org/build/puppet/file/742b9e62a0d9/modules/mockbuild/files/mozilla-centos6-x86_64.cfg#l48
http://hg.mozilla.org/build/puppet/file/742b9e62a0d9/setup/common/yum-setup-repos.erb#l12

(In reply to Dustin J. Mitchell [:dustin] from comment #3)
> Is there anything accessing 'repos' other than mozharness?  That's a serious
> reliability flaw anyway, so I'd rather just fix mozharness to use pip.conf,
> and manage all /etc/hosts files identically.

One little thing here. We'll need to manage pip.conf on Windows in this case.

Dustin, do you think that we can use split horizon DNS and return different IPs depending on the source IP?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

We'll definitely manage pip.conf on Windows.

We're already using scoped DNS, but that's not the point - any system that accesses a puppetmaster needs to know about *all* puppetmasters and be willing to try them all until it finds one that works, so that involves more than a DNS hack.

Mock build could be a problem - I've filed bug 899548 for that purpose, since it's out of scope here.

By the way, socket.getfqdn() is an abomination.  It will hang forever in certain circumstances (like on a laptop on a plane).  Another good fix may be to pass the real fqdn as `local_hostname` to SMTP(..)[1], although it looks like our ubuntu hosts have an unqualified hostname in `uname -n`.

[1] http://docs.python.org/2/library/smtplib.html#smtplib.SMTP

Comment 6

[Rail Aliiev [:rail]]

This is fixed for the current masters.

I added a step to the master docs to make sure we get this set up properly in the future.

https://wiki.mozilla.org/index.php?title=ReleaseEngineering/AWS_Master_Setup&diff=707802&oldid=675895