975191 - Self serve API for oak not building Oak nightly builds even know there is a nightly from 2 days ago

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Brian R. Bondy [:bbondy]]

+++ This bug was initially created as a clone of Bug #857397 +++

I was hoping someone can help me troubleshoot why I can't initiate nightly builds on Oak anymore.

You can see on 2014-02-18 that a nightly was built:
https://tbpl.mozilla.org/?tree=Oak&rev=4102831c51b4

I tried to build one at 3 different times on 2014-02-20 but no nightly builds are showing up on tbpl and on the ftp.

I tried to do the nightly builds with the tip of Oak:
1c875d373815

To do this I submitted that changeset in the box for nightly and hit submit here:
https://secure.pub.build.mozilla.org/buildapi/self-serve/oak

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

From bm62's self-serve agent log:

2014-02-20 11:07:34,027 Received {u'action': u'new_nightly_at_revision', u'body': {u'priority': 0, u'revision': u'1c875d373815', u'when': 1392923254, u'branch': u'oak', u'request_id': 652967}, u'who': u'[e-mail]'}
2014-02-20 11:07:34,028 New nightly by [e-mail] of oak 1c875d373815
2014-02-20 11:08:07,911 buildernames are []
2014-02-20 11:08:07,912 INSERT INTO sourcestamps
                (`branch`, `revision`, `patchid`, `repository`, `project`)
                VALUES
                (:branch, :revision, NULL, '', '')

2014-02-20 11:08:08,161 Created sourcestamp 7102600
2014-02-20 11:08:08,162 INSERT INTO buildsets
        (`external_idstring`, `reason`, `sourcestampid`, `submitted_at`, `complete`, `complete_at`, `results`)
        VALUES
        (:idstring, :reason, :sourcestampid, :submitted_at, 0, NULL, NULL)
[buildset and buildrequest creation left off]

The key bit is 'buildernames are []' from
  http://hg.mozilla.org/build/buildapi/file/73dbae9cb3db/buildapi/scripts/selfserve-agent.py#l478

Tentatively pegging this as a regression from bug 846104.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I strongly suspect the problem is that we're doing this:
 SELECT DISTINCT buildernames FROM buildrequests WHERE
     buildername     LIKE :inclusion1 AND buildername     LIKE :inclusion2 \
 AND buildername NOT LIKE :exclusion1 AND buildername NOT LIKE :exclusion2
 AND submitted_at > :two_weeks_ago

The first AND should be an OR, with appropriate parentheses.

In other news, that query takes 30 seconds because it has to look at all 32 million rows in the table, and OR makes slower. The index on buildername doesn't help because of the LIKE conditions, and there's no index on the submitted_at column. We could add an index, but there's already one on claimed_at we could use instead.

Better would be querying like this:
 SELECT DISTINCT buildernames FROM buildrequests WHERE buildername LIKE '%oak%nightly%' \
   AND claimed_at > :two_weeks_ago
which runs in 1.5 seconds or so. Then convert the inclusions and exclusions to regex that get run in the python code.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

This approach neglects other consumers of _create_build_for_revision() though.

zeller, will bug 793989 resolve this issue by making the bad code go away ?

Comment 4

[John Zeller [:zeller]]

(In reply to Nick Thomas [:nthomas] from comment #3)
> This approach neglects other consumers of _create_build_for_revision()
> though.
> 
> zeller, will bug 793989 resolve this issue by making the bad code go away ?

Nothing in bug 793989 is going to modify _create_build_for_revision() code. In regards to selfserve-agent, all bug 793989 changes are going to be within the new function do_new_build_for_builder().

Comment 5

[Brian R. Bondy [:bbondy]]

Would resetting (somehow?) away oak help here?  For example currently to test bug 974570 I can at best (If I make no mistakes) land it by Friday because I need to wait a day for a Nightly build to trigger.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Pretty sure the problem lies entirely in buildapi itself.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I backed out bug 846104, could you give self-serve another go ?

Comment 8

[Brian R. Bondy [:bbondy]]

Thanks, I issued another nightly:

> {
>   "status": "OK",
>   "request_id": 670791
> }

... (2 minutes later)...

And the N icons are showing up, which is good.  Looks like self serve nightly builds broke from bug 846104 at least on Oak.