1039739 - Flame builds no longer have gaia commit info in settings app

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[William Lachance (:wlach)]

We used to have a file in the setting's application's 'application.zip' in resources/ called gaia_commit.txt which had gaia commit info. This seems to no longer be present in builds, which breaks getting proper gaia version info for eideticker and probably other testing tools via mozversion (http://hg.mozilla.org/mozilla-central/file/f6e46d1fc903/testing/mozbase/mozversion/mozversion/mozversion.py#l123).

I did some quick bisection on b2g-inbound on flame, and it looks like this problem first starts occuring in this build:

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-eng/20140716013201/

The previous inbound build doesn't have this problem:

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-eng/20140716012802/

The commit on the inbound build with the problems is d74dd8f351866264f198985a9414633e957dbfff, which corresponds to:

https://github.com/mozilla-b2g/gaia/commit/d74dd8f351866264f198985a9414633e957dbfff

This commit was intended to fix creating this file on windows, but it also alters the way we copy the git commit override file into the settings application. I *believe* (not 100% sure) we use a commit file override in the gaia build process, so perhaps this change isn't working as expected:

https://github.com/mozilla-b2g/gaia/commit/d74dd8f351866264f198985a9414633e957dbfff#diff-00aa0db2ae450f56abde226f0551699bL91

Yuren, did you test this codepath before committing? It doesn't actually seem to have anything to do with your intended change, if I understand it correctly. Maybe the solution is to just back that one part out of the patch?

Comment 1

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

:wlach, I use the command to verify but looks good:

$ echo "aaa" > gaia_commit_override.txt && APP=settings make && cat build_stage/settings/resources/gaia_commit.txt

I'll investigate it.

Comment 2

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: github PR: https://github.com/mozilla-b2g/gaia/pull/21861

I made a quick patch to restore that part to override commit file in settings/build/build.js but I'm not sure it can fix the problem since it can not be reproduced on my side.

:wlach, do you have any way to test this pull request if it can fix that problem?

Comment 3

[William Lachance (:wlach)]

This works for me too. Aki, do you have any idea why this change might cause the gaia revision to not be picked up by the build?

Comment 4

[Aki Sasaki (not active)]

I don't, no.  Usually when I'm faced with a problem like this I add more verbose logging around the problem.

Comment 5

[Jonathan Griffin (:jgriffin)]

One thing that looks potentially suspicious is this line:

  git.initPath(utils.getEnvPath());

I'm not sure what this does, but it may be relying on something that's not set up on the slaves.

Comment 6

[William Lachance (:wlach)]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> One thing that looks potentially suspicious is this line:
> 
>   git.initPath(utils.getEnvPath());
> 
> I'm not sure what this does, but it may be relying on something that's not
> set up on the slaves.

This was set up before without causing any problems:

https://github.com/mozilla-b2g/gaia/commit/d74dd8f351866264f198985a9414633e957dbfff#diff-00aa0db2ae450f56abde226f0551699bL96

I did some detective work, and it looks like, in fact, we *are* using git to get the revision in the builds, I see this kind of log message in the build:

[cmd] git --git-dir=/builds/slave/b2g_b2g-in_flame_eng_dep-00000/build/gaia/.git log -1 --format=%H%n%ct HEAD

HOWEVER, I found out that we're not emitting any error messages if the git command doesn't succeed, which we should be. I'll file a seperate bug for this.

The only thing I can think of for the problem is that we were quoting the string in --format before, whereas we aren't now. I'm not sure though. Almost certainly the command is erroring out somehow though. I suggest we get in some code to add some logging to figure out what's going on, then do up a proper fix.

Comment 7

[William Lachance (:wlach)]

Attachment: Pull request to add logging

Comment 8

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8458198 [details] [review]
Pull request to add logging

r=yurenju and your pr have some jshint error, I will fork your pr and fix it before merge.

Comment 9

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

merged.

https://github.com/mozilla-b2g/gaia/commit/3aa82ecc1fb1a1b410cb24b0f992cf80cfe85b33

Comment 10

[William Lachance (:wlach)]

Thanks Yuren! So we're now getting a bit of debugging information:

E.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=44126836&tree=B2g-Inbound&full=1 :

10:56:09     INFO -  [cmd] git --git-dir=/builds/slave/b2g_b2g-in_flame_eng_dep-00000/build/gaia/.git log -1 --format=%H%n%ct HEAD
10:56:09     INFO -  [settings-app-build] Error writing git commit file!
10:56:09     INFO -  stderr:
10:56:09     INFO -  undefined
10:56:09     INFO -  stdout: undefined

I did a bit of investigation and don't see why this wouldn't work. It's a very mysterious error. At least locally, if I try to run a command that we can't find (i.e. git wasn't in $PATH), I get a very obvious error. Likewise if I pass arguments to git that it doesn't understand. So I'm not sure what's going on. I think the best thing to do is to figure out the exact location of the git binary that we're trying to execute. I think that might give us a better hint about exactly what's happening here. I'll do up another pull request to get that information.

Also, I think we can say that we should not continue with the build if we get an error like this, in general. Filed bug 1040918 for that. :)

Comment 11

[William Lachance (:wlach)]

Attachment: Make it more clear what command we're actually executing (e.g. /usr/bin/git vs. just git)

Comment 12

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8458888 [details] [review]
Make it more clear what command we're actually executing (e.g. /usr/bin/git vs. just git)

r=yurenju,

https://github.com/mozilla-b2g/gaia/commit/cab9e0dd0dee93f31f12313d012e660f074663cf

Comment 13

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

try to clone B2G and build for flame but I got gaia_commit.txt in gaia/profile/settings.gaiamobile.org/application.zip.

Comment 14

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Request a loaner machine for this issue on bug 1039739.

Comment 15

[Zac C (:zac)]

Yuren, what is the ETA on this? It's affected our reporting and a bit of a nuisance. We can wait a day or two but if it's in the region of a week or more can we back out and re-land it with the fix?

Comment 16

[William Lachance (:wlach)]

We just ran a build with my patch applied. Here's what we're seeing on the Flame:

01:04:32     INFO -  [cmd] /usr/local/bin/git --git-dir=/builds/slave/b2g_b2g-in_flame_eng_dep-00000/build/gaia/.git log -1 --format=%H%n%ct HEAD
01:04:32     INFO -  [settings-app-build] Error writing git commit file!
01:04:32     INFO -  stderr:
01:04:32     INFO -  undefined
01:04:32     INFO -  stdout: undefined

I just did some testing and can replicate this error message from the gaia build in the following scenarios:

1. User does not have permission to access /usr/local/bin/git
2. /usr/local/bin/git is an empty file or otherwise non-executable

(git errors always produce at least some output)

On the Hamachi, we're seeing the same result with /usr/bin/git.

:aki, does this give any clues as to why this might be happening? Could you see any reason why the gaia build system might not have permission to access git?

Comment 17

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I think we can reproduce it on loaner machine, so let's back out commit on bug 1036821

Comment 18

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

backed out three commits:

[master cb854d0] Revert "Bug 1039739 - Make it more clear what command we're executing;r=yuren"

[master 00349ce] Revert "Bug 1039739 - Add some logging when writing the git commit id into the settings app;r=yurenju"

[master e92c1ce] Revert "Bug 1036821 - git commit hash isn't wrote to settings/resources/gaia_commit.txt on Windows"

:wlach, could you mark it as reolved if those revert commits can rosolve the problem?

Comment 19

[William Lachance (:wlach)]

(In reply to Yuren [:yurenju] from comment #18)
> backed out three commits:
> 
> [master cb854d0] Revert "Bug 1039739 - Make it more clear what command we're
> executing;r=yuren"
> 
> [master 00349ce] Revert "Bug 1039739 - Add some logging when writing the git
> commit id into the settings app;r=yurenju"
> 
> [master e92c1ce] Revert "Bug 1036821 - git commit hash isn't wrote to
> settings/resources/gaia_commit.txt on Windows"
> 
> :wlach, could you mark it as reolved if those revert commits can rosolve the
> problem?

Yes, this fixes the problem, I see the commit file in the latest versions of gaia. However, I don't think we should back out my changes as they provide information which might be useful in debugging either this or similar problems in the future and are almost certainly not the cause of the bug (since they were applied after we recognized bug 1036821 as the cause). Could you reapply them? I'm fine with resolving this and following up in bug 1036821 after that.

Comment 20

[William Lachance (:wlach)]

Let's follow up with aki's needinfo request in the original bug.

Comment 21

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Wlach, sure, I backed them out because your commits are top of my commit so back all of three commits out  is easy then extract my commit. I will re-land it if it can be applied to current settings/build/build.js

BTW, I'm leaving office, so feel free to apply it or I will do it tomorrow.

Comment 22

[Viorela Ioia [:viorela]]

This issue is fixed in the latest master build. The commit info is displayed in Settings app. 

Build info:
Gaia      cd50cb6ddeb430ddaf43e4fcf5774b9d262a8cf3
Gecko     https://hg.mozilla.org/mozilla-central/rev/e5ced39f443b
BuildID   20140722160202
Version   34.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014

Comment 23

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

mark as resolved since we have backout commit from bug 1036821.