1224747 - [Aries] Mar files have different build ID than full flash

Status: VERIFIED FIXED

(Taskcluster :: General, defect)

Description

[Adam Alldredge [:AdamA][Inactive]]

Description:
On Aries when the user full flashes the build they will have a different build ID than when they OTA the build using a mar file. These builds have the same gecko and gaia and are the same builds in all respects except for the build ID. this occurs when the mar file and the full flash file are pulled from the same task on taskcluster.

Repro Steps:
1) Go to a taskcluster Aries Task and pull down the Full flash and Mar files. (Example: https://tools.taskcluster.net/task-inspector/#F-hLXBKfRdORNTCoM_RHIw/)
2) Full flash the build using the Aries.zip pulled from taskcluster.
3) Pull the build variables from the terminal
4) OTA the Aries build using the b2g-aries-gecko-update.mar on an update server.
5) Pull the build variables from the terminal
6) Compare the Build Id's

Result:
the builds have different build ID's but were made in the same task

Expected:
It is expected that builds made in the same task have the same build ID's.

Full Flash Variables:
Device: Aries 2.6 [Full Flash]
BuildID: 20151113123209
Gaia: e8c15ae4e5324a210000ee0a869a962aa542009f
Gecko: faf815a0fa9b052a38bce00c0c2aa1e2c9610936
Gonk: a19052e4389c3ae2d8fc3e7a74a475401baacc56
Version: 45.0a1 (2.6) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:45.0) Gecko/45.0 Firefox/45.0

OTA variables:
Device: Aries 2.6 [Full Flash]
BuildID: 220151113121953
Gaia: e8c15ae4e5324a210000ee0a869a962aa542009f
Gecko: faf815a0fa9b052a38bce00c0c2aa1e2c9610936
Gonk: a19052e4389c3ae2d8fc3e7a74a475401baacc56
Version: 45.0a1 (2.6) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:45.0) Gecko/45.0 Firefox/45.0

Repro frequency: 5/5

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

This might be expected.  The mar file is built at a separate time from the full build.

I'm not certain we want to fix that.

Comment 2

[Chris AtLee [:catlee]]

IMO, it should be fixed. It's pretty important that people end up with the same bits (including buildid) whether they're installing fresh, or getting updated from a previous build.

It also simplifies finding which build was responsible for creating which artifacts if there's only one buildid used for all the package formats.

I suspect this is due to not setting MOZ_BUILD_DATE anywhere in the build environment.

Comment 3

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Ok, that's a valid point.  Let's try to get this fixed.

Comment 5

[Chris AtLee [:catlee]]

I believe that this also impacts the various types of MAR files that are generated - they all seem to have different buildids.

Comment 6

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

We'll probably need to capture the ID prior to all builds being built set the BUILDID to that and push it in all the builds for that task.

Seems like this might be more of a taskcluster task issue... moving this to taskcluster.

Comment 7

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

fyi for catlee that I moved this task.

Comment 8

[Chris AtLee [:catlee]]

I think the right way to fix this is to have the decision task set MOZ_BUILD_DATE in the environment based on the pushdate.

Comment 9

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Agreed.  Mahe, Jean, we'll need someone to be assigned to this bug.

(+johan as a fyi of one of the issues that needs to be fixed; this will fix the issue with the build id for the mar and the full flash being different )

Comment 10

[Johan Lorenzo [:jlorenzo]]

FYI received.

Comment 11

[Greg Arndt [:garndt]]

If "%Y%m%d%H%M%S" is a suitable format, we already make "pushdate" available as a template parameter[1] so the task definition for building Aries would just need an environment variable set to this.

such as:
task:
  payload:
    env:
      MOZ_BUILD_DATE: '{{pushdate}}'


[1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/mach_commands.py#343

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

so I tried the comment, and the pushdate comes out blank : 
https://github.com/nhirata/gecko-dev/commit/174664ab1712c4302746479411b3c23a88bd7cf6

Should I be modifying : gecko-dev/testing/taskcluster/tasks/builds/b2g_phone_base.yml and b2g_phone_eng_base.yml instead?

Comment 13

[Greg Arndt [:garndt]]

Hrm, I added your changes, and I was able to run taskcluster-graph and it added it as I would expect

I ran:
$ mach taskcluster-graph --project='mozilla-central' --message='some message' --owner='[e-mail]' --head-repository https://hg.mozilla.org/mozilla-central --head-rev tip

and it produced this:
https://gist.githubusercontent.com/gregarndt/2d56d0cc080f275cb781/raw/6d0e74e2a135a704a753e4ee5c4c20aa7dcaba9e/mozilla-central%2520graph

Which has moz_build_date:
"MOZ_BUILD_DATE": "20151216110158"


And the diff:

diff --git a/testing/taskcluster/tasks/phone_build.yml b/testing/taskcluster/tasks/phone_build.yml
--- a/testing/taskcluster/tasks/phone_build.yml
+++ b/testing/taskcluster/tasks/phone_build.yml
@@ -45,16 +45,17 @@ task:
       'public/build':
         type: directory
         path: '/home/worker/artifacts-public/'
         expires: '{{#from_now}}1 year{{/from_now}}'


     env:
       # Common environment variables for checking out gecko
+      MOZ_BUILD_DATE: '{{pushdate}}'
       GECKO_BASE_REPOSITORY: '{{base_repository}}'
       GECKO_HEAD_REPOSITORY: '{{head_repository}}'
       GECKO_HEAD_REV: '{{head_rev}}'
       GECKO_HEAD_REF: '{{head_ref}}'

   extra:
     build_product: 'b2g'
     build_name: '{{build_name}}'
diff --git a/testing/taskcluster/tasks/phone_test.yml b/testing/taskcluster/tasks/phone_test.yml
--- a/testing/taskcluster/tasks/phone_test.yml
+++ b/testing/taskcluster/tasks/phone_test.yml
@@ -18,16 +18,17 @@ task:
     - 'docker-worker:capability:device:phone'

   payload:
     image: '{{#docker_image}}tester-device{{/docker_image}}'
     maxRunTime: 3600
     cache:
       tc-vcs: '/home/worker/.tc-vcs'
     env:
+      MOZ_BUILD_DATE: '{{pushdate}}'
       GAIA_HEAD_REPOSITORY: '{{{gaia_head_repository}}}'
       GAIA_BASE_REPOSITORY: '{{{gaia_base_repository}}}'
       GAIA_REF: '{{{gaia_ref}}}'
       GAIA_REV: '{{{gaia_rev}}}'

     # All builds share a common artifact directory for ease of uploading.
     artifacts:
       'private/logs':

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: bug_1224747_fix_buildids.patch

I had been using this format : 
./mach taskcluster-build --head-repo=https://github.com/nhirata/gecko-dev --head-rev=bug_1224747_fix_buildids --owner=[e-mail] tasks/builds/b2g_aries_spark_debug.yml | sed -e 's/0\.0\.22/0.99.22/'

in order to make builds.  I guess it needs to go through taskcluster-graphs in order to get the pushdate?  Is there another way to make changes to a repo other than using a gecko-dev branch that's better than the way I described?

Comment 15

[Greg Arndt [:garndt]]

currently the querying of pushinfo comes from taskcluster-graph....that stuff could be ported over into taskcluster-build as well.  Typically taskcluster-build has just been used for debugging stuff, but if there is a need to give it more love, then a bug could be opened detailing the use cases.

Comment 16

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Ok thanks for the info.  I think I'll withhold for now.  There are other pressing matters that need to be resolved.  I can make due with my hacks.

Thanks for the review+.

Comment 17

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Does checkin-needed get looked at for taskcluster ?
Is there a bot for that?  or does it need to manually get done?

Comment 18

[Ryan VanderMeulen [:RyanVM]]

checkin-needed requests are handled manually, and they typically only look at core components. Probably worth pinging a sheriff to get it landed for you (which I haven't been since early September).

Comment 19

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/integration/b2g-inbound/rev/3a4b329934ab

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3a4b329934ab

Comment 21

[Martin Shuman [:Marty]]

This issue is verified Fixed. The build ID for the mar file and full flash have matched for the last two Aries daily smoketests.

Task ID: Js1EOZ6ARA2QF9r8vIqp2Q
Build ID: 20160105105910

Task ID: KJmBJVG8Q2yayv4qwDFDxA
Build ID: 20160104105544

Comment 22

[Jean Gong :jgong]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #9)
> Agreed.  Mahe, Jean, we'll need someone to be assigned to this bug.
> 
> (+johan as a fyi of one of the issues that needs to be fixed; this will fix
> the issue with the build id for the mar and the full flash being different )

This program is closed.