1134226 - Provide nightly builds of Ignite for Aries device

Status: RESOLVED FIXED

(Taskcluster :: General, defect, P1)

Description

[Wander Lairson Costa]

Aries device will be used for v3 dogfooding. Provide builds and OTA updates for it.

Comment 1

[Doug Sherk (:drs) (inactive)]

Wander and I talked about this on IRC for a while. We're facing a problem because the lightsaber build [1] process has some extra steps, using some node modules, which mozharness and the B2G repo [2] don't have support for. [3]

The way we see it, we have two options to solve this:

1. We overwrite the B2G repo's copy of Gaia with the un-built Lightsaber Gaia branch + pre-built custom apps, then run |B2G: ./build.sh| afterwards. We would provide a script that does this overwriting and pre-building. This is more work on mozharness and less for us.
2. We fork the B2G repo and move all of our custom build processes to the manifest, |./build.sh| script, etc. This is more work for Lightsaber people (probably me) as it will require basically redoing our build process and also maintaining a fork of the B2G repo.

We agreed that maintaining a fork of B2G sounds like a really bad idea. Thus we're going to do option 1.

I'm going to write this script to do the pre-building and overriding and provide it to Wander, who is working on adding support in mozharness for adding a script between the fetch->build steps.

[1] https://github.com/fxos/lightsaber
[2] https://github.com/mozilla-b2g/B2G
[3] https://github.com/fxos/lightsaber/blob/master/Makefile#L3-L18

Comment 2

[Doug Sherk (:drs) (inactive)]

Also note that we will need added support to mozharness for setting prefs. [1] Wander says that he will work on this as well. Perhaps we should file a separate bug for this.

[1] https://github.com/fxos/lightsaber#set-the-javascriptoptionsdiscardsystemsource-pref

Comment 3

[Wander Lairson Costa]

Attachment: MozReview Request: bz://1134226/wcosta

/r/6559 - Bug 1134226: Add lightsaber nightly builds.

Pull down this commit:

hg pull -r e08708ab6ea7212c4cbb3d21d64e0941e6af3be5 https://reviewboard-hg.mozilla.org/gecko/

Comment 4

[Greg Arndt [:garndt]]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

https://reviewboard.mozilla.org/r/6557/#review5445

::: testing/taskcluster/scripts/phone-builder/build-lightsaber-nightly.sh
(Diff revision 1)
> +  --complete-mar-url https://queue.taskcluster.net/v1/task/$TASK_ID/runs/0/artifacts/public/build/$mar_file \

So this is pulling the artifact from the first run (0), what happens if this run fails? Admittedly I see this issue in the other phone builder scripts as well and not something new that was introduced here.

::: testing/taskcluster/scripts/phone-builder/build-lightsaber-nightly.sh
(Diff revision 1)
> +  --config balrog/staging.py \

When would this be changed to prod? When the vpn proxy is setup? (I'm still working on that and hopeuflly will have that soon)

::: testing/taskcluster/scripts/phone-builder/build-lightsaber-nightly.sh
(Diff revision 1)
> +./mozharness/scripts/b2g_lightsaber.py \

Does this clone gaia at all?  I know that there were some changes elsewhere to mozharness commands to specify something like "no-pull" and let tc-vcs pull gaia.

::: testing/taskcluster/tasks/builds/b2g_aries_lightsaber_nightly.yml
(Diff revision 1)
> +  workerType: flame-kk

Is there any value in moving this into it's own "lightsaber" worker type?

::: testing/docker/phone-builder/Dockerfile
(Diff revision 1)
> +RUN           pip install awscli

So each one of these lines starting with "RUN" will create a layer inthe image.  Would it be better to either combine them all in one line "npm install -g bower gulp ..." or create a package.json that will get installed in one layer?

Comment 5

[Greg Arndt [:garndt]]

Flag me again for review once we sort out the few questions in comment 4

Comment 6

[Wander Lairson Costa]

https://reviewboard.mozilla.org/r/6557/#review5459

> Does this clone gaia at all?  I know that there were some changes elsewhere to mozharness commands to specify something like "no-pull" and let tc-vcs pull gaia.

mozharness itself doesn't, but config.sh does. But this is useless, because lightsaber will clone its own gaia branch.

> When would this be changed to prod? When the vpn proxy is setup? (I'm still working on that and hopeuflly will have that soon)

Yep.

> So this is pulling the artifact from the first run (0), what happens if this run fails? Admittedly I see this issue in the other phone builder scripts as well and not something new that was introduced here.

Nop, this is used when submiting to balrog. You tell balrog where to find the generated mar file.

> Is there any value in moving this into it's own "lightsaber" worker type?

Sooner or later, lightsaber will be merged into gaia (I guess) and we won't need specific stuff to build it. So when this time come, having things in separate files will make the cleanup easier.

Comment 7

[Wander Lairson Costa]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

/r/6559 - Bug 1134226: Add lightsaber nightly builds.

Pull down this commit:

hg pull -r 5a44195264d5ce6be261ef57152b5676b300d2d0 https://reviewboard-hg.mozilla.org/gecko/

Comment 8

[Greg Arndt [:garndt]]

https://reviewboard.mozilla.org/r/6557/#review5469

> mozharness itself doesn't, but config.sh does. But this is useless, because lightsaber will clone its own gaia branch.

So is there no value in having tc-vcs do this so that it's cached?

> Nop, this is used when submiting to balrog. You tell balrog where to find the generated mar file.

But if run 0 failed, there will be no mar file....

Comment 9

[Wander Lairson Costa]

https://reviewboard.mozilla.org/r/6557/#review5471

> So is there no value in having tc-vcs do this so that it's cached?

Nop.

Comment 10

[Greg Arndt [:garndt]]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

https://reviewboard.mozilla.org/r/6557/#review5473

::: testing/taskcluster/scripts/phone-builder/build-lightsaber-nightly.sh
(Diff revision 2)
> +  --complete-mar-url https://queue.taskcluster.net/v1/task/$TASK_ID/runs/0/artifacts/public/build/$mar_file \

So here we might be able to do something similar to: 
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/mach_commands.py#35

Where if you call this endpoint without a runId, it will grab the latest artifact for that task.
https://github.com/taskcluster/taskcluster-queue/blob/master/routes/artifacts.js#L428

I think this might ensure that if for some reason run 0 was aborted (worker-shutdown, etc) that when run 1 succeeds, this task will know where to find the artifact for the successful run (run 1) instead of hard code to 0.

Comment 11

[Wander Lairson Costa]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

/r/6559 - Bug 1134226: Add lightsaber nightly builds.

Pull down this commit:

hg pull -r f8bb075117a6b89e3ecf5cb41b3dfad6f6fe3caf https://reviewboard-hg.mozilla.org/gecko/

Comment 12

[Dale Harvey (:daleharvey)]

Just to be clear, when the nightly builds are enabled am I going to get a normal build or a lightsaber one?

Comment 13

[Wander Lairson Costa]

(In reply to Dale Harvey (:daleharvey) from comment #12)
> Just to be clear, when the nightly builds are enabled am I going to get a
> normal build or a lightsaber one?

Lightsaber, but if you need normal builds, just file a bug and I can provide you.

Comment 14

[Greg Arndt [:garndt]]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

r+ provided a link to a successful aries build using this configuration is added to the bug.

Comment 15

[Wander Lairson Costa]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

/r/6559 - Bug 1134226: Add lightsaber nightly builds.

Pull down this commit:

hg pull -r b4749202f328bb2375f45ac300912348c1afb3bb https://reviewboard-hg.mozilla.org/gecko/

Comment 16

[Wander Lairson Costa]

(In reply to Greg Arndt [:garndt] from comment #14)
> Comment on attachment 8587598 [details]
> MozReview Request: bz://1134226/wcosta
> 
> r+ provided a link to a successful aries build using this configuration is
> added to the bug.

I needed apply some fixes to the original patch. Would you mind reviewing again?

Comment 17

[Greg Arndt [:garndt]]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

https://reviewboard.mozilla.org/r/6557/#review5719

::: testing/taskcluster/scripts/phone-builder/build-lightsaber-nightly.sh
(Diff revision 4)
> +#mv $WORKSPACE/B2G/upload/b2g-*.crashreporter-symbols.zip $HOME/artifacts/b2g-crashreporter-symbols.zip

Do we not need crashreporter symbols?  There was some work that James was just doing to make sure our builds have them.  Didn't know if this was the case for nightlies.

::: testing/taskcluster/scripts/phone-builder/pre-build.sh
(Diff revision 4)
> +rm -rf $WORKSPACE/B2G/objdir*

So it appears that we are using device specific caches in the task (build-aries-lightsaber-nightly) but we're going to remove obj caches for every device?  Aries nightly builds can't use cached obj dirs?  Interested to know what the problem might be.

Also, if the aires nightly is the only place we want to blow away the cache, maybe there is an ENV flag we set in the nightly task that the pre-build script looks for before blowing it away.

Comment 18

[Greg Arndt [:garndt]]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

r+ with the comment about caches being addressed.  Would not like to see the caches removed for all phone builds if not necessary.

Also, could you add a comment with a taskcluster task using this new diff?

Comment 19

[Wander Lairson Costa]

https://reviewboard.mozilla.org/r/6557/#review5721

> Do we not need crashreporter symbols?  There was some work that James was just doing to make sure our builds have them.  Didn't know if this was the case for nightlies.

This is because mozharness tries to upload symbols in nightly builds, and this case the build would fail. Bug 1148630 will address this issue.

> So it appears that we are using device specific caches in the task (build-aries-lightsaber-nightly) but we're going to remove obj caches for every device?  Aries nightly builds can't use cached obj dirs?  Interested to know what the problem might be.
> 
> Also, if the aires nightly is the only place we want to blow away the cache, maybe there is an ENV flag we set in the nightly task that the pre-build script looks for before blowing it away.

I was reported that caching aries binaries causes wifi issues. I decided to play safe and not caching binaries for all devices, because I don't know if there are issues for other devices (current or future). Not sure what is the better approach, however.

Comment 20

[Wander Lairson Costa]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

/r/6559 - Bug 1134226: Add lightsaber nightly builds.

Pull down this commit:

hg pull -r 264e390b0eb0abb87bce410ceb44e56a8ef8940c https://reviewboard-hg.mozilla.org/gecko/

Comment 21

[Wander Lairson Costa]

https://tools.taskcluster.net/task-inspector/#liWZvKreR6O94KHpoaKLQw/1

https://treeherder.allizom.org/#/jobs?repo=try&revision=f479ca82da9c

Comment 22

[Greg Arndt [:garndt]]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

https://reviewboard.mozilla.org/r/6557/#review5751

::: testing/taskcluster/scripts/phone-builder/build-lightsaber-nightly.sh
(Diff revision 5)
> +  rm -rf $WORKSPACE/B2G/objdir*

I think this would be ok, if there is no other way we could accomplish it.  Is there a way you can tell mozharness to clobber things?  does that take care of both directories?

Comment 23

[Wander Lairson Costa]

https://reviewboard.mozilla.org/r/6557/#review5775

> I think this would be ok, if there is no other way we could accomplish it.  Is there a way you can tell mozharness to clobber things?  does that take care of both directories?

mozharness has clobbering stuff, but I gave it a try and it doesn't work as expected, it will delete the whole tree.

Comment 24

[Wander Lairson Costa]

Comment on attachment 8587598 [details]
MozReview Request: bz://1134226/wcosta

/r/6559 - Bug 1134226: Add lightsaber nightly builds.

Pull down this commit:

hg pull -r 252ce8ca5a9b75e33d69e05f7ac45149e3a9d020 https://reviewboard-hg.mozilla.org/gecko/

Comment 25

[Wander Lairson Costa]

Added nightly to cypress branch.

Comment 26

[Wander Lairson Costa]

Also configured indexing.

Comment 27

[Wander Lairson Costa]

https://hg.mozilla.org/integration/b2g-inbound/rev/c37edf9eaedc

Comment 28

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c37edf9eaedc

Comment 30

[Wander Lairson Costa]

Attachment: MozReview Request: Bug 1134226: Add lightsaber nightly builds.

Comment 31

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...