Closed Bug 1186944 Opened 4 years ago Closed 4 years ago

[Aries][OTA] Unable to find latest build from OTA channel 'dogfood-latest'

Categories

(Taskcluster :: General, defect, P2)

ARM
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: onelson, Unassigned)

References

Details

(Keywords: qablocker, Whiteboard: [2.5-Daily-Testing] [Spark] [b2g-balrog])

Attachments

(1 file, 1 obsolete file)

Description:
Unable to find a latest build from Aries devices on build 20150722124026 through the 'dogfood-latest' update channel.
Tested looking for a channel from:
* Flash RC4, change channel 'dogfood-latest', updates from 20150629* => 20150722124026
* Confirm 'dogfood-latest' channel on 20150722124026, check for updates, reboot phone, check for updates on restart

Actual: 
No update can be found

Expected:
Find an update to reach build 20150723134401


Environmental Variables:
---------------------------
Build with no update:
Device: Aries 2.5
Build ID: 20150722124026
Gaia: b57aef5b7f52c40f88ee4c069ff722404e8e8521
Gecko: 954706e7611c
Gonk: 2916e2368074b5383c80bf5a0fba3fc83ba310bd
Version: 42.0a1 (2.5)
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:42.0) Gecko/42.0 Firefox/42.0


Latest available Aries on Balrog:
Device: Aries 2.5
BuildID: 20150723134401
Gaia: aa1698251e86c820c50c045b0a3ff65fd6b0eee7
Gecko: eee2d49d055c
Gonk: 2916e2368074b5383c80bf5a0fba3fc83ba310bd
Version: 42.0a1 (2.5) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:42.0) Gecko/42.0 Firefox/42.0
Link: https://tools.taskcluster.net/task-inspector/#Hwk9XzIqQd6OOfiP9_6VhQ/0
--------------------------
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(pbylenga)
Whiteboard: [2.5-Daily-Testing] → [2.5-Daily-Testing], [Spark]
[Blocking Requested - why for this release]:
This channel is important for QA to always test latest.
blocking-b2g: --- → 2.5?
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(pbylenga) → needinfo?(jlorenzo)
Keywords: qablocker
I just checked for updates a moment ago on the Z3/dogfood-latest and received an update to 20150723134401.
I reached the same conclusion as Marcia.
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(pbylenga)
Resolution: --- → WORKSFORME
Same thing for me. This link[1] pointed me to the same build as comment 2.

[1]
Status: RESOLVED → VERIFIED
Flags: needinfo?(jlorenzo)
Flags: needinfo?(pbylenga)
I think it has to do with whenever the tasks/builds/b2g_aries_spark_dogfood.yml script is run.

http://mxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/mozilla-central/job_flags.yml#20

The b2g build scripts for taskclusters can be found : 
http://mxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/

Taskcluster readme seems to give some help: http://mxr.mozilla.org/mozilla-central/source/testing/taskcluster/README.md 
It doesn't seem to say how to schedule the script running.

That's the line that creates a task:
http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/taskcluster_helper.py#40

Looks like Taskcluster does use buildbot?

Side note: 
https://hg.mozilla.org/build/buildbot-configs/rev/2d745da0f04f
https://hg.mozilla.org/build/buildbot-configs/file/e7176ef993ef/mozilla/b2g_config.py
That's the config for bug 962363; aries is not listed here.  This is for the pvtbuild server.

Buildbot platforms that are built: 
http://mxr.mozilla.org/build/source/tools/lib/python/release/platforms.py#45

Buildbot manifest files:
http://mxr.mozilla.org/build/source/mozharness/configs/b2g_bumper/master.py#73

There's a DB:
http://mxr.mozilla.org/build/source/braindump/buildbot-related/get_build_times_for_builders.py#47
http://mxr.mozilla.org/build/source/buildbot/master/buildbot/util/collections.py

There's actually a bridge for buildbot and taskcluster:
http://hearsum.ca/blog/buildbot-taskcluster-bridge-now-in-production.html
They become tasks on the tasksgraph

http://hearsum.ca/blog/buildbot-taskcluster-bridge-an-overview.html
Slides are available here : http://hearsum.ca/slides/buildbot-bridge-overview/#/

Based on all this... shouldn't we be able to use buildbot to get builds approximately at the time we normally get pvtbuilds?
Status: VERIFIED → REOPENED
blocking-b2g: 2.5? → ---
Product: Firefox OS → Taskcluster
Resolution: WORKSFORME → ---
Note: I reopened this because we're not seeing builds at the times we need to see the builds for aries.
Can I ask for your help here in getting aries builds based on the buildbot script at the times that we need them?  We need them roughly at the same time we have the flame builds.

If not, can you point me to someone who can help with this please?
Flags: needinfo?(bhearsum)
As far as I know this is 100% B2G/Taskcluster Automation...Wander is the go to person for that.
Flags: needinfo?(bhearsum) → needinfo?(wcosta)
dogfood-latest seem messy because it is conflicting with normal aries-ota builds. I mean, the BUILD_TARGET and mapping are the same. So, dogfood-latest is grabbing aries-ota updates and vice-versa.
Flags: needinfo?(wcosta)
The blob name is the bridge between balrog rules and balrog releases.
The current pattern for the blob name is
<product>-<branch>-<nightly/release>-<buildid/latest>, which causes
conflicts in case of different builds for the same target. One
particular example is Aries dogfood builds, which share the same blob
name with nightly (ota) builds.

Here we allow the customization of the blob name through the environment
variable BALROG_BLOB_NAME. If it is defined, its content is used as the
blob name.

You can use python formatters style for replacement of the following
builtin values:
Nightly: PRODUCT, BUILDID, BRANCH and BUILD_TYPE.
Release: PRODUCT, VERSION and BUILD_NUMBER.

Example:
export BALROG_BLOB_NAME="{PRODUCT}-{BRANCH}-{BUILD_TYPE}-custom-{BUILD_ID}"
Comment on attachment 8642170 [details] [diff] [review]
Custom blob name through environment variable.

This is a slightly different solution from what we talked about in Vidyo. Instead of a command line option, we pass the custom blob name through the environment. This way we avoid patching all layers between Taskcluster and balrog cli.
Attachment #8642170 - Flags: review?(bhearsum)
Attachment #8642170 - Attachment is obsolete: true
Attachment #8642170 - Flags: review?(bhearsum)
The blob name is the bridge between balrog rules and balrog releases.
The current pattern for the blob name is
<product>-<branch>-<nightly/release>-<buildid/latest>, which causes
conflicts in case of different builds for the same target. One
particular example is Aries dogfood builds, which share the same blob
name with nightly (ota) builds.

Here we allow the customization of the blob name through the environment
variable BALROG_BLOB_NAME. If it is defined, its content is used as the
blob name.

You can use python formatters style for replacement of the following
builtin values:
Nightly: PRODUCT, BUILDID, BRANCH and BUILD_TYPE.
Release: PRODUCT, VERSION and BUILD_NUMBER.

Example:
export BALROG_BLOB_NAME="{PRODUCT}-{BRANCH}-{BUILD_TYPE}-custom-{BUILD_ID}"
Comment on attachment 8642172 [details] [diff] [review]
Custom blob name through environment variable.

This is a slightly different solution from what we talked about in Vidyo. Instead of a command line option, we pass the custom blob name through the environment. This way we avoid patching all layers between Taskcluster and balrog cli.
Attachment #8642172 - Flags: review?(bhearsum)
Blocks: 1189408
No longer depends on: 1189408
Comment on attachment 8642172 [details] [diff] [review]
Custom blob name through environment variable.

Review of attachment 8642172 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry, I have to r- this. I know it's a pain, but this code should not be looking directly at the environment. The defined way to get information to this class is through the constructor, via the submitter script's args. I'm fine with adding a full override for release name, and having the replacements you mention, but it must be passed via the submitter script's args.
Attachment #8642172 - Flags: review?(bhearsum) → review-
Whiteboard: [2.5-Daily-Testing], [Spark] → [2.5-Daily-Testing], [Spark], [tc-build-support]
Whiteboard: [2.5-Daily-Testing], [Spark], [tc-build-support] → [2.5-Daily-Testing], [Spark], [b2g-build-support]
Priority: -- → P2
Whiteboard: [2.5-Daily-Testing], [Spark], [b2g-build-support] → [2.5-Daily-Testing], [Spark], [b2g-build-support][dogfood blocker]
Whiteboard: [2.5-Daily-Testing], [Spark], [b2g-build-support][dogfood blocker] → [2.5-Daily-Testing], [Spark], [b2g-build-support][dogfood-blocker]
I think with the b2g-ota and we move all the dogfood builds to this branch, we shouldn't need to do this.  Having said that, if we want a different channel for the same branch build in the future, we should continue to investigate this path.
Whiteboard: [2.5-Daily-Testing], [Spark], [b2g-build-support][dogfood-blocker] → [2.5-Daily-Testing], [Spark], [b2g-build-support]
Whiteboard: [2.5-Daily-Testing], [Spark], [b2g-build-support] → [2.5-Daily-Testing] [Spark] [b2g-balrog]
Switched dogfood-latest to use B2G-b2g-ota-nightly-latest blob.
At the same time, it's hard to say if it's needed.  

For the future though, do we want two or more different channels on the same release blob?  If a new dogfooding program comes up for desktop or fennec or something, we may face this issue again.
Fox fooding program to end for phone : 1248791
We will probably need to switch people to nightly.  We're switching people off from dogfood=1 to dogfood=0.
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → WONTFIX
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #19)
> Fox fooding program to end for phone : 1248791
> We will probably need to switch people to nightly.  We're switching people
> off from dogfood=1 to dogfood=0.

Naoki, will this be automated or should we change the update channel ourselves?
Flags: needinfo?(nhirata.bugzilla)
FOTA for internal Mozilla employees and OTA for outside people have already been set.
It should be through our update server.
Flags: needinfo?(nhirata.bugzilla)
You need to log in before you can comment on or make changes to this bug.