Closed Bug 1297585 Opened 3 years ago Closed 3 years ago

Failure while building URL for Fennec artifact build

Categories

(Firefox Build System :: General, defect)

ARM
Android
defect
Not set

Tracking

(firefox51 fixed)

RESOLVED DUPLICATE of bug 1306614
mozilla51
Tracking Status
firefox51 --- fixed

People

(Reporter: atsai, Assigned: atsai)

References

Details

Attachments

(1 file, 3 obsolete files)

Environment: Mac OS X


mozconfig:
# Build Firefox for Android Artifact Mode:
ac_add_options --enable-application=mobile/android
ac_add_options --target=arm-linux-androideabi
ac_add_options --enable-artifact-builds

# With the following Android SDK:
ac_add_options --with-android-sdk="/Users/atsai/.mozbuild/android-sdk-macosx"

# Write build artifacts to:
mk_add_options MOZ_OBJDIR=./objdir-frontend



Error Message:
 0:34.77 Retrieving the last 50 pushheads starting with id 80700 on integration/mozilla-inbound
 0:34.77 Retrieving the last 50 pushheads starting with id 20333 on integration/fx-team
 0:34.77 Retrieving the last 50 pushheads starting with id 30571 on mozilla-central
 0:38.27 Error running mach:
 0:38.27
 0:38.27     ['--log-no-times', 'artifact', 'install']
 0:38.27
 0:38.27 The error occurred in code that was called by the mach command. This is either
 0:38.27 a bug in the called code itself or in the way that mach is calling it.
 0:38.27
 0:38.27 You should consider filing a bug for this issue.
 0:38.27
 0:38.27 If filing a bug, please include the full output of mach, including this error
 0:38.27 message.
 0:38.27
 0:38.28 The details of the failure are as follows:
 0:38.28
 0:38.28 AttributeError: 'unicode' object has no attribute 'iteritems'
 0:38.28
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 1513, in artifact_install
 0:38.28     return artifacts.install_from(source, self.distdir)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1037, in install_from
 0:38.28     return self.install_from_recent(distdir)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1000, in install_from_recent
 0:38.28     return self._install_from_hg_pushheads(hg_pushheads, distdir)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 986, in _install_from_hg_pushheads
 0:38.28     self._job, hg_hash)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 902, in find_pushhead_artifacts
 0:38.28     urls = task_cache.artifact_urls(tree, job, pushhead)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 437, in wrapper
 0:38.28     result = method(self, *args, **kwargs)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 626, in artifact_urls
 0:38.28     url = self._queue.buildUrl('getLatestArtifact', taskId, artifact_name)
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/objdir-frontend/_virtualenv/lib/python2.7/site-packages/taskcluster/baseclient.py", line 177, in buildUrl
 0:38.28     for key, value in six.iteritems(replDict):
 0:38.28   File "/Users/atsai/workspace/src/mozilla-central/objdir-frontend/_virtualenv/lib/python2.7/site-packages/six.py", line 599, in iteritems
 0:38.28     return d.iteritems(**kw)
 0:38.30 make[4]: *** [recurse_artifact] Error 1
 0:38.31 make[3]: *** [artifact] Error 2
 0:38.31 make[2]: *** [default] Error 2
 0:38.31 make[1]: *** [realbuild] Error 2
 0:38.31 make: *** [build] Error 2
 0:38.34 0 compiler warnings present.
Summary: Build Artifact build failed with Mac OS X → Failure while building URL for Fennec artifact build
In the beginning, I thought it's because the return format. Therefore, to avoid unicode issues, I modified six.py Line 599 with:

>    def iteritems(d, **kw):
>        import ast
>        return d.iteritems(ast.literal_eval(**kw))

I got:

» ./mach build                                                                                           [2016-08-25 17:36:08] atsai@Al-MBP
 0:00.21 /usr/bin/make -f client.mk -s
 0:01.17 Adding client.mk options from /Users/atsai/workspace/src/mozilla-central/mozconfig:
 0:01.17     CONFIG_GUESS=arm-linux-androideabi
 0:01.17     MOZ_OBJDIR=/Users/atsai/workspace/src/mozilla-central/objdir-frontend
 0:01.17     OBJDIR=/Users/atsai/workspace/src/mozilla-central/objdir-frontend
 0:01.17     FOUND_MOZCONFIG=/Users/atsai/workspace/src/mozilla-central/mozconfig
 0:02.43 Attempting to find a pushhead containing 01748a2b1a463f24efd9cd8abad9ccfd76b037b8 on mozilla-central.
 0:02.43 Attempting to find a pushhead containing 01748a2b1a463f24efd9cd8abad9ccfd76b037b8 on integration/fx-team.
 0:02.43 Attempting to find a pushhead containing 01748a2b1a463f24efd9cd8abad9ccfd76b037b8 on integration/mozilla-inbound.
 0:02.43 Attempting to find a pushhead containing 01748a2b1a463f24efd9cd8abad9ccfd76b037b8 on releases/mozilla-aurora.
 0:03.56 Retrieving the last 50 pushheads starting with id 80992 on integration/mozilla-inbound
 0:03.56 Retrieving the last 50 pushheads starting with id 20384 on integration/fx-team
 0:03.56 Retrieving the last 50 pushheads starting with id 30597 on mozilla-central
 0:03.58 Tried 40 pushheads, no built artifacts found.
 0:03.64 make[4]: *** [recurse_artifact] Error 1
 0:03.64 make[3]: *** [artifact] Error 2
 0:03.64 make[2]: *** [default] Error 2
 0:03.65 make[1]: *** [realbuild] Error 2
 0:03.65 make: *** [build] Error 2
 0:03.70 0 compiler warnings present.

Given the fact that even the unicode string is returned, we cannot find a valid artifacts to use. I'll check if we have any valid artifact build on public.mozilla.org first.
This comment could be an invalid message.
taskId: Z7xOFOOISSmkJ6qI_KjkPw
artifact name: public/build/fennec-51.0a1.en-US.android-arm.apk

https://tools.taskcluster.net/task-inspector/#Z7xOFOOISSmkJ6qI_KjkPw/0

Currently, goal is to find out this file. However, we have trouble in composing URL
Attached patch build_script.patch (obsolete) — Splinter Review
Attachment #8785201 - Flags: review?(mh+mozilla)
Comment on attachment 8785201 [details] [diff] [review]
build_script.patch

Review of attachment 8785201 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/artifacts.py
@@ +637,5 @@
>              # We can easily extract the task ID from the URL.  We can't easily
>              # extract the build ID; we use the .ini files embedded in the
>              # downloaded artifact for this.  We could also use the uploaded
>              # public/build/buildprops.json for this purpose.
> +            BASE_URL = 'https://queue.taskcluster.net/v1/task/{taskId}/runs/0/artifacts/{artifact_name}'

You're not using this variable.

@@ -637,5 @@
>              # We can easily extract the task ID from the URL.  We can't easily
>              # extract the build ID; we use the .ini files embedded in the
>              # downloaded artifact for this.  We could also use the uploaded
>              # public/build/buildprops.json for this purpose.
> -            url = self._queue.buildUrl('getLatestArtifact', taskId, artifact_name)

There is another call to buildUrl in testing/docker/rust-build/tcbuild.py that has the same problem. Please change it too.
Attachment #8785201 - Flags: review?(mh+mozilla)
Attached patch build_script.patch (obsolete) — Splinter Review
Remove redundant line. fix buildURL issues for testing/docker/rust-build/tcbuild.py as well.
Attachment #8785201 - Attachment is obsolete: true
Attachment #8787078 - Flags: review?(mh+mozilla)
Assignee: nobody → atsai
Attachment #8787078 - Flags: review?(mh+mozilla) → review+
Keywords: checkin-needed
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d392d81c5b9e
Modify parameters for buildUrl to satisfy taskcluster changes. r=glandium
Keywords: checkin-needed
busted on inbound

02:30:53     INFO -  Error running mach:
02:30:53     INFO -      ['--log-no-times', 'artifact', 'install']
02:30:53     INFO -  The error occurred in code that was called by the mach command. This is either
02:30:53     INFO -  a bug in the called code itself or in the way that mach is calling it.
02:30:53     INFO -  You should consider filing a bug for this issue.
02:30:53     INFO -  If filing a bug, please include the full output of mach, including this error
02:30:53     INFO -  message.
02:30:53     INFO -  The details of the failure are as follows:
02:30:53     INFO -  TaskclusterFailure: Arguments "{u'name': u'public/build/firefox-51.0a1.en-US.linux-x86_64.common.tests.zip', u'taskId': u'aUFKQMlSRgeVtT62ReYVcQ'}" to getLatestArtifact is not a string or int
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1516, in artifact_install
02:30:53     INFO -      return artifacts.install_from(source, self.distdir)
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/artifacts.py", line 1053, in install_from
02:30:53     INFO -      return self.install_from_recent(distdir)
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/artifacts.py", line 1016, in install_from_recent
02:30:53     INFO -      return self._install_from_hg_pushheads(hg_pushheads, distdir)
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/artifacts.py", line 1002, in _install_from_hg_pushheads
02:30:53     INFO -      self._job, hg_hash)
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/artifacts.py", line 918, in find_pushhead_artifacts
02:30:53     INFO -      urls = task_cache.artifact_urls(tree, job, pushhead)
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/artifacts.py", line 452, in wrapper
02:30:53     INFO -      result = method(self, *args, **kwargs)
02:30:53     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/artifacts.py", line 642, in artifact_urls
02:30:53     INFO -      url = self._queue.buildUrl('getLatestArtifact', replDict=replDict)
02:30:53     INFO -    File "/home/worker/workspace/build/src/obj-firefox/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 149, in buildUrl
02:30:53     INFO -      apiArgs = self._processArgs(entry, *args, **kwargs)
02:30:53     INFO -    File "/home/worker/workspace/build/src/obj-firefox/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 249, in _processArgs
02:30:53     INFO -      raise exceptions.TaskclusterFailure('Arguments "%s" to %s is not a string or int' % (arg, entry['name']))
02:30:54     INFO -  gmake[4]: *** [recurse_artifact] Error 1
02:30:54     INFO -  gmake[4]: Leaving directory `/home/worker/workspace/build/src/obj-firefox'
02:30:54     INFO -  gmake[3]: *** [artifact] Error 2
02:30:54     INFO -  gmake[3]: Leaving directory `/home/worker/workspace/build/src/obj-firefox'
02:30:54     INFO -  gmake[2]: *** [default] Error 2
02:30:54     INFO -  gmake[2]: Leaving directory `/home/worker/workspace/build/src/obj-firefox'
02:30:54     INFO -  gmake[1]: *** [realbuild] Error 2
02:30:54     INFO -  gmake[1]: Leaving directory `/home/worker/workspace/build/src'
02:30:54     INFO -  gmake: *** [build] Error 2
02:30:54     INFO -  0 compiler warnings present.
Depends on: 1300964
https://hg.mozilla.org/mozilla-central/rev/d392d81c5b9e
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
backed out for causing bug 1301348 in https://hg.mozilla.org/mozilla-central/rev/f0d120c2e63bcb6a542a1d0393ecaa046fd5e606
Status: RESOLVED → REOPENED
Flags: needinfo?(atsai)
Resolution: FIXED → ---
Thanks. I am working on it.

The plan B is to lock taskcluster version on 0.0.32 which is the build system originally plan to use.
Flags: needinfo?(atsai)
Attached patch bug1297585.patch (obsolete) — Splinter Review
After testing on Mac OS X and Linux, bumping mozregression version to 2.3.6 will also lock the version of taskcluster on 0.0.32 which works perfect for both OS.
Attachment #8787078 - Attachment is obsolete: true
Attachment #8789653 - Flags: review?(mh+mozilla)
Comment on attachment 8789653 [details] [diff] [review]
bug1297585.patch

Per discuss https://bugzilla.mozilla.org/show_bug.cgi?id=1301359#c6 we'll upgrade the taskcluster version to the newest.
Attachment #8789653 - Flags: review?(mh+mozilla)
Attached file bug1297585
Attachment #8789653 - Attachment is obsolete: true
Attachment #8790573 - Flags: review?(mh+mozilla)
Comment on attachment 8790573 [details]
bug1297585

Issues found while building Linux opt build. Checking now.
Attachment #8790573 - Flags: review?(mh+mozilla)
Depends on: 1302598
Al, do you still want to use version 0.3.4 of Taskcluster client even as you know it is totally busted? Can you please explain the reasoning for it? I feel it awkward to have such a version in use by code in-tree.
Henrik, would you suggest any stable version we should use for building system? I don't insist to use the newest version and have no idea about which version we should use.

You can check the comment for the story.
https://bugzilla.mozilla.org/show_bug.cgi?id=1301359#c5
Flags: needinfo?(hskupin)
Al, I'm not the maintainer of taskcluster-client. You really want to get the feedback from John Ford.
Flags: needinfo?(hskupin) → needinfo?(jhford)
Hi.  I'm still trying to figure out exactly what's up, but unless you're depending on Async code, please use version 0.2 of the taskcluster python client.  The API references that shipped with it are likely a little out of date, but I can do a release for you with an updated version of the api references but the correct interface.
Flags: needinfo?(jhford)
See Also: → 1317327
We tracked this down in bug 1306614. The patch there ought to fix this.
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1306614
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.