922145 - publish nightly builds of GeckoView library and assets

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Comment 1

[Brad Lassey [:blassey] (use needinfo?)]

Armen, what do we need to do to put the GeckoView library and assets

Comment 2

[Armen [:armenzg]]

I think I need a little bit of more context in here.

Is the code already part of the Android builds?
Is this a change for Releng? or for mfinkle to land a change to enable the code on the builds?
Is this code in a given branch?

Sorry I could not be of more help at first.

Comment 3

[Brad Lassey [:blassey] (use needinfo?)]

This code is on central. Essentially we want to make sure that the contents of dist/geckoview_library/ end up on ftp. currently that is geckoview_assets.zip and geckoview_library.zip.

Comment 4

[Armen [:armenzg]]

After an IRC conversation I understand this a bit better.
We are now creating a zip file in our build process for a zip file that developers can use to create apps based on gecko for mobile.

* There's no need to sign it
* We will be releasing this on every fennec release

I don't know much of the code, however, I believe these pieces of code are related:
http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#203
http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#2419

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: upload_geckoview.patch

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 814463 [details] [diff] [review]
upload_geckoview.patch

Should you add a check for MOZ_DISABLE_GECKOVIEW? We added that to stop l10n builds from attempting to make the archives.

Comment 7

[Armen [:armenzg]]

After asking to the rest of releng, that a build patch is the only thing needed to add this to the nightly directories.

For the release side we might only need to modify push_fennec.sh:
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Push_Fennec_to_mirrors

Comment 8

[(not currently active) Ted Mielczarek]

Comment on attachment 814463 [details] [diff] [review]
upload_geckoview.patch

Review of attachment 814463 [details] [diff] [review]:
-----------------------------------------------------------------

We actually wildcard everything in UPLOAD_EXTRA_FILES, so having it missing is not harmful.

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Landed in https://hg.mozilla.org/integration/mozilla-inbound/rev/8a4946c07c92


But backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/a959889ae978 for breaking Android: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=8a4946c07c92

Comment 10

[Armen [:armenzg]]

It seems that the two files get uploaded as expected:
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381270935/geckoview_assets.zip
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381270935/geckoview_library.zip

It seems that the mozharness scripts get confused and tried to download the crashreporter-symbols.zip for them:
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381270935/geckoview_assets.crashreporter-symbols.zip

I think the problem is the way the sendchanges are constructed:
python /builds/slave/m-in-and-000000000000000000000/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 5 --stdout-regexp 'change sent successfully' buildbot sendchange --master buildbot-master81.build.mozilla.org:9301 --username sendchange-unittest --branch mozilla-inbound-android-opt-unittest --revision 8a4946c07c92 --comments 'bug 922145 - publish nightly builds of GeckoView library and assets r=ted' --property buildid:20131008152215 --property pgo_build:False --property builduid:86cf43a4ba21411a93bd6fc49336a3ef http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381270935/geckoview_assets.zip http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381270935/fennec-27.0a1.en-US.android-arm.tests.zip

It specifies this:
geckoview_assets.zip
when it should have specified this:
fennec-27.0a1.en-US.android-arm.apk

Comment 11

[Armen [:armenzg]]

Attachment: [untested] ignore geckoview

This is untested.

I think this is the fix.

Comment 12

[Armen [:armenzg]]

aki, would you like me to test it?

(I so wish testing these type of changes were easier to test)

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 815039 [details] [diff] [review]
[untested] ignore geckoview

I'm pretty sure this will do the trick.

Comment 14

[Armen [:armenzg]]

Comment on attachment 815039 [details] [diff] [review]
[untested] ignore geckoview

https://hg.mozilla.org/build/buildbotcustom/rev/717dccdcb472

I will push this live tomorrow morning and give the bug back to blassey to land his patch.
I will also do a push to try before giving the bug back just to verify that it works.

Comment 15

[Armen [:armenzg]]

The code is now live.
I tried pushing it to try, however, my pushes to try are not going through.
Instead I decided to re-trigger the Android build jobs for revision 8a4946c07c92.
We should see in an hour or so the builds complete and hopefully not see the test jobs fail in less than 10 minutes.

Comment 16

[Armen [:armenzg]]

I don't understand. This should have worked.
I will have to go to staging.

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=28935824&tree=Mozilla-Inbound&full=1

Comment 17

[Armen [:armenzg]]

My new fix did not make the cut.
I'm now testing this:

diff --git a/process/factory.py b/process/factory.py
--- a/process/factory.py
+++ b/process/factory.py
@@ -225,17 +225,17 @@ def parse_make_upload(rc, stdout, stderr
         elif m.endswith('apk') and 'robocop' in m:
             retval['robocopApkUrl'] = m
         elif 'jsshell-' in m and m.endswith('.zip'):
             retval['jsshellUrl'] = m
         elif m.endswith('.complete.mar'):
             retval['completeMarUrl'] = m
         elif m.endswith('.mar') and '.partial.' in m:
             retval['partialMarUrl'] = m
-        elif m.startswith('geckoview'):
+        elif m.find('geckoview') >= 0:
             pass
         else:
             retval['packageUrl'] = m
     return retval
 
 
 def short_hash(rc, stdout, stderr):
     ''' This function takes an hg changeset id and returns just the first 12 chars'''

Comment 18

[Armen [:armenzg]]

Attachment: geckoview.diff

The right properties get set:
http://dev-master01.build.scl1.mozilla.com:8040/builders/Android%202.2%20mozilla-inbound%20build/builds/2

The right sendchange is created:
buildbot sendchange --master dev-master01.build.scl1.mozilla.com:9901 --username sendchange-unittest --branch mozilla-inbound-android-opt-unittest --revision 8a4946c07c92 --property buildid:20131010155817 --property pgo_build:False --property builduid:26f7446c38fd4cdb85d7a1e085d3f6ce http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381445897/fennec-27.0a1.en-US.android-arm.apk http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1381445897/fennec-27.0a1.en-US.android-arm.tests.zip

Comment 19

[Lucas Rocha (:lucasr)]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #3)
> This code is on central. Essentially we want to make sure that the contents
> of dist/geckoview_library/ end up on ftp. currently that is
> geckoview_assets.zip and geckoview_library.zip.

We should probably investigate the possibility of shipping a single aar (Android Archive) containing both assets and library instead.

Comment 20

[Armen [:armenzg]]

This is live now.

I pushed a build to try:
https://tbpl.mozilla.org/?tree=Try&rev=13eed7b55142

blassey: this should be good to good.
You can wait on try if you want to be safe, however, I believe we're good.

Comment 21

[Ben Bucksch (:BenB)]

https://tbpl.mozilla.org/?tree=Try&rev=13eed7b55142 links to
https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-13eed7b55142/
(which is currently, 30 minutes later, still "404 not found").

Comment 22

[Armen [:armenzg]]

Builds take a while.

Comment 23

[Armen [:armenzg]]

The try build and test have completed successfully.

Comment 24

[Ben Bucksch (:BenB)]

whooo! Thanks a ton!

Comment 25

[Brad Lassey [:blassey] (use needinfo?)]

pushed to inbound https://hg.mozilla.org/integration/mozilla-inbound/rev/c2e371d16e65

(In reply to Lucas Rocha (:lucasr) from comment #19)
> (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #3)
> > This code is on central. Essentially we want to make sure that the contents
> > of dist/geckoview_library/ end up on ftp. currently that is
> > geckoview_assets.zip and geckoview_library.zip.
> 
> We should probably investigate the possibility of shipping a single aar
> (Android Archive) containing both assets and library instead.

Sounds good, file a bug for that?

Comment 26

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c2e371d16e65