487208 - How to make use of Mozilla's AUS update system for Solaris contributed builds?

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Alfred Peng]

Thanks a lot for moving all the Solaris contributed builds to bouncer. The Mozilla download page is available in OpenSolaris community: http://opensolaris.org/os/community/desktop/communities/mozilla/development/. It highlights all the contributed builds with pointers to bouncer.

Another feature that Solaris/OpenSoalris users heavily expect is the automatic update. Currently, the Solaris/OpenSolaris users have to download the Firefox/Thunderbird builds every time there is a new official release, about 20M each, remove the old binary and change to use the new one. That's painful. It'll be a wonderful improvement if we could enable the "check for updates" for Solaris builds.

From bug 381420, It seems that Mozilla community has several different AUS server. Could the snippets (*.mar) for Solaris contributed Firefox/Thunderbird builds be hosted on one of the AUS servers? How can we generate snippets for Solaris builds ? Which URL should "app.update.url" point to? Any guidance on how to make this happen for Solaris builds will be appreciated. We'd really hope that the automatic update can be enabled in the upcoming Firefox 3.0.9 release.

Comment 1

[Johnny Stenback (:jst)]

Mike, what's needed to make this possible?

Comment 2

[Phil Ringnalda (:philor)]

gozer: heads up for our forthcoming yet-another-AUS :)

Comment 3

[Philippe M. Chiasson (:gozer)]

(In reply to comment #2)
> gozer: heads up for our forthcoming yet-another-AUS :)

Is this something we want to be doing on our aus server going onwards? I believe as long as there are complete .MARs for these Solaris builds, I think patcher will be able to produce the partial updates and our AUS server serve them.

Comment 4

[Alfred Peng]

gozer，I'd hope that Mozilla community can provide the support for automatic update of Solaris contributed builds (Firefox/Thunderbird with only en-US locale). Also there seems to be some interests from Mozilla Messaging to provide the Solaris builds for Thunderbird 3.0. I guess automatic update will be enabled if that happens.

How should we generate the .MARs? Could the .MARs be uploaded to some AUS server?

Comment 5

[Philippe M. Chiasson (:gozer)]

(In reply to comment #4)
> gozer，I'd hope that Mozilla community can provide the support for automatic
> update of Solaris contributed builds (Firefox/Thunderbird with only en-US
> locale).

Quite likely, and most likely easy.

> Also there seems to be some interests from Mozilla Messaging to
> provide the Solaris builds for Thunderbird 3.0. I guess automatic update will
> be enabled if that happens.

Yes, if/when that happens, I'd expeect that to run off the Mozilla Messaging AUS server.

> How should we generate the .MARs?

First step would be to generate complete .MARs alongside the 2.0.0.x builds you do.

> Could the .MARs be uploaded to some AUS server?

The complete MARs should be uploaded to ftp.mozilla.org right beside the regular .tar.bz2.

Once these exist, the next step is to either let the AUS server know about these and serve them up as complete updates. Or, generate partial MARs to allow for smaller/faster partial updates.

Comment 6

[Alfred Peng]

(In reply to comment #5)
> Quite likely, and most likely easy.

Thanks!


> First step would be to generate complete .MARs alongside the 2.0.0.x builds you
> do.

As the Solaris users complaint that the "check for updates..." menu does nothing, we added "--disable-updater" to the mozconfig for all the previous contributed builds, including 2.0.0.x. However, the "check for updates..." menu item wasn't grayed out and it should work for 2.0.0.x. Unfortunately, all the 3.0.x/3.1 builds disable this menu item.

What's these complete .MARs of 2.0.0.x builds for in this case? If it's necessary to have them, I'll re-make the builds and upload them to the contrib directory.

> The complete MARs should be uploaded to ftp.mozilla.org right beside the
> regular .tar.bz2.

We'll upload the complete MARs for future milestone builds. Not sure whether it's necessary to upload for the previous releases?

> Once these exist, the next step is to either let the AUS server know about
> these and serve them up as complete updates. Or, generate partial MARs to allow
> for smaller/faster partial updates.

The plan is to generate particial MARs for faster updates. Are the partial MARs supposed to be uploaded to the .tar.bz2 directory as well?

BTW, I guess we should patch the code to point "app.update.url" to certain Mozilla AUS server.

Comment 7

[Alfred Peng]

Seems to be pretty straight forward to generate the complete MARs for the builds by using the update-packaging tools with the following in mozconfig:

ac_add_options --enable-updater
ac_add_options --enable-update-packaging
ac_add_options --enable-update-channel=release

Do I need to upload the partial MARs also? Or patcher2.pl will do the magic automatically?

Comment 8

[Philippe M. Chiasson (:gozer)]

(In reply to comment #6)
> (In reply to comment #5)
> > Quite likely, and most likely easy.
> 
> Thanks!
> 
> > First step would be to generate complete .MARs alongside the 2.0.0.x builds you
> > do.
> 
> As the Solaris users complaint that the "check for updates..." menu does
> nothing, we added "--disable-updater" to the mozconfig for all the previous
> contributed builds, including 2.0.0.x. However, the "check for updates..." menu
> item wasn't grayed out and it should work for 2.0.0.x. Unfortunately, all the
> 3.0.x/3.1 builds disable this menu item.
> 
> What's these complete .MARs of 2.0.0.x builds for in this case? If it's
> necessary to have them, I'll re-make the builds and upload them to the contrib
> directory.

Unfortunately, you can easily re-make these, as they have to be bit-for-bit identical as to what has been shipped. Better just start making them for the next release.

> > The complete MARs should be uploaded to ftp.mozilla.org right beside the
> > regular .tar.bz2.
> 
> We'll upload the complete MARs for future milestone builds. Not sure whether
> it's necessary to upload for the previous releases?

Yes, please make sure to generate them and upload them in the future. For past builds, see above.

> > Once these exist, the next step is to either let the AUS server know about
> > these and serve them up as complete updates. Or, generate partial MARs to allow
> > for smaller/faster partial updates.
> 
> The plan is to generate particial MARs for faster updates.

Yes, but unfortunately, partial MARs can only be generated from 2 complete MARs, and since you don't have these for previous builds, you'll only be able to partial update folks to the 2nd next release (next release you'll have your first set of complete MARs, then release after that, we'll be able to make partial updates)

> Are the partial MARs
> supposed to be uploaded to the .tar.bz2 directory as well?

Yes, but that's not generated by the regular build, so they will just be placed there.

> BTW, I guess we should patch the code to point "app.update.url" to certain
> Mozilla AUS server.

Yes, and depending on what AUS server we are talking about the url will be different.

For 2.0.0.x, I suspect that would have to be the community AUS server.
For 3.0.x, that would be aus.mozillamessaging.com

Comment 9

[Philippe M. Chiasson (:gozer)]

(In reply to comment #7)
> [...]
> 
> Do I need to upload the partial MARs also? Or patcher2.pl will do the magic
> automatically?

Patcher will generate the partial MARs for you. So just generate the complete MARs and that should be enough.

Comment 10

[Alfred Peng]

(In reply to comment #8)
> > BTW, I guess we should patch the code to point "app.update.url" to certain
> > Mozilla AUS server.
> 
> Yes, and depending on what AUS server we are talking about the url will be
> different.
> 
> For 2.0.0.x, I suspect that would have to be the community AUS server.
> For 3.0.x, that would be aus.mozillamessaging.com

Thanks for the response! Still confused about the url stuff however. Currently for Firefox 3.1b3 in OpenSolaris, firefox.js contains the line:
pref("app.update.url", "https://aus2.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml");

For Thunderbird 2.0.0.21, all-thunderbird.js contains:
pref("app.update.url", "https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml");

Those should be the default url as I understand. So for the next Firefox 3.0.9 (Apr. 21) contributed builds, how should I update the preference if there is a community AUS server we can use?

Comment 11

[Alfred Peng]

Forgot to mention that the Firefox 3.5b4 is also coming soon. Should I update the preference as well?

Comment 12

[Alfred Peng]

After discussed with gozer on IRC, I've updated the preference in firefox.js as:

pref("app.update.url", "https://aus2-community.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml");

The Firefox 3.0.9 contributed builds and the complete .MARs have been uploaded to http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0.9-candidates/build2/solaris_tarball/ and will be moved to release contrib folder later. Please feel free to notify me if anything is missing.

Comment 13

[Robert Kaiser]

Ause, would you have cycles to look into this? I figure if there's someone @sun who has access to the aus2-community server, he might have the ability to allocate time for this ;-)

If ause doesn't have time, I guess it would be up to me to try and find whatever time needed to get things up and running for you guys...

Comment 14

[Alfred Peng]

(In reply to comment #13)
> Ause, would you have cycles to look into this? I figure if there's someone @sun
> who has access to the aus2-community server, he might have the ability to
> allocate time for this ;-)

I assume the thing we need to do is to get the account for aus2-community server, upload the complete .MARs to the server every time when the contributed builds are ready.

Will the partial .MARs be generated automatically on the server side? Or they should be created by hand and uploaded as well?

> If ause doesn't have time, I guess it would be up to me to try and find
> whatever time needed to get things up and running for you guys...

Thanks for the help, whoever has the time for this! :)

Comment 15

[Philippe M. Chiasson (:gozer)]

(In reply to comment #14)
> (In reply to comment #13)
> > Ause, would you have cycles to look into this? I figure if there's someone @sun
> > who has access to the aus2-community server, he might have the ability to
> > allocate time for this ;-)
> 
> I assume the thing we need to do is to get the account for aus2-community
> server, upload the complete .MARs to the server every time when the contributed
> builds are ready.

Actually, all that's uploaded to the aus2-community server would be update snippets. Very simple text files that describe what updates are available. The actual updates (partial or complete) are just referred by hash and url, so they
don't need to live on the aus server.

> Will the partial .MARs be generated automatically on the server side? Or they
> should be created by hand and uploaded as well?

They are created separately by a tool called patcher, and can be generated anywhere, like on the machine you use to produce the builds in the first place.

Then the generated partial MARs can just be rsync'ed to ftp.mozilla.org

> > If ause doesn't have time, I guess it would be up to me to try and find
> > whatever time needed to get things up and running for you guys...
> 
> Thanks for the help, whoever has the time for this! :)

I can probably help there as well, last I checked, I had access to that system

Comment 16

[ause]

i don't feel like being the right one, especially when it comes to create new users/groups and importing their ssh keys. afaik, that's the next step to get the uploads of the update snippets running.
if this is done, adopting config.php should be the last step.

Comment 17

[Alfred Peng]

(In reply to comment #15)
> Actually, all that's uploaded to the aus2-community server would be update
> snippets. Very simple text files that describe what updates are available. The
> actual updates (partial or complete) are just referred by hash and url, so they
> don't need to live on the aus server.

Thanks for the explanation :)

> They are created separately by a tool called patcher, and can be generated
> anywhere, like on the machine you use to produce the builds in the first place.
> 
> Then the generated partial MARs can just be rsync'ed to ftp.mozilla.org

The partial MARs will be uploaded starting from the next Firefox 3.0.x release.

(In reply to comment #16)
> i don't feel like being the right one, especially when it comes to create new
> users/groups and importing their ssh keys. afaik, that's the next step to get
> the uploads of the update snippets running.
> if this is done, adopting config.php should be the last step.

Should I post the ssh public key here for a new upload account? Or should I file a bug every time the update snippets are ready and someone else will take care of the rest of the stuff? Like what we have been doing for moving the contributed builds to bouncer?

Comment 18

[Philippe M. Chiasson (:gozer)]

(In reply to comment #17)
> (In reply to comment #15)
> [...]
> > 
> > Then the generated partial MARs can just be rsync'ed to ftp.mozilla.org
> 
> The partial MARs will be uploaded starting from the next Firefox 3.0.x release.

You'll probably need to figure out patcher configs that work for you. See:

<http://mxr.mozilla.org/mozilla/source/tools/patcher/>
and
<http://mxr.mozilla.org/mozilla/source/tools/patcher-configs/>

For inspiration.

> (In reply to comment #16)
> > i don't feel like being the right one, especially when it comes to create new
> > users/groups and importing their ssh keys. afaik, that's the next step to get
> > the uploads of the update snippets running.
> > if this is done, adopting config.php should be the last step.
> 
> Should I post the ssh public key here for a new upload account?

Yes, you might as well, as that's going to be required, one way or another.

> Or should I
> file a bug every time the update snippets are ready and someone else will take
> care of the rest of the stuff? Like what we have been doing for moving the
> contributed builds to bouncer?

Nope, it's normally self-service with ssh+rsync

Comment 19

[Alfred Peng]

(In reply to comment #18)
> You'll probably need to figure out patcher configs that work for you. See:
> 
> <http://mxr.mozilla.org/mozilla/source/tools/patcher/>
> and
> <http://mxr.mozilla.org/mozilla/source/tools/patcher-configs/>
> 
> For inspiration.

Thanks for the pointers. The partial MARs for Firefox 3.0.10 have been uploaded to http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0.10-candidates/build1/solaris_tarball/. I've tried the OpenSolaris i386 one and it works.

> Yes, you might as well, as that's going to be required, one way or another.

My public key is here: https://bug344041.bugzilla.mozilla.org/attachment.cgi?id=228609

> Nope, it's normally self-service with ssh+rsync

That's great. Could anyone please help upload the partial/compelete MARs for Firefox 3.0.10 to the community AUS server so that Solaris users can start to test/use this feature? Or I'll do that myself if I can get the account before Apr. 30th. Thanks.

Comment 20

[Alfred Peng]

(In reply to comment #19)
> > Nope, it's normally self-service with ssh+rsync
> 
> That's great. Could anyone please help upload the partial/compelete MARs for
> Firefox 3.0.10 to the community AUS server so that Solaris users can start to
> test/use this feature? Or I'll do that myself if I can get the account before
> Apr. 30th. Thanks.

The builds have been just moved to http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.10/contrib/solaris_tarball/, with four partial MARs inside.

A small correction, as Gozer has pointed out, we don't need to upload the MARs to the community AUS server, only a "very simple text files that describe what updates are available.

Comment 21

[Chris Cooper [:coop] (he/him)]

Re-assigning to Alfred since he seems to be doing the work here.

Comment 22

[Alfred Peng]

Coop, Gozer, any idea on how I can proceed? Guess I still need:
1. The account to upload some description file to the community AUS server.
2. The manual on how to generate the description file and where to upload.

Comment 23

[Chris Cooper [:coop] (he/him)]

Some comments:

(In reply to comment #8)
> For 2.0.0.x, I suspect that would have to be the community AUS server.

If I were you, I might forgo trying to generate updates (well, at least partial updates) for 2.0.0.x at this point. Yes, you still have the update mechanism turned on, but as gozer indicates, you would have to wait two release cycles to have partial updates pay off. That could be a *long* time given Thunderbird release schedules, and presumably the 2.0.0.x builds will be EOLed after 3.0 is officially release. Better to turn on the update code in 3.0 and get that working IMO.

(In reply to comment #22)
> Coop, Gozer, any idea on how I can proceed? Guess I still need:
> 1. The account to upload some description file to the community AUS server.
> 2. The manual on how to generate the description file and where to upload.

1. I can't create accounts any more on the community AUS server. Not sure whether gozer can, but probably best to reassign this bug to [e-mail] to get an account created for yourself.

2. Are you talking about snippet generation? There are no really good docs there, but maybe gozer can show you an example of an existing Thunderbird snippet as illustration?

Comment 24

[Alfred Peng]

Coop, Thanks for the comments!

(In reply to comment #23)
> If I were you, I might forgo trying to generate updates (well, at least partial
> updates) for 2.0.0.x at this point. Yes, you still have the update mechanism
> turned on, but as gozer indicates, you would have to wait two release cycles to
> have partial updates pay off. That could be a *long* time given Thunderbird
> release schedules, and presumably the 2.0.0.x builds will be EOLed after 3.0 is
> officially release. Better to turn on the update code in 3.0 and get that
> working IMO.

As 2.0.0.x has been EOLed for Firefox and in slow progress for Thunderbird, we won't do that any more. Thanks for the suggestion. The auto update will be enabled for Firefox/Thunderbird contributed builds since 3.0.x. FYI, the partial updates are available for Firefox 3.0.10 and 3.0.11:
http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.10/contrib/solaris_tarball/
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0.11-candidates/build1/solaris_tarball/

> 1. I can't create accounts any more on the community AUS server. Not sure
> whether gozer can, but probably best to reassign this bug to
> [e-mail] to get an account created for yourself.

Re-assigned.

> 2. Are you talking about snippet generation? There are no really good docs
> there, but maybe gozer can show you an example of an existing Thunderbird
> snippet as illustration?

Yes, snippet generation. I've tried to manually install the partial MARs and it works (https://wiki.mozilla.org/Software_Update:Manually_Installing_a_MAR_file). Gozer, any idea for that?

Comment 25

[Alfred Peng]

Any idea on how can I make this to happen? Or I need to wait until after the release of Firefox 3.5? Thanks!

Comment 26

[Shyam Mani [:fox2mike]]

(In reply to comment #25)
> Any idea on how can I make this to happen? Or I need to wait until after the
> release of Firefox 3.5? Thanks!

Alfred, you now have access to cb-ausstage01. Please use your ssh keys to login.

Comment 27

[Alfred Peng]

(In reply to comment #26)
> Alfred, you now have access to cb-ausstage01. Please use your ssh keys to
> login.

Thanks for the help!

Am I supposed to have a user account to login? what does cb-ausstage01 mean here? I search a little bit. It seems that cb-ausstage01 is one of the Mozilla vmware servers. What's the hostname and domainname for it?

Back to Gozer's comment #18, I think I need to upload a config file similar to http://mxr.mozilla.org/mozilla/source/tools/patcher-configs/moz191-branch-patcher2.cfg to the AUS server. Is there any specific path that I should upload the file to?

Comment 28

[Shyam Mani [:fox2mike]]

(In reply to comment #27)

> Am I supposed to have a user account to login? what does cb-ausstage01 mean
> here? I search a little bit. It seems that cb-ausstage01 is one of the Mozilla
> vmware servers. What's the hostname and domainname for it?

Login as root to cb-ausstage01.sj.mozilla.com with your ssh keys, you should have access to the box.
 
> Back to Gozer's comment #18, I think I need to upload a config file similar to
> http://mxr.mozilla.org/mozilla/source/tools/patcher-configs/moz191-branch-patcher2.cfg
> to the AUS server. Is there any specific path that I should upload the file to?

I'm not really sure. mxr.mozilla.org looks like an entirely different box.

Comment 29

[Robert Kaiser]

(In reply to comment #28)
> Login as root to cb-ausstage01.sj.mozilla.com with your ssh keys, you should
> have access to the box.

Right. You can also use the "aus2-community.mozilla.org" host name to log in, it's the same machine. Just don't mess up with the stuff we are using as well, i.e. the aus2 code itself or the Sunbird or SeaMonkey snippets - or if you need to (esp. in case of the rules in the aus2 code that tells what builds to send where or so), please ping me on IRC so I know.

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

Can I respectfully suggest that a user level account would be sufficient for Alfred on aus2-community.

Comment 31

[Shyam Mani [:fox2mike]]

(In reply to comment #30)
> Can I respectfully suggest that a user level account would be sufficient for
> Alfred on aus2-community.

Nick,

We did consider that. Seems like everyone else (non moco) and there's a few people in the list, already have access to the box as mentioned above. There are no accounts for them either. While that does sound like a bad reason not to have accounts, it's already been that way for a while, hence I decided to stick to it.

Comment 32

[Alfred Peng]

Guys, thanks all for your kind help! After doing some experiments with gozer's pointer for patcher, I've successfully generated the snippets for Firefox Solaris contributed builds and uploaded them to the aus community server. Now, the automatic update works properly in beta channel(for test purpose) for the Firefox 3.0.10 contributed builds. However, got some problem when I followed the same procedure for the Firefox 3.5 builds.

I tried to do a partial update for Firefox 3.5b99->3.5rc3 for verification. The snippet and partial mar are generated. If I run Firefox 3.5b99 to do the update, the partial mar can be downloaded. But when I restart Firefox, no small update window pops up and Firefox stays still with Firefox 3.5b99. And it starts to download the complete mar automatically. Well, the complete mar doesn't work either :(

BTW, I've tried to manually install the partial mar for verification and it works (https://wiki.mozilla.org/Software_Update:Manually_Installing_a_MAR_file).

Is there anything specific that I should check about? Appreciated for any comments.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

What are appv and extv set to in the snippets ? Due to some updater changes they must be "3.5" for the update to apply.

Comment 34

[Alfred Peng]

Attachment: patcher config file to create the snippet

Nick, thanks for the comment. I tried to set both appv and extv to 3.5 in the snippet, but still no lucky. Partial/complete mars can be downloaded, but can't update.

Attached is the config file to create the snippet.

Comment 35

[Alfred Peng]

Attachment: partial snippet generated by patcher

Comment 36

[Alfred Peng]

Attachment: update xml from the aus server

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Should have read comment 32 more carefully it seems. It sounds like the Firefox fails to launch the updater and that may be a Solaris specific bug. What does updates/0/last-update.log contain ? Please check it was modified at your last update attempt (or try to update again). Can Firefox restart itself to update an add-on ?

Comment 38

[Alfred Peng]

I can't find the log file for Firefox 3.5b99. For comparison, there is one for my Firefox 3.0.10 binary after successfully updating (updates/last-update.log).

To restart Firefox 3.5b99 after downloading the partial mar, it seems that Firefox will remove the "updates/0" folder and try to get the complete mar. The updater binary should work as I can manually apply the partial mar. Maybe something wrong with the code logic on calling updater on Solaris.

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Probably you want to talk to Rob Strong about this, but a couple more pointers.
1, Download the "partial", chose Restart Later, exit Firefox. Inspect the updates/0 directory. There should be several files in there, including an update.version which is new since 3.0.11
2, To get logging of the in-app JS code, you can create a boolean pref app.update.log.all and look in the Error Console.

Comment 40

[Alfred Peng]

Attachment: log with app.update.log.all set to true

Thanks for the pointer. Attached is the log output in the terminal after app.update.log.all is set to true.

It seems that the key point is that updates/0 does exist after "partial" is downloaded. But it's gone when Firefox restarts. So Firefox tries to get the "complete" mar. Something wrong with updater?

Comment 41

[Chris Cooper [:coop] (he/him)]

Resolving FIXED based on Alfred's comment #6 in bug 516403.