Closed Bug 600317 Opened 14 years ago Closed 12 years ago

Fedora/Firefox - Repository setup for Firefox nightly and aurora

Categories

(Release Engineering :: General, defect)

All
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: armenzg, Unassigned)

References

(Depends on 2 open bugs)

Details

(Whiteboard: [q4 goal][carry over])

Attachments

(3 files, 8 obsolete files)

With this we would like to ease Fedora users to reach our nightly, beta and even stable releases.

At first this seems that can be broken down into three subprojects:
* Build configuration for the RPM files.
* Repository configuration RPMs.
* Getting SELinux to work with the nightlies.
Here is a blog post explaining how to get updates on Fedora through yum:
http://armenzg.blogspot.com/2010/12/getting-firefox-nightly-updates-on.html

To install the repo use this rpm:
http://chile.proximity.on.ca/ffrepo/noarch/firefox-repo-0.3-2.noarch.rpm

For instructions on how to do the setup look at posts by Tarin in December:
http://tarinc.wordpress.com/2010/12
Thanks to the work on the previous semester we can now have nightly yum updates for Firefox on the school repositories.

For this quarter we have defined to get that work integrated into Mozilla's buildbot automation. I believe is better for me to take on that.

If this semester I get more students involved I believe there is room for current students to work on the following:
* signing
* beta and stable releases
* ctyler what else do you think?

The only way to know if a new beta is out is to check in here:
http://releases.mozilla.org/pub/mozilla.org/firefox/releases
Once beta9 is out this week we should have a "4.0b9" directory in there.
The only two ways to query what is the next expected directory to appear in there are:
A) Check http://mxr.mozilla.org/mozilla-central/source/browser/config/version.txt?raw=1, remove "pre" and substract 1 ("4.0b10pre" - "pre" - 1 => "4.0b9"
B) Check in http://releases.mozilla.org/pub/mozilla.org/firefox/releases and increment whatever "4.0b" directory you see there (currently "4.0b8"; we should expect to see "4.0b9" at some point)

One day we will have pulse notifications but for now this is what we have.

spot has already been doing some work on the betas which can help you leverage:
http://repos.fedorapeople.org/repos/spot/firefox4/fedora-14/

The following information will also be valuable to get help and info:

General Devel List (devel@fedoraproject.org):
https://admin.fedoraproject.org/mailman/listinfo/devel

Packaging List (packaging@fedoraproject.org):
https://admin.fedoraproject.org/mailman/listinfo/packaging

Also, you might try reaching out to the normal firefox maintainers:
gecko-maint AT fedoraproject.org

Last but not least, the yum maintainer may be able to help you with any
yum issues:
skvidal
Whiteboard: [q4 goal] → [q4 goal][carry over]
Attached file ~/includes/BuildDesktopFile.bash (obsolete) —
Attached file ~/includes/BuildSpecFile.bash (obsolete) —
Attached file ~/UpdateRepo.bash (obsolete) —
I thought I had attached these files but I guess I didn't.

This file gets called everyday and call the other two I have attached.

The script gets called twice a day (EST times below) by a cronjob. Once for the Linux 32-bit build and another for the 64-bit one.

Tarin I have modified the cronjob to call always with the right version even after it bumps for a beta. I modified the "cut" calls since my URL is a little longer (I think yours was "ftp://ftp.mozilla.org/pub/firefox/" which does not contain "mozilla.org" in it).

0 7 * * * ~/UpdateRepo.bash http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-`curl -s http://mxr.mozilla.org/mozilla-central/source/browser/config/version.txt?raw=1`.en-US.linux-i686.tar.bz2 
0 8 * * * ~/UpdateRepo.bash http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-`curl -s http://mxr.mozilla.org/mozilla-central/source/browser/config/version.txt?raw=1`.en-US.linux-x86_64.tar.bz2
Priority: P3 → P4
Whiteboard: [q4 goal][carry over] → [q4 goal][carry over] To tackle week of Feb. 21st or following one
Due to goals prioritization jhford will be finishing up the work for this quarter.
I will debrief him tomorrow.
Assignee: armenzg → jhford
Priority: P4 → --
Whiteboard: [q4 goal][carry over] To tackle week of Feb. 21st or following one → [q4 goal][carry over]
Attachment #503904 - Attachment mime type: application/octet-stream → text/plain
Attachment #503905 - Attachment mime type: application/octet-stream → text/plain
Attachment #503911 - Attachment mime type: application/octet-stream → text/plain
Currently the repo is down as I can't see anything on here:
http://chile.proximity.on.ca/ffrepo/
How can I check why the http server is not running?

Because I am lazy I set up an rsync to my people account:
rsync -aP armenzg@chile.proximity.on.ca:/var/www/html/ffrepo ~/fedora
rsync -aP ~/fedora/ffrepo armenzg@people.mozilla.com:~/public_html

This means the repo is now in http://people.mozilla.com/~armenzg/ffrepo/
It might simply not work but I wanted to keep the directory structure of what was available somewhere else.

It seems that we have lost the original http://chile.proximity.on.ca/ffrepo/noarch/firefox-repo-0.3-2.noarch.rpm (now http://people.mozilla.com/~armenzg/ffrepo/noarch/).
The other rpms are still being generated.

I think it would a good opportunity to integrate this system as a mozharness script.

This is how the code works.
* on chile.p.o.ca the account of Tarin has a cronjob that calls ~/UpdateRepo.bash [1]
* UpdateRepo.bash calls ~/includes/BuildSpecFile.bash and then ~/includes/BuildDesktopFile.bash
* BuildSpecFile.bash generates ~/rpmbuild/SPECS/minefield.spec
* BuildDesktopFile.bash generates ~/rpmbuild/SOURCES/minefield.desktop
* At some point we generate the rpm [2]
* At some point we generate the srpm [3]
* At some point we generate the rpm deltas [4]

This page shows how to do updates for the betas and could be helpful for when we tackle adding the betas in our systems:
http://www.if-not-true-then-false.com/2010/install-firefox-4-on-fedora/

[1] ################
0 8 * * * ~/UpdateRepo.bash http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-`curl -s http://mxr.mozilla.org/mozilla-central/source/browser/config/version.txt?raw=1`.en-US.linux-i686.tar.bz2 
0 9 * * * ~/UpdateRepo.bash http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-`curl -s http://mxr.mozilla.org/mozilla-central/source/browser/config/version.txt?raw=1`.en-US.linux-x86_64.tar.bz2

[2] rpmbuild -ba --quiet --target $arch ~/rpmbuild/SPECS/minefield.spec || exit 1
[3] mv ~/rpmbuild/SRPMS/* $location/$arch/srpms || exit 1
[4] createrepo --update -d --deltas --oldpackagedirs $location/$arch $location/$arch
Attachment #503904 - Attachment is obsolete: true
Attachment #503905 - Attachment is obsolete: true
Attachment #503911 - Attachment is obsolete: true
I am looking into merging in the RPM packaging code into the mozilla build system which makes these patches obsolete.

Once I have the code to generate RPMs I am going to create a nightly builder that triggers beside the regular nightly builder.  This has to be done in a separate builder because I need to build with mk_add_options MOZ_APP_NAME=minefield to avoid conflicting with the OS version of Firefox.
Attached patch buildlbotcustom-v1 (obsolete) — Splinter Review
Diff against 76a6cfcddf47+ tip
Attached patch buildbot-configs v1 (obsolete) — Splinter Review
Diff against c56c95c4f8c1+ tip
Attached patch buildbotcustom v2 (obsolete) — Splinter Review
-allows setting per-platform unit test master list
-allows setting per-platform toggling of dep builds
--caused a massive diff on dep builder, mostly indentation
-allows setting per-platform toggling of partials and snippets
-allows setting per-platform xulrunner
-finds rpms in make upload output
-allows setting a different platform than default for things that go to stage, defaults to old platform value
-sends debug build (malloc + sdleak) logs to the debug directory
-leaves codesize in the opt directory
-Uses make vars as needed to generate RPMs
-causes mozilla-2.0 platform ordering to change because it no longer copies mozilla-central's platform list

When this lands we will need to mv/cp the leak test logs into the correct -debug directory.  Maybe a symlink would help here?  Also, we should communicate this.  This could fix 634968 if we wanted it to
Attachment #523711 - Attachment is obsolete: true
Attached patch buildbot-configs v2 (obsolete) — Splinter Review
matching config patch
Attachment #523712 - Attachment is obsolete: true
running dump masters + diffs on that then will will ask for review
same patch as before other than the place where the stagePlatform default is set.  In v2 it happend before the stripping of 'cruft', which meant that every platform is touched (outside scope of this bug).  That might be something worth doing, but later.  Moved the default of stagePlatform to after the stripping of 'cruft' to make sure that the current location is what is used.
Attachment #525516 - Attachment is obsolete: true
Attachment #525587 - Flags: review?(coop)
Same patch as before but instead of enabling nightlies for the platform then disabling on non-mc branches, disable nightlies on all branches that don't explicitly opt into rpm.  Kind of a hack, but better than reinventing the wheel.  Once rpms become important to people other than me, it might be a good idea to run these rpms on all branches that have nightlies.
Attachment #525517 - Attachment is obsolete: true
Attachment #525589 - Flags: review?(coop)
Attached patch dumpmasters.py diff (obsolete) — Splinter Review
diff of a pre and post patch version of the v3 patches
Attachment #525591 - Flags: feedback?(coop)
this time, instead of an interdiff between v2 and v3 of the buildbotcustom patch, i am uploading a dump master diff of before and after my patch
Attachment #525591 - Attachment is obsolete: true
Attachment #525591 - Flags: feedback?(coop)
changing title to reflect that what was 'beta' is now 'aurora'.
Summary: Fedora/Firefox - Repository setup for Firefox nightly and betas → Fedora/Firefox - Repository setup for Firefox nightly and aurora
Comment on attachment 525587 [details] [diff] [review]
buildbotcustom v3

>diff --git a/misc.py b/misc.py
>@@ -1102,6 +1105,9 @@ def generateBranchObjects(config, name):
>                nightly_builder in l10nNightlyBuilders:
>                 triggeredSchedulers=[l10nNightlyBuilders[nightly_builder]['l10n_builder']]
> 
>+            createSnippet = pf.get('create_snippet', config['create_snippet'])
>+            createPartial = pf.get('create_partial', config['create_partial'])
>+
>             mozilla2_nightly_factory = NightlyBuildFactory(
>                 env=pf['env'],
>                 objdir=pf['platform_objdir'],
>@@ -1124,8 +1130,8 @@ def generateBranchObjects(config, name):
>                 uploadPackages=uploadPackages,
>                 uploadSymbols=pf.get('upload_symbols', False),
>                 nightly=True,
>-                createSnippet=config['create_snippet'],
>-                createPartial=config['create_partial'],
>+                createSnippet=createSnippet,
>+                createPartial=createPartial,
>                 ausBaseUploadDir=config['aus2_base_upload_dir'],
>                 updatePlatform=pf['update_platform'],
>                 downloadBaseURL=config['download_base_url'],

Is there a reason you don't inline the calls to pf.get('create_*') like we seem to do for the others?

>@@ -1149,6 +1155,7 @@ def generateBranchObjects(config, name):
>                 useSharedCheckouts=pf.get('enable_shared_checkouts', False),
>                 testPrettyNames=pf.get('test_pretty_names', False),
>                 l10nCheckTest=pf.get('l10n_check_test', False),
>+                stagePlatform=pf.get('stage_platform'),
>             )
> 
>             mozilla2_nightly_builder = {

>@@ -1425,7 +1432,7 @@ def generateBranchObjects(config, name):
>                 blocklistBuilder = generateBlocklistBuilder(config, name, platform, pf['base_name'], pf['slaves'])
>                 branchObjects['builders'].append(blocklistBuilder)
> 
>-        if config['enable_xulrunner'] and platform not in ('wince',):
>+        if pf.get('enable_xulrunner', config['enable_xulrunner']):
>              xr_env = pf['env'].copy()
>              xr_env['SYMBOL_SERVER_USER'] = config['stage_username_xulrunner']
>              xr_env['SYMBOL_SERVER_PATH'] = config['symbol_server_xulrunner_path']

Bonus points for removing references to wince! ;)

>diff --git a/process/factory.py b/process/factory.py
>--- a/process/factory.py
>+++ b/process/factory.py
>@@ -1,5 +1,6 @@
> from datetime import datetime
> import os.path, re
>+import posixpath
> from time import strftime
> import urllib

Why do we need posixpath? Does it provide something different than os.path?
 
>+        # it turns out that the cruft is useful sometimes.  Instead
>+        # of refactoring the entire MercurialBuildFactory and children,
>+        # lets create a list of attributes that cause variation.

Please be specific as to what the "sometimes" are in the comment, and leave out the bit about refactoring.

r+ with comments addressed.
Attachment #525587 - Flags: review?(coop) → review+
Comment on attachment 525589 [details] [diff] [review]
buildbot-configs v3

Just a general comment on this one:

As we discussed, having separate platforms for this is OK as an interim step. If we're serious about providing rpms on a long-term basis, rpm creation should be a part of the regular linux build process, and we should be able to toggle it on and off with a single var (or handful of vars).

Can we get a bug on file for that improvement, please?
Attachment #525589 - Flags: review?(coop) → review+
(In reply to comment #21)
> Comment on attachment 525587 [details] [diff] [review]
> buildbotcustom v3
> Is there a reason you don't inline the calls to pf.get('create_*') like we seem
> to do for the others?

I had thought we didn't have nested dictionary lookups, but it turns out that we do.  Will change.
 
> Bonus points for removing references to wince! ;)

:D

> Why do we need posixpath? Does it provide something different than os.path?

ooops, I thought I had removed that (hg meld sometimes opens temp files instead of real files).  I was using posixpath in code i was adding because the os the master runs on is very often unrelated to the code I need to use os.path in.  os.path is just posixpath if your interpreter is running on a posix platform.

> Please be specific as to what the "sometimes" are in the comment, and leave out
> the bit about refactoring.

Ok, will do.

> r+ with comments addressed.

Thanks!
(In reply to comment #22)
> Comment on attachment 525589 [details] [diff] [review]
> buildbot-configs v3
> 
> Just a general comment on this one:
> 
> As we discussed, having separate platforms for this is OK as an interim step.
> If we're serious about providing rpms on a long-term basis, rpm creation should
> be a part of the regular linux build process, and we should be able to toggle
> it on and off with a single var (or handful of vars).
> 
> Can we get a bug on file for that improvement, please?

Yes, I agree.  Until the linux startup process stops using the firefox script, this is a little more difficult.  I will file the bug now.
Depends on: 650482
Depends on: 650516
I am going to be working on other projects for the foreseeable future and I won't have time to drive these bugs through to completion.  I am marking outstanding packaging fixes as INCOMPLETE.
Assignee: jhford → nobody
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → INCOMPLETE
Product: mozilla.org → Release Engineering
QA Contact: mshal
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: