Closed Bug 481685 Opened 15 years ago Closed 15 years ago

clean up ftp.mozilla.org Lightning/Sunbird/Thunderbird nightly directories to be less confusing

Categories

(Mozilla Messaging Graveyard :: Release Engineering, defect)

Product:
Mozilla Messaging Graveyard
Component:
Release Engineering
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dmosedale, Assigned: gozer)

References

(
URL
)

Details

(Keywords: dev-doc-complete)

Attachments

(1 file)

Part buildbot config patch
1.16 KB, patch
Details | Diff | Splinter Review 
We need to start nightly builds of Lightning that come from the same mozilla-central branch that Tb 3.0 will be shipping from.
Currently we have only Lightning builds for comm-central/mozilla-1.9.1. 
Do you meant a different branch?
It appears that I got confused by naming structure differences here.

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/ has several relevant directories here:

latest-comm-central/
latest-comm-central-trunk/
latest-comm-1.9.1/

It would appear that latest-comm-central points to latest-comm-1.9.1

On the calendar side, we see:

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/ with

latest-comm-central/
latest-trunk/
latest-comm-central-calendar/

If we could make the sub-directory names for Lightning match the Tb ones, it would be much easier to figure out which builds are which (and which ones should be tested together).
Summary: Need builds of Lightning based on mozilla-central 1.9.1 → clean up ftp.mozilla.org lightning nightly directory to be less confusing
I can easily symlink things around, but keep in mind that lightning *only* builds against releases/mozilla-1.9.1 at this time, so everything is really just a symlink to:

latest-comm-central-calendar/

So I could get rid latest-comm-central and/or latest-trunk as well as creqate latest-comm-1.9.1/ if that would help.
If we're not actually spinning mozilla-central trunk builds, latest-trunk should definitely go away (or at least be empty).  Maybe also rename latest-comm-central-calendar to latest-comm-1.9.1?

Although I wonder if any of these changes will break the lightning nightly updater extension...
(In reply to comment #2)
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/ has several
> relevant directories here:
> 
> latest-comm-central/
> latest-comm-central-trunk/
> latest-comm-1.9.1/

There is also
  latest-trunk

> If we could make the sub-directory names for Lightning match the Tb ones, it
> would be much easier to figure out which builds are which (and which ones
> should be tested together).

I would like to propose for both Thunderbird and Lightning (and maybe even Sunbird) the following:

latest-comm-1.9.1
latest-comm-central

l10n directories would be:

latest-comm-1.9.1-l10n
latest-comm-central-l10n

Any other latest-* directories that aren't 1.8 or l10n we kill.

Would we want latest-trunk being a symlink to latest-comm-central? This would be empty for Lightning, but we could always put a readme or additional symlink in that directory for the time being.

We can publicise this a couple of days before we make the change (blogs, md.planning) then everyone will be up to date.
Thunderbird and Lightning builds from 1.9.1 branch are offered at:

 thunderbird/nightly/latest-comm-central/
 calendar/lightning/nightly/latest-comm-central/

 thunderbird/nightly/latest-trunk/
 calendar/lightning/nightly/latest-trunk/

 thunderbird/nightly/latest-comm-1.9.1/
 [no matching lightning folder - create one?]

Thunderbird builds from 1.9.2 branch are offered at:

 thunderbird/nightly/latest-comm-central-trunk/
 [no matching lightning builds]
(In reply to comment #6)
> Thunderbird and Lightning builds from 1.9.1 branch are offered at:
> 
>  thunderbird/nightly/latest-comm-central/
>  calendar/lightning/nightly/latest-comm-central/

I would rather these become "trunk" builds rather than 1.9.1 builds - this would be closer to what Firefox does.

>  thunderbird/nightly/latest-trunk/
>  calendar/lightning/nightly/latest-trunk/

These should definitely become "trunk"/1.9.2 builds if we keep them.

>  thunderbird/nightly/latest-comm-1.9.1/
>  [no matching lightning folder - create one?]

Yep, we should have a lightning folder there.

> > Thunderbird builds from 1.9.2 branch are offered at:
> 
>  thunderbird/nightly/latest-comm-central-trunk/
>  [no matching lightning builds]

I think we shouldn't have the -trunk, then we match the Firefox structure.
Can someone please double-check what the Lightning Nightly Updater relies on ?
The Lightning Nightly Updater (Unofficial) 0.9.081202 extension checks for builds in lightning/nightly/latest-comm-central/.
(In reply to comment #9)
> The Lightning Nightly Updater (Unofficial) 0.9.081202 extension checks for
> builds in lightning/nightly/latest-comm-central/.

Note: we have previously been in contact with the extension author before so with notice I expect that could be changed.
OS: Mac OS X → All
Hardware: x86 → All
This bug has been hanging around too long. We should get this reorganisation done and make it clearer for our nightly testers.

I have therefore put together a proposed set of changes to the Thunderbird, Calendar and Sunbird nightly ftp directories to clarify what I previously put on this bug. The proposal is here:

https://wiki.mozilla.org/User:Standard8/FTP_directory_Reorg

Here's the plan:

- Give it a few days for feedback on this bug.
- Post it to mda.thunderbird/mda.calendar/mda.l10n, at the same time contact the author of the lightning extension (check for feedback, just in case).
- Agree date with gozer.
- Blog about the change & when it will be made.
- Do it.

Also adding dev-doc-needed as we should a) document where builds go for devs & extension users, b) ensure our links are up to date for nightly testers.

So please take a quick look at the wiki page and provide feedback, I'll drive this forward over the next week or so.
Assignee: gozer → bugzilla
URL: https://wiki.mozilla.org/User:Standar...
Component: Build Config → Release Engineering
Keywords: dev-doc-needed
Product: Calendar → Mozilla Messaging
QA Contact: build → release
Version: unspecified → other
Your proposal looks fine to me, I'd value ssitter's opinion on this too though.
Why are those three dirs (win32-xpi/linux-xpi/macosx-xpi) for Lightning created at all? Does this need a change to the buildbot config to stop? btw The proposal for Calendar/Lightning looks good! ;)
(In reply to comment #11)

> - Give it a few days for feedback on this bug.

Looks good to me. Do we have plan to link to these from the website. It's seems difficult to find the nightlies from a google search.
(In reply to comment #13)
> Why are those three dirs (win32-xpi/linux-xpi/macosx-xpi) for Lightning created
> at all? Does this need a change to the buildbot config to stop? btw The
> proposal for Calendar/Lightning looks good! ;)

I assume they were quick-access to the xpis without having to go down into two directories. Seeing as you could potentially have trunk and branch with xpis, having a symlink to one set would be confusing, so we'd need to at least rename them. I thought removing would be easier ;-) (especially as if people need to go there frequently I'd expect them to bookmark the appropriate directory).

All the changes (apart from old removals) need changes to the buildbot config so removing generation of the -xpi links would be part of that.

(In reply to comment #14)
> (In reply to comment #11)
> > - Give it a few days for feedback on this bug.
> 
> Looks good to me. Do we have plan to link to these from the website. It's seems
> difficult to find the nightlies from a google search.

There are a few links about. My current thoughts are that we should have a couple of pages on devmo (one for each app) that point to the various latest-* directories in nightly/ and explain what they are, what they are based on, development details etc. We can then update our links/pointers to devmo.
From Thunderbird perspective, some cleanup of wiki pages would be required, mostly testing related - /latest-comm-1.9.1/ is rarely if ever used.  

~55 UNCO bugs ask to test 3.0 citing nightly urls which are to be deleted
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=MailNews+Core&product=Thunderbird&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=UNCONFIRMED&resolution=---&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&bug_severity=trivial&emailreporter1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=longdesc&type0-0-0=substring&value0-0-0=&field0-1-0=longdesc&type0-1-0=anywordssubstr&value0-1-0=thunderbird%2Fnightly%2Flatest-comm-central+thunderbird%2Fnightly%2Flatest-trunk

~50 confirmed bugs with comments since 2007-04-01
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=MailNews+Core&product=Thunderbird&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&resolution=---&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&bug_severity=trivial&emailreporter1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=longdesc&type0-0-0=changedafter&value0-0-0=2007-04-01&field0-1-0=longdesc&type0-1-0=anywordssubstr&value0-1-0=thunderbird%2Fnightly%2Flatest-comm-central+thunderbird%2Fnightly%2Flatest-trunk&field1-0-0=noop&type1-0-0=noop&value1-0-0=
(In reply to comment #16)
> From Thunderbird perspective, some cleanup of wiki pages would be required,
> mostly testing related - /latest-comm-1.9.1/ is rarely if ever used.  

Hence my previous comments about updating pages/links and having a "central" page explaining it all and linking.

> ~55 UNCO bugs ask to test 3.0 citing nightly urls which are to be deleted

We could delay removing of latest-trunk for, say, 2 weeks but syntactically its wrong to point to latest-trunk anyway, because latest-trunk should really mean 3.1* builds.

> ~50 confirmed bugs with comments since 2007-04-01

I wouldn't be too fussed about these. These seem to be older links in bugs and I think most people can work out to try going up a directory, if not, I'm sure they will post on the bug.
I fully agree with the proposal from comment 11. Go for it!
(In reply to comment #18)
> I fully agree with the proposal from comment 11. Go for it!

Looks good to me too. :)
Summary: clean up ftp.mozilla.org lightning nightly directory to be less confusing → clean up ftp.mozilla.org Lightning/Sunbird/Thunderbird nightly directories to be less confusing
(In reply to comment #18)
> I fully agree with the proposal from comment 11. Go for it!

Same here.
Proposal looks good to me.

But as far as I know currently the initial latest- directory is specified by the buildbot name. For Sunbird it is latest-comm-central-sunbird, for Lightning it is latest-comm-central-calendar.

Or is it now possible to specify the directory independent of the buildbot name?
(In reply to comment #21)
> But as far as I know currently the initial latest- directory is specified by
> the buildbot name. For Sunbird it is latest-comm-central-sunbird, for Lightning
> it is latest-comm-central-calendar.
> 
> Or is it now possible to specify the directory independent of the buildbot
> name?

Hmm interesting point. We may need some buildbot magic, but I'll take a look (or get gozer to).
CCing Oliver Saier, developer of the Lightning Nightly Updater extension.
The buildbot config is currently correct for Thunderbird nightly builds (trunk & 1.9.1 branch). This patch

- Fixes Sunbird build configuration to upload to latest-comm-1.9.1
- Fixes the l10n builds to use latest-<milestone>-l10n. Hence fixing TB & SB l10n builds.

Lightning isn't fixed in this patch - I can't see why it is going to the wrong places because the platform milestone is set to comm-central/linux-xpi (for example) and that doesn't tie up with the current ftp directory.

Also, possible affects to be aware of:

- Name of tinderbox-builds directory will be changed.
- milestone for Sunbird as passed to CreateCompleteUpdateSnippet will be changed. I believe this isn't an issue and will just keep everything in sync.

Once I've spoken to gozer later about a date I'll do a newsgroup post & blog.
IMHO trunk builds ought to be spinned for every application, if only to help find and fix trunk bugs; and <whatever>/nightly/latest-trunk directories ought to exist (if only as symlinks) to point to them.

Comments welcome (by me, and if they aren't welcome as bug comments, send them to my email address, above).
(In reply to comment #25)
> IMHO trunk builds ought to be spinned for every application, if only to help
> find and fix trunk bugs; and <whatever>/nightly/latest-trunk directories ought
> to exist (if only as symlinks) to point to them.

First, the plan allows for trunk directories for all applications. Whether or not each application provides trunk builds is a discussion to be had with the developers of that project and where/how they wish to focus resources.

Secondly, what is your the reason we should provide latest-trunk? Is latest-comm-central-trunk not descriptive enough?
(In reply to comment #26)
[...]
> Secondly, what is your the reason we should provide latest-trunk? Is
> latest-comm-central-trunk not descriptive enough?

It's not so much "being more or less descriptive" as an application of the "principle of least surprise". For as long as I can remember, "bleeding-edge development" nightlies were in nightly/latest-trunk (which may be a symlink). Why change? And if you do, wouldn't it throw most of the testers off their feet? On the day it changes, won't they just imagine "there's something wrong with the nightly builders again" and just stop downloading the nightlies -- which may mean, reduce the numbers of eyes on new builds and therefore the chance of fixing bugs before the product goes final?
(In reply to comment #27)
> (In reply to comment #26)
> [...]
> > Secondly, what is your the reason we should provide latest-trunk? Is
> > latest-comm-central-trunk not descriptive enough?
> 
> It's not so much "being more or less descriptive" as an application of the
> "principle of least surprise". For as long as I can remember, "bleeding-edge
> development" nightlies were in nightly/latest-trunk (which may be a symlink).
> Why change? And if you do, wouldn't it throw most of the testers off their
> feet? On the day it changes, won't they just imagine "there's something wrong
> with the nightly builders again" and just stop downloading the nightlies --
> which may mean, reduce the numbers of eyes on new builds and therefore the
> chance of fixing bugs before the product goes final?

I think testers are generally capable of working out that if we've deleted something (rather than just left it there) then we've changed something.

However, given that latest-trunk has been around for ages, it is probably best to keep it. However, this is where inconsistencies will come in again. So we'll just have to do the best. I'm just about to update the wiki page with:

TB: latest-trunk -> latest-comm-central-trunk
SB: latest-trunk -> latest-comm-1.9.1
Lightning: latest-trunk -> latest-comm-1.9.1

Then they will be pointing to the latest available builds for each of those apps. If people don't like it, we can change it later.
But might cause additional confusion because Lightning latest-trunk won't work with Thunderbird latest-trunk. This was part of the initial motivation for this bug, see also comment #3, #4, #5 above.
(In reply to comment #30)
> But might cause additional confusion because Lightning latest-trunk won't work
> with Thunderbird latest-trunk. This was part of the initial motivation for this
> bug, see also comment #3, #4, #5 above.

I'd forgotten that point. Apart from Tony here, and KaiRo on md.planning (about complaints when SM removed it), I've not had any comments. So I guess we could just delete latest-trunk and see what complaints we get and if we do get any, then we can try and find an appropriate solution or get them to change.
(In reply to comment #31)
> (In reply to comment #30)
> > But might cause additional confusion because Lightning latest-trunk won't work
> > with Thunderbird latest-trunk. This was part of the initial motivation for this
> > bug, see also comment #3, #4, #5 above.
> 
> I'd forgotten that point. Apart from Tony here, and KaiRo on md.planning (about
> complaints when SM removed it), I've not had any comments. So I guess we could
> just delete latest-trunk and see what complaints we get and if we do get any,
> then we can try and find an appropriate solution or get them to change.

I like that approach, personally.
Over to gozer for implementation.
Assignee: bugzilla → gozer
Done, should all be working. I've updated wiki document in comment #11 to reflect the status of what was done.

Note, the lightning builds are the only one I am not yet 100% certain they will turn out okay, so I am still watching them and might need to tweak something further.
ftp://ftp.mozilla.org/pub/calendar/lightning/nightly/README.html

contains an invalid link to the previous ftp folder structure.
Simon, could you take care of the lightning links?
(In reply to comment #35)
> Once finished we should update the download links on the following pages too:
> http://www.mozilla.org/projects/calendar/sunbird/download.html#nightly
> http://www.mozilla.org/projects/calendar/lightning/download.html#nightly
> http://weblogs.mozillazine.org/calendar/

I fixed those and the link in the readme file on FTP.
Blogged about it as well.
I've just been checking builds. Summary so far:

Thunderbird & Sunbird l10n builds are broken - cannot download the en-US files because they are in a different place now.

Thunderbird en-US all seem to be heading to the right place (windows/mac trunk not yet spun, others all ok).

Sunbird en-US all ok.

Lightning en-US Linux has spun and is in the right place.
Keywords: dev-doc-needed → dev-doc-complete
According to the reorg page:

https://wiki.mozilla.org/User:Standard8/FTP_directory_Reorg

The lightning <os>-xpi folder symlinks should no longer exist, yet it looks like they have been regenerated (they were previously deleted).  As such, the Lightning "Status" is not correct on that page.

Also, fyi, the README.html still has the broken link in it... perhaps the change has yet to be pushed out?
(In reply to comment #42)
> According to the reorg page:
> 
> https://wiki.mozilla.org/User:Standard8/FTP_directory_Reorg
> 
> The lightning <os>-xpi folder symlinks should no longer exist, yet it looks
> like they have been regenerated (they were previously deleted).  As such, the
> Lightning "Status" is not correct on that page.

I've updated the Status to show this is not done yet. It's a side-effect of the stage uploading buildbot code, and I haven't found an elegant way to disable it yet. (I am abusing a feature to upload XPIs in the first place)

> Also, fyi, the README.html still has the broken link in it... perhaps the
> change has yet to be pushed out?

I am not sure what the source of that README.html file is (from source-control) ? Otherwise, I have access to that file on ftp.m.o and could hand-edit it.
Depends on: 499060
(In reply to comment #42)
> Also, fyi, the README.html still has the broken link in it... perhaps the
> change has yet to be pushed out?

It had been pushed out, but I had pushed out the wrong change. I fixed that now. The readme.html file with the correct link should show up on FTP within the next 30 minutes.
I'm declaring this bug as fixed - we did the major work a long time ago. Although Lightning still has the -xpi links I suggest we tidy those up whenever Lightning trunk builds are set up.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
FTR bug 499060 already exists to remove those -xpi links.
No longer depends on: 499060
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: