896474 - Port /about/forums pages from SVN to bedrock and/or other locations

Status: RESOLVED FIXED

(www.mozilla.org :: Bedrock, defect, P2)

Description

[Steven Garrity [:sgarrity]]

The only files left in the /about dir on the old mozilla.org SVN repo are in the /about/forums dir.

We need to determine which files are still relevant, and where they should be ported.

The files are:

http://www.mozilla.org/about/forums/index.html
http://www.mozilla.org/about/forums/etiquette.html
http://www.mozilla.org/about/forums/cancellation.html

There are also three Perl scripts that live the /about/forums dir (checklist, commit, and gen-ng-info). These seem to be scripts that grab the list of forums from news.mozilla.org, dump them to a text file, and generates the list for the /about/forums page.

The etiquette.html and cancellation.html pages seem like they could live under http://mozilla.org/about/governance/policies/ but it might also make sense for them to stay with the overview page (index.html) where ever it ends up.

So, Forums page (http://www.mozilla.org/about/forums/). Should it be ported to Bedrock as is? Does it belong elsewhere?

Comment 1

[Gervase Markham [:gerv]]

The way this works is that groups are added to raw-ng-list.txt, with a classification (indicated by position), a name and a description. "gen-ng-info" turns that raw list into some HTML with appropriate subscription links etc. "commit" runs gen-ng-info, uses a script called "replace-between" to stick the generated HTML into index.html, and then checks all changed files in to the SVN repo.

"checklist" grabs the list of newsgroups from news.mozilla.org to check and see if the raw-ng-list.txt file has got out of date compared with reality. So when someone asks me to update the list, I run checklist and fix all the problems it finds.

I'm certainly open to other ways of doing this which do not require more effort. 

Haven't we had this discussion before by email? Jennifer?

Gerv

Comment 2

[Jennifer Bertsch [:jbertsch]]

Hi Gerv and Steven-

Yes - we've talked about this previously.

Justin is working on a publishing solution that will allow us to publish the about/forums content on Bedrock.

Thx,
Jen

Comment 3

[Jennifer Bertsch [:jbertsch]]

Moving this one to the Icebox on Kanban until the legacy publishing system is ready to go

Comment 4

[Jennifer Bertsch [:jbertsch]]

Hi Gerv-

We'd like to propose the a new work flow to publish the forums index, etiquette, and cancellation pages directly to Bedrock and remove it from the SVN repo:

1.  Content owner will file a bug with information about the requested updates to the page
2.  Content owner or Bedrock developer will make the update in Github and file a pull request
3.  Another Bedrock developer will review the PR
4.  Updates will be published on Bedrock

Will this work for these pages, Gerv?

Thx,
Jen

Comment 5

[Gervase Markham [:gerv]]

Hi Jennifer,

I was under the impression that you were developing a self-publishing system for Bedrock, and that this is what we've been waiting for to take action on this bug. See comment 2. Does that system now exist, or not?

Having to file bugs and make Github PRs which someone else has to approve is really not an ideal workflow for me, particularly if something better is actually available... The etiquette and cancellation pages rarely change, so it's not so much of an issue, but the forums index gets updated regularly.

Gerv

Comment 6

[Jennifer Bertsch [:jbertsch]]

Hi Gerv-

We did develop a self-publishing system to publish Firefox release notes, but it isn't a turn-key solution and would require not-insignificant back-end development hours to be applied to the Forums and Credits workflow (I'll comment separately in the Credits bug).

In other words, nothing better is readily available.  We do publish almost daily on www.mozilla.org, so we can guarantee a pretty quick turn around.  And we  could incorporate the Forums script into the Git process if that would help you.

I think we need to move forward in this direction.

Thx,
Jen

Comment 7

[Chris More [:cmore]]

(In reply to Jennifer Bertsch [:jbertsch] from comment #6)
> Hi Gerv-
> 
> We did develop a self-publishing system to publish Firefox release notes,
> but it isn't a turn-key solution and would require not-insignificant
> back-end development hours to be applied to the Forums and Credits workflow
> (I'll comment separately in the Credits bug).
> 
> In other words, nothing better is readily available.  We do publish almost
> daily on www.mozilla.org, so we can guarantee a pretty quick turn around. 
> And we  could incorporate the Forums script into the Git process if that
> would help you.
> 
> I think we need to move forward in this direction.
> 
> Thx,
> Jen

What if Gerv just runs his normal perl scripts and the output of the script is what is submitted via Github's pull request? This will require no extra magic than we have now and using the same workflow [1] that everyone in the world uses for contributing to mozilla.org. If we have an existing workflow for content updates and contributions on bedrock that is good enough for any contributor (paid or volunteer), why would we have anything more complex than that for the forums page?

The self-publishing system [2] requires substantial technical investment to set it up and only make sense for updates that are frequent and with many changes each publish. The ROI of setting up Nucleus for the forum updates will be negative for quite a while because the updates appear to only happen every month or two. Firefox release notes uses Nucleus because updates happen every 6 weeks and 100% of the content changes each time.

[1] https://bedrock.readthedocs.org/en/latest/contribute.html
[2] https://wiki.mozilla.org/Websites/Mozilla.org/Publishing

Comment 8

[Jennifer Bertsch [:jbertsch]]

+pmac and jgmize

Comment 9

[Jennifer Bertsch [:jbertsch]]

We are going to begin work moving the output into Github and the templates into Bedrock in the next two weeks.

Steven - I'll ping you about getting starting.

Thx,
Jen

Comment 10

[Gervase Markham [:gerv]]

I'm sad that it's possible in 2014 to set up a copy of Wordpress in 5 minutes and yet the combined might of our web team can't come up with a self-publish solution for our main website which doesn't require a lot of hassle for them to set up for each new set of pages which want to use it :-|

If we can do what Chris More says in comment 7 (let me pull a git repo which I can run my scripts to update, and then submit a pull request), then that's OK.

Gerv

Comment 11

[Jennifer Bertsch [:jbertsch]]

(In reply to Gervase Markham [:gerv] from comment #10)
> I'm sad that it's possible in 2014 to set up a copy of Wordpress in 5
> minutes and yet the combined might of our web team can't come up with a
> self-publish solution for our main website which doesn't require a lot of
> hassle for them to set up for each new set of pages which want to use it :-|
> 
> If we can do what Chris More says in comment 7 (let me pull a git repo which
> I can run my scripts to update, and then submit a pull request), then that's
> OK.
> 
> Gerv

We will do what Chris More suggests in comment 7.

Comment 12

[Gervase Markham [:gerv]]

Fantastic :-)

Gerv

Comment 13

[Steven Garrity [:sgarrity]]

Gerv: Do you have a preference between these two options?

A. You provide output from your gen-ng-info script (including HTML tables, etc.) and we include it in our template, or;

B. You provide raw-ng-list.txt and we move the HTML-ification from gen-ng-info into bedrock

I don't feel strongly either way. Just taking your output from gen-ng-info (HTML and all) is probably the quicker option.

Comment 14

[Gervase Markham [:gerv]]

I don't have a preference. Happy to provide HTML if that's easier for you guys. And it gives me more flexibility to update the list format if things change.

Gerv

Comment 15

[Steven Garrity [:sgarrity]]

Here's a PR to add the the template for /about/forums/ to bedrock: https://github.com/mozilla/bedrock/pull/2116

This will still need to include the generated HTML from Gerv's script somehow. Gerv, it would also be helpful if the tables in your output could get the CSS class "table" added. I could work around that with the output you have, but it would be cleaner to just add the CSS class.

Next, I'll update the PR to add the etiquette and cancellation pages, and the necessary redirects.

Comment 16

[Paul [:pmac] McLanahan]

Can we really not just include the current output and then update the HTML in PRs after that? It's an infrequently changing list of things. Coding automation for this seems silly. It'd also be MUCH easier to do l10n (if that's required) if the strings were in the template.

If we can't do that, then it looks like we can do something similar to what we've done to credits, which is to consume the data file (raw-ng-list.txt) and provide the data to the template context where we'll handle the HTML (like :sgarrity mentioned in comment #13 option B). I'd much rather have the HTML in our templates so that we can control styling issues like that mentioned in comment #15. I'm much more trusting of formatted data I can verify than HTML that we'd have to parse and sanitize.

This leaves l10n. Are we planning on translating this page (forums)?

Comment 17

[Gervase Markham [:gerv]]

I'd be fine with you guys working directly with raw-ng-list.txt if you prefer.

The auto-generated bit won't be translated. The header could be, although perhaps we should add a warning that, except in regional groups, the standard language is English.

Gerv

Comment 18

[Jennifer Bertsch [:jbertsch]]

We do not need L10N for this page.  Thx!

Comment 19

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/67686be27b63c5e961454020c53eaec3be627b04
Add templates for /about/forums port for Bug 896474

https://github.com/mozilla/bedrock/commit/98c455b6fe15dfb5884c069e06bc6972817c96c9
Bug 896474: Generate forums from data file in SVN.

https://github.com/mozilla/bedrock/commit/19e6cd5548d761ea8964ebb7e8cf9dbd22714d17
Output forum list with simpler markup Bug 896474

Comment 20

[Steven Garrity [:sgarrity]]

Attachment: bug-896977-remove-about.diff

Comment 21

[Steven Garrity [:sgarrity]]

(In reply to Steven Garrity [:sgarrity] from comment #20)
> Created attachment 8498869 [details] [diff] [review]
> bug-896977-remove-about.diff

This patch removes the remaining /about/forums/ files, now that these pages are in bedrock.

Comment 22

[Paul [:pmac] McLanahan]

Comment on attachment 8498869 [details] [diff] [review]
bug-896977-remove-about.diff

Review of attachment 8498869 [details] [diff] [review]:
-----------------------------------------------------------------

Some of these files are still the source of data for the new pages in bedrock. Gerv maintains naw-ng-list.txt and bedrock gets it out of SVN from this location. We should check into moving it elsewhere so that we can continue cleaning up this directory, but I know at least the following files are scripts and may be in use:

gen-ng-info
commit
checklist
replace-between
raw-ng-list.txt

We should discuss with Gerv moving what he wants of this stuff to a new and more permanent location in the SVN repo and let him clean up the rest.

Comment 23

[Jennifer Bertsch [:jbertsch]]

Hi Gerv-

Do you want to discuss this in the bug (see above comment) or should I try to find time to meet on vidyo?

Thx,
Jen

Comment 24

[Gervase Markham [:gerv]]

I don't think this needs a Vidyo meeting. I'm happy for this info to move elsewhere in SVN or even to its own repo if you just tell me where it's gone.

If Bedrock works directly on raw-ng-list.txt then we only need the following files to remain:

raw-ng-list.txt
checklist

Gerv

Comment 25

[Steven Garrity [:sgarrity]]

Gerv: Do you have a preference where this would end up? Do you still want it in SVN, or would a github repo work? Could it just live directly in bedrock?

Comment 26

[Gervase Markham [:gerv]]

Somewhere where I can check in without having to file a pull request and wait would be strongly preferred. SVN is fine. Github is fine if it's now OK by policy to throw random repos up there.

Gerv

Comment 27

[Jennifer Bertsch [:jbertsch]]

Hi Pmac-

Please review comments 22-26, and let's brainstorm a location.

Thx,
Jen

Comment 28

[Paul [:pmac] McLanahan]

I'd suggest keeping it in SVN. It's working fine. We should just move it to a new folder. I don't really see a good place for such a thing in the existing hierarchy though. Perhaps a new top level "data" folder? 


/data/forums/raw-ng-list.txt

and

/data/credits/names.csv


Alternately data like this could be considered a "project" and go somewhere like...

/projects/mozorg_data/credits/names.csv
/projects/mozorg_data/forums/raw-ng-list.txt

Either is fine with me. Once we move it's a simple change in bedrock settings.

Comment 29

[Jennifer Bertsch [:jbertsch]]

(In reply to Paul McLanahan [:pmac] from comment #28)
> I'd suggest keeping it in SVN. It's working fine. We should just move it to
> a new folder. I don't really see a good place for such a thing in the
> existing hierarchy though. Perhaps a new top level "data" folder? 
> 
> 
> /data/forums/raw-ng-list.txt
> 
> and
> 
> /data/credits/names.csv
> 
> 
> Alternately data like this could be considered a "project" and go somewhere
> like...
> 
> /projects/mozorg_data/credits/names.csv
> /projects/mozorg_data/forums/raw-ng-list.txt
> 
> Either is fine with me. Once we move it's a simple change in bedrock
> settings.

Hi Gerv-

Do you have a preference between either of these locations?

Thx,
Jen

Comment 30

[Paul [:pmac] McLanahan]

Note that either of these locations would require someone with more privileges in SVN than I to create them and move things. I only have write permissions in projects/mozilla.com.

Comment 31

[Jennifer Bertsch [:jbertsch]]

(In reply to Paul McLanahan [:pmac] from comment #30)
> Note that either of these locations would require someone with more
> privileges in SVN than I to create them and move things. I only have write
> permissions in projects/mozilla.com.

Noted.  Maybe sgarrity does? :)  If not, we'll find someone else.

Comment 32

[Gervase Markham [:gerv]]

I have no preference as to the location, as long as I have write permissions there :-))

Gerv

Comment 33

[Paul [:pmac] McLanahan]

This is long done :) I'm closing this and opening a new bug for moving the forums data and scripts to a new git repo somewhere since SVN will be shutting down next year.