874137 - [tracker] Build a system to publish content without engineering intervention

Status: RESOLVED FIXED

(Websites :: Nucleus, defect, P3)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

There are a handful of content types on mozilla.org/mozilla.com that can or must be published by the content owners outside of the standard Bedrock engineering workflow (PR, review, merge, deploy). Some examples:

* http://www.mozilla.org/security/announce/ (etc.)
* http://www.mozilla.org/en-US/firefox/21.0beta/releasenotes/ (etc.)

There are others.

This bug will track efforts to enable these publishing workflows in a lightweight, decoupled manner.

Comment 1

[Justin Crawford [:hoosteeno] [:jcrawford]]

Attachment: Diagram of Loosid publishing system

Attaching a high-level, general diagram depicting the major system components of a proposed publishing system that might accommodate various workflows (known and unknown). The workflows themselves are captured in blocking bugs; we continue to discover new ones.

Comment 2

[Justin Crawford [:hoosteeno] [:jcrawford]]

Pmac, I'm moving the conversation we had this morning and later in IRC here for now. We have two proposed approaches to publishing content on mozilla.org, which are described and evaluated in this etherpad:

https://webprod.etherpad.mozilla.org/publishing-system-brainstorm

Since the consumer face should be unchanged by this decision, the two biggest stakeholder groups are the people who have to maintain it and the people who have to use it. Can you represent the former group and let us know whether you'd prefer to hack on and manage a homegrown, loosely-coupled content synchronization suite (earlier described in comment 1), or instead to build out bedrock to use more of django's core functionality (database layer, authorization, data management interfaces, and more models)? Why do you prefer the one you prefer? 

I will seek input from several of the content owners and report back.

Comment 3

[Paul [:pmac] McLanahan]

We've continued the conversation even more since, and have basically decided that we'll need to do both most likely. Most new implementations and some existing ones with inadequate tooling will likely use a database backend. The more established scripted workflows will likely stick with generated files checked into version control and consumed by a cron script on bedrock in much the same way that product details is now.

There are many details to be worked out for a database system; only some of which are l10n, dev and stage tier synchronization, access control, approval flows, and preview mode. So I believe the non-db direction will be easier to implement and maintain, but the DB does provide advantages for flexibility and control by us.

Comment 4

[Justin Crawford [:hoosteeno] [:jcrawford]]

In light of comment 3, I suggest we proceed with building the system roughly outlined in comment 1, since we already have some buy-in to implement such a system for nearly half of the legacy publishing flows. We can consider a database solution for any/all publishing flows.

Comment 5

[Justin Crawford [:hoosteeno] [:jcrawford]]

This system will be called "Nucleus" and has its own component. We're working on articulating the major elements of the system in preparation for coding in 2013Q4.

Comment 6

[Julien Vehent]

(responding here to discussion on bug 925553)

Justin: Curtis and I would like to discuss Nucleus at a higher level: data and risk, mostly. 
What we need is a 30 minutes meeting to get a feel of how critical this project is, from a business point of view:
 * Who owns this project?
 * Are there any PR/financial/legal/operational risks?
 * Set some expectations on data confidentiality, availability and auditability.
We are trying something new here, to help you go through the secreviews faster. The idea is to do this early on, so we can pick the right infrastructure and security measures at the start, instead of doing it right before you go live.

Can we meet on vidyo in the next couple of weeks?

Comment 7

[Justin Crawford [:hoosteeno] [:jcrawford]]

Invite resent :)

Comment 8

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

adding stefan for appsec

Comment 9

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

(In reply to Curtis Koenig [:curtisk] from comment #8)
> adding stefan for appsec

scratch that it's rforbes

Comment 10

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

(In reply to Curtis Koenig [:curtisk] from comment #9)
> (In reply to Curtis Koenig [:curtisk] from comment #8)
> > adding stefan for appsec
> 
> scratch that it's rforbes

Scratch that again, this is going to go to Adam (:adamm) as he has experience with bedrock.
Justin - can you add him to the meeting please?

Comment 11

[Justin Crawford [:hoosteeno] [:jcrawford]]

done.

Comment 12

[Julien Vehent]

Thanks for taking the time to meet today!

Risk review notes: https://wiki.mozilla.org/Security/Reviews/Nucleus
tldr: Nucleus is a publishing system for Bedrock. It's risk level is P-1 A-2 R-2 Au-2. The system is suitable to run in PaaS, as long as no confidential information is stored in it (ex: unreleased security advisories).

Comment 13

[Julien Vehent]

Justin: did you get a chance to clear this action item from the meeting?
"hoosteeno ask lsblakk about auditability of changes"

Comment 14

[Justin Crawford [:hoosteeno] [:jcrawford]]

Julien, I did. :lsblakk agrees that the need for auditability is low with a trusted group of content managers (which we'll have) and, if needed, can be sufficiently satisfied by logs of authentications (which we'll have). Thanks for following up!

Comment 15

[Julien Vehent]

Excellent. Thanks for the quick reply. I've updated the wiki page at https://wiki.mozilla.org/Security/Reviews/Nucleus

Comment 16

[Justin Crawford [:hoosteeno] [:jcrawford]]

In early 2014 we built and launched Nucleus, which is described on the wiki:

https://wiki.mozilla.org/Websites/Mozilla.org/Publishing

This bug is resolved. Thanks to everyone who contributed!