[tracker] Build a system to publish content without engineering intervention

RESOLVED FIXED

Status

Websites
Nucleus
P3
normal
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: hoosteeno, Assigned: hoosteeno)

Tracking

Details

Attachments

(1 attachment)

There are a handful of content types on mozilla.org/mozilla.com that can or must be published by the content owners outside of the standard Bedrock engineering workflow (PR, review, merge, deploy). Some examples:

* http://www.mozilla.org/security/announce/ (etc.)
* http://www.mozilla.org/en-US/firefox/21.0beta/releasenotes/ (etc.)

There are others.

This bug will track efforts to enable these publishing workflows in a lightweight, decoupled manner.
Depends on: 869495
Depends on: 874202
Blocks: 801909
Priority: -- → P3
Blocks: 818316
Created attachment 753980 [details]
Diagram of Loosid publishing system

Attaching a high-level, general diagram depicting the major system components of a proposed publishing system that might accommodate various workflows (known and unknown). The workflows themselves are captured in blocking bugs; we continue to discover new ones.
Blocks: 876810
Blocks: 885797
Pmac, I'm moving the conversation we had this morning and later in IRC here for now. We have two proposed approaches to publishing content on mozilla.org, which are described and evaluated in this etherpad:

https://webprod.etherpad.mozilla.org/publishing-system-brainstorm

Since the consumer face should be unchanged by this decision, the two biggest stakeholder groups are the people who have to maintain it and the people who have to use it. Can you represent the former group and let us know whether you'd prefer to hack on and manage a homegrown, loosely-coupled content synchronization suite (earlier described in comment 1), or instead to build out bedrock to use more of django's core functionality (database layer, authorization, data management interfaces, and more models)? Why do you prefer the one you prefer? 

I will seek input from several of the content owners and report back.
Flags: needinfo?(pmac)
We've continued the conversation even more since, and have basically decided that we'll need to do both most likely. Most new implementations and some existing ones with inadequate tooling will likely use a database backend. The more established scripted workflows will likely stick with generated files checked into version control and consumed by a cron script on bedrock in much the same way that product details is now.

There are many details to be worked out for a database system; only some of which are l10n, dev and stage tier synchronization, access control, approval flows, and preview mode. So I believe the non-db direction will be easier to implement and maintain, but the DB does provide advantages for flexibility and control by us.
Flags: needinfo?(pmac)
In light of comment 3, I suggest we proceed with building the system roughly outlined in comment 1, since we already have some buy-in to implement such a system for nearly half of the legacy publishing flows. We can consider a database solution for any/all publishing flows.
Blocks: 896474
No longer blocks: 818316
Depends on: 909845
This system will be called "Nucleus" and has its own component. We're working on articulating the major elements of the system in preparation for coding in 2013Q4.
Component: Bedrock → Nucleus
Product: www.mozilla.org → Websites
Depends on: 925553
(responding here to discussion on bug 925553)

Justin: Curtis and I would like to discuss Nucleus at a higher level: data and risk, mostly. 
What we need is a 30 minutes meeting to get a feel of how critical this project is, from a business point of view:
 * Who owns this project?
 * Are there any PR/financial/legal/operational risks?
 * Set some expectations on data confidentiality, availability and auditability.
We are trying something new here, to help you go through the secreviews faster. The idea is to do this early on, so we can pick the right infrastructure and security measures at the start, instead of doing it right before you go live.

Can we meet on vidyo in the next couple of weeks?
Flags: needinfo?(hoosteeno)
Invite resent :)
Flags: needinfo?(hoosteeno)
(In reply to Curtis Koenig [:curtisk] from comment #8)
> adding stefan for appsec

scratch that it's rforbes
(In reply to Curtis Koenig [:curtisk] from comment #9)
> (In reply to Curtis Koenig [:curtisk] from comment #8)
> > adding stefan for appsec
> 
> scratch that it's rforbes

Scratch that again, this is going to go to Adam (:adamm) as he has experience with bedrock.
Justin - can you add him to the meeting please?
Flags: needinfo?(hoosteeno)
done.
Flags: needinfo?(hoosteeno)
Thanks for taking the time to meet today!

Risk review notes: https://wiki.mozilla.org/Security/Reviews/Nucleus
tldr: Nucleus is a publishing system for Bedrock. It's risk level is P-1 A-2 R-2 Au-2. The system is suitable to run in PaaS, as long as no confidential information is stored in it (ex: unreleased security advisories).
Justin: did you get a chance to clear this action item from the meeting?
"hoosteeno ask lsblakk about auditability of changes"
Flags: needinfo?(hoosteeno)
Julien, I did. :lsblakk agrees that the need for auditability is low with a trusted group of content managers (which we'll have) and, if needed, can be sufficiently satisfied by logs of authentications (which we'll have). Thanks for following up!
Flags: needinfo?(hoosteeno)
Excellent. Thanks for the quick reply. I've updated the wiki page at https://wiki.mozilla.org/Security/Reviews/Nucleus
Blocks: 939134
Depends on: 964971
Depends on: 964985

Updated

4 years ago
No longer blocks: 885797
No longer depends on: 874202
In early 2014 we built and launched Nucleus, which is described on the wiki:

https://wiki.mozilla.org/Websites/Mozilla.org/Publishing

This bug is resolved. Thanks to everyone who contributed!
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.