996570 - Create a data store for compatibility data

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: BrowserCompat, defect)

Description

[Jeremie Patonnier :Jeremie]

The first step to the compatibility data project is to build a data base to store the compatibility data.

Comment 1

[Jeremie Patonnier :Jeremie]

Discussion on the data model started on the dev-mdn mailing list: https://groups.google.com/d/msg/mozilla.dev.mdn/RX4Vx4xacZw/PhjIy1MAyPcJ

The discussion can continue here. Here's my comments based on this proposed data model: https://docs.google.com/a/mozilla.com/document/d/1YF7GJ6kgV5_hx6SJjyrgunqznQU1mKxp5FaLAEzMDl4/ 


 * Can the spec link associated with a feature be a collection of links?

Many feature are defined by more than one specification. See the first note on: https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Specification 
https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#URLs 


 * Must the technology associated with a feature be deduce from the spec links? If it's the case, how does it work with the previous point?

The requirements clearly state that we need to be able to display the technology supporting a given feature to avoid any name collision or confusion. See:
https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Technologies 


 * How to handle implementation notes at different levels? Shouldn't be better to have implementation notes being independent in the data model?

The current model attach implementation notes to the Browser_Version-feature level but we need to be able to write implementation notes at various level. See: 
https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Implementation_notes 


 * How to handle localized of content?

The current data model does not make obvious how to handle the data that needs to be localized. See: 
https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Localisable_contents 


 * Can a Feature be associated with several Feature_Sets?

It is not formally asked within the requirements but that will be very useful. 


 * Would it be possible to add a prefix entry to the Browser_Version-Feature model?

It's not very clear in the requirement but a prefix is specific to a given feature for a given Browser_Version. There is no easy rule of thumb to deduce a prefix so it will be better if we can record it in parallel of the support (rather than within the support as a prefixed version can provide various level of support independently from the prefix). 

Furthermore, it would be nice if the prefix could be a collection of string as a single browser for a given feature can support several prefix.


Luke, Les, Do you have further comments? What do we need to move forward?

Comment 2

[Jeremie Patonnier :Jeremie]

Another question:

* How data versioning is handled with this schema?

To prevent mistake or vandalism we need to be able to revert our data to a previous state. I'm unsure it's possible with the current proposed schema. See:
https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Dealing_with_vandalism_and_mistakes

Comment 3

[Luke Crouch [:groovecoder]]

I dug into this more and I really like the WebPlatform browser compat data model here:

https://webplatform.github.io/browser-compat-model/

Jeremie - do you see anything missing from that model?

Les - is it reasonable to map this data model into a MySQL schema?

Comment 4

[Les Orchard [:lorchard]]

I think I might understand the spec enough to produce a revised data model:

https://etherpad.mozilla.org/b4YZ3PbNTK

Some open questions for clarification on the spec:

* Does this new data model represents all items described in the spec, along with the associated relationships that will be used to build collections?

* :jezdez (or anyone else)
    * Need feedback on a localization scheme. 
    * Need ideas on if/how the change tracking model can work.

* Added a link field to many models, as a way to link to wiki pages with more information. Is this useful?

* Rather than have notes and a flat list of URLs associated with a Specification - would it be better to just link to a wiki page with more free-form content? Don't want to reinvent a less-featured wheel, here.

* Does this data really need to have overall revision control support? Or is reversible changes really what's needed? (There's a subtle difference, IMO)

* KumaScript and localization are described together in the specification
    * But, neither depends on the other - these need to be considered separately.
    * What are the exact use cases expected for KumaScript support?
        * Do the names of browsers, features, technologies, and specifications really need to support KumaScript macros?
        * Could the addition of a link field to many models be sufficient? i.e. because they can link to wiki pages that support KumaScript.

Comment 5

[Les Orchard [:lorchard]]

(In reply to Luke Crouch [:groovecoder] from comment #3)
> I dug into this more and I really like the WebPlatform browser compat data
> model here:
> 
> https://webplatform.github.io/browser-compat-model/
> 
> Jeremie - do you see anything missing from that model?
> 
> Les - is it reasonable to map this data model into a MySQL schema?

That data model is missing quite a lot of what's in our spec. It also seems to assume that versioning can be sanely broken down for browser, OS, and device - which I don't think is really safe to assume.

Comment 6

[Jeremie Patonnier :Jeremie]

(In reply to Luke Crouch [:groovecoder] from comment #3)
I will follow Les on that, many things are quite hard to achieve with that model. From my perspective, my biggest concerns are about localisation (it is not taken into account), aside notes (there is none) and prefix support (it's to limited).

Versioning is also a big gap (but this model is clearly made as an exchange model, not a storage model)

(In reply to Les Orchard [:lorchard] from comment #5)
> It also seems
> to assume that versioning can be sanely broken down for browser, OS, and
> device - which I don't think is really safe to assume.
+1

(In reply to Les Orchard [:lorchard] from comment #4)
> I think I might understand the spec enough to produce a revised data model:
> https://etherpad.mozilla.org/b4YZ3PbNTK

I also put my comments in there and add comments about the schema itself :)

> Some open questions for clarification on the spec:
> 
> * Does this new data model represents all items described in the spec, along
> with the associated relationships that will be used to build collections?

That looks good to me but I will take the time to do a full check by the end of the week. 

> * Added a link field to many models, as a way to link to wiki pages with
> more information. Is this useful?

Yes, it can be a valid way to handle aside content such as note. Even better, as it is bound to the wiki and all it's features, it makes easy to localise and version the content, and also allow to use the whole power of KumaScript to use link macro (such as domsxref) or build specific macro (I'm pretty sur Florian will be able to imagine something about this) :)

> * Rather than have notes and a flat list of URLs associated with a
> Specification - would it be better to just link to a wiki page with more
> free-form content? Don't want to reinvent a less-featured wheel, here.

Yes, see above. This will require some fine tuning to make contribution as easy as possible, but I'll discuss the contribution work flow with the UX guys then we'll iterate with you. But in a technical point of view, it makes total sens.

> * Does this data really need to have overall revision control support? Or is
> reversible changes really what's needed? (There's a subtle difference, IMO)

I'm not sure to see the difference. What we need is a way to reverse mistakes and fight again vandalisme. The former require just an undo ability but the latter requires that we trace the user in order to discuss with him and eventually ban him if necessary. I'm also a bit concerne as we could see a mistake long after it occurred and it could required to undo more than once if some users tried to correct it but with less complete data than previously.

> * KumaScript and localization are described together in the specification
>     * But, neither depends on the other - these need to be considered separately.

Okay, we only needs to be able to use kumascript within raw content (such as compatibility notes) and those raw contents always need to be localizable. That say, yes, this is an orthogonal subject.

>     * What are the exact use cases expected for KumaScript support?
>         * Do the names of browsers, features, technologies, and
> specifications really need to support KumaScript macros?

No, they don't, only raw content needs to be able to use kumascript.

>         * Could the addition of a link field to many models be sufficient?
> i.e. because they can link to wiki pages that support KumaScript.

It looks like so. KumaScript is only necessary for unstructured content such as implementation notes and, yes, such content can be hosted directly in the wiki with links to it.

I like the idea to host the compatibility data raw content on the wiki but wonder how to handle such content inside the URL tree (this is not a technical issue, just something I need to think about, again, discussion with UX should clarify things on that matter).

Comment 7

[Luke Crouch [:groovecoder]]

In the interest of simplifying as much as possible ...

+1 to separate the structured data in this data store from script-able content via a 'notes' field with a url of an MDN wiki page.

Can the Feature-Version notes also be a url to an MDN wiki page?

And if these fields are a url to an MDN wiki page, does it need to be localized, or can we do locale-agnostic links and use locale-detection when someone visits?

If that's the case, the only localized fields left are:

* Browser name
* Technology name
* Feature name
* Specification title

Do we really need to translate these things? If not, we can avoid the overhead of the Localizations model.

To trace per-user, field-level edits over time we would have to build revision control into the system which adds another 3 data models. :/ Can we ignore the audit trail and just maintain a single set of data? If so, we can avoid the overhead of Change Log, Change Set, and Change Set-Change Log.

Comment 8

[Les Orchard [:lorchard]]

(In reply to Luke Crouch [:groovecoder] from comment #7)

> Can the Feature-Version notes also be a url to an MDN wiki page?

The links are meant for diving deeper for more information, navigating away from the compatibility display.

But, that doesn't work if we want to display notes in-context as tool-tips or asides.

For example, lots of little inline notes here - https://developer.mozilla.org/en-US/docs/Web/CSS/display#Browser_compatibility

We *could* possibly use links to sections of wiki pages, and use the ?raw API to extract the text. But, that might just add unnecessary complexity.

> And if these fields are a url to an MDN wiki page, does it need to be
> localized, or can we do locale-agnostic links and use locale-detection when
> someone visits?

I don't generally trust links to MDN without an explicit locale. They're kind of an emergency fallback measure, and don't always work.

> Do we really need to translate these things? If not, we can avoid the
> overhead of the Localizations model.

Yes, if we don't need localization at all, then we don't need the models. We also wouldn't need a localization UI. But, I'd be surprised if it turns out we don't need localization at all.

> Can we ignore the audit trail and just maintain a single set of data? If so, we can
> avoid the overhead of Change Log, Change Set, and Change Set-Change Log.

Without an audit trail, there's no undo or tracking of vandalism. Unless there's a different way to get those features, of course.

Comment 9

[Jeremie Patonnier :Jeremie]

(In reply to Les Orchard [:lorchard] from comment #8)
> (In reply to Luke Crouch [:groovecoder] from comment #7)
> 
> > Can the Feature-Version notes also be a url to an MDN wiki page?
> 
> The links are meant for diving deeper for more information, navigating away
> from the compatibility display.
> 
> But, that doesn't work if we want to display notes in-context as tool-tips
> or asides.
> 
> For example, lots of little inline notes here -
> https://developer.mozilla.org/en-US/docs/Web/CSS/
> display#Browser_compatibility

I opened bug 1006030 to ask UX to work on this in order to get clear idea about what is the best presentation for those notes in various contexts

> We *could* possibly use links to sections of wiki pages, and use the ?raw
> API to extract the text. But, that might just add unnecessary complexity.

But maybe this complexity is less important than reinventing the wheel for localization and all the wiki content feature we want to use in those aside notes.

> > And if these fields are a url to an MDN wiki page, does it need to be
> > localized, or can we do locale-agnostic links and use locale-detection when
> > someone visits?
> 
> I don't generally trust links to MDN without an explicit locale. They're
> kind of an emergency fallback measure, and don't always work.

+1

> > Do we really need to translate these things? If not, we can avoid the
> > overhead of the Localizations model.
> 
> Yes, if we don't need localization at all, then we don't need the models. We
> also wouldn't need a localization UI. But, I'd be surprised if it turns out
> we don't need localization at all.

Yes, they all need to be localizable:
* Browser name : Local marketing and usage can implied different names
* Technology name : It depend on the technology but some of the new technologies introduce by HTML5 that are not technical acronym are used to be localized.
* Feature name : Same as above
* Specification title : Most likely localized

Jean-Yves, Florian, any extra though about this?

But remind that, in a contributor point of view, localization is optional and if it's not localized we will always fall back to the English version so it could be possible to ease the model by making the English mandatory and all the rest optional.

Comment 10

[Jannis Leidel [:jezdez]]

Les, I would strongly recommend going with a out of the box solution like django-hvad (http://django-hvad.readthedocs.org/en/latest/) to handle the localization bits. It has a great API for the definition of the models, querying and adding items, supports South and is well maintained. Check out the quickstart to get a feeling for it: http://django-hvad.readthedocs.org/en/latest/public/quickstart.html

Lot's of people have thought about translatable content before and I've in the past seen projects go totally up in flames because they tried to reinvent things. There are quite a few options to chose in Django land, so please check them out first before doing this by hand (https://www.djangopackages.com/grids/g/model-translation/).

Comment 11

[Jannis Leidel [:jezdez]]

For tracking changes there are two options I know that have worked well in my past commercial projects, are well maintained and easy to use:

django-reversion http://django-reversion.readthedocs.org/
django-simple-history https://django-simple-history.readthedocs.org/

The first is well known and has a longer history than the latter. The first stores the historical data in a central Revision model, while the latter creates a in-memory "Historical*" model to store past data. Both work with South, the latter probably a bit better as it doesn't have to deal with past serialized data. If the source model changes (eg. a field is added) South will also pick up the changes to the in-memory model.

I would recommend the latter as I find storing serialized historical data in the database is a misuse of relational databases.

Comment 12

[Les Orchard [:lorchard]]

Not sure what the needinfo for me is asking... It sounds like it won't necessarily be me who builds this out. So, this will all be useful input to whomever does.

Comment 13

[Florian Scholz (Open Web Docs)]

> Yes, they all need to be localizable:
> * Browser name : Local marketing and usage can implied different names
> * Technology name : It depend on the technology but some of the new
> technologies introduce by HTML5 that are not technical acronym are used to
> be localized.
> * Feature name : Same as above
> * Specification title : Most likely localized
> 
> Jean-Yves, Florian, any extra though about this?
> 
> But remind that, in a contributor point of view, localization is optional
> and if it's not localized we will always fall back to the English version so
> it could be possible to ease the model by making the English mandatory and
> all the rest optional.

Sounds good to me so far. Some strings might be nice to have in Verbatim as they will likely be present in almost all compat tables (and should only be translated once and forever) and some strings (like notes, text) are very different and should be translated individually.

I agree with the English fall back rule.

Comment 14

[Andrew Overholt [:overholt]]

One thing I'd like to add into the data storage is availability in Web Workers.

Comment 15

[Luke Crouch [:groovecoder]]

FWIW, the first alpha site is up at http://doesitwork-dev.allizom.org/. It's not the final thing. :) :jwhitlock has filed many issues here and we'll continue to iterate on it.

Comment 16

[John Whitlock [:jwhitlock]]

The BrowserCompat project is canceled.  See https://github.com/mdn/browsercompat for current effort. Bulk status change includes the random word TEMPOTHRONE.