[Compat Data] Add data to compat datastore



4 years ago
3 years ago


(Reporter: jwhitlock, Unassigned)



(Whiteboard: [specification][type:feature])



4 years ago
What problems would this solve?
The Compat Data API is only useful if it has data.  Some of this needs to be expertly curated to ensure data quality (browsers, versions, specifications).  Some of the data should be lifted from existing MDN data (features, supports)

Who would use this?
Initially, only MDN staff working directly on the project.  As the data is populated, it will expand to the wider staff and selected volunteers.  Finally, the data will be used by MDN readers, contributors, and other projects.

What would users see?
Browsers will have a standard name used across compat data, without near-name duplicates.  Version numbering will be consistent.  MDN pages will be validated against curated data before being converted to API-backed tables.

What would users do? What would happen as a result?
Domain experts will ensure that new data matches the quality of existing data.  Users will be able to view data and contribute, without degrading data quality.  External developers will be able to build on top of the standardized data set, either through the API or directly with the data.

Is there anything else we should know?
This will be an iterative process.  Initial data will be stored as JSON in a github project, with contributions through pull requests.  Validation of MDN pages will require adjustments to the MDN pages themselves, to standardize browser names, formatting, etc.  When the contribution UI ships, the API can become the source of truth, with periodic releases of the JSON data.


4 years ago
Blocks: 996570
Severity: normal → enhancement

Comment 1

4 years ago
Commits pushed to master at https://github.com/mozilla/web-platform-compat

bug 1079932 - Convert load_spec_data to sync

tools/load_spec_data.py now reads existing data, compares it to data
loaded from SpecName and Spec2, determines new, updated, deleted, and
unchanged data, confirms with the caller, and then executes the

- Adds tools/resources.py, for storing related API data
- Adds tools/tests/test_resources.py, for unit tests
- Adds subset of tools to automated tests and coverage
- Adds support to Client for deletes, updates, and URL generation

bug 1079932 - Handle null to TranslatedTextField

Handle passing JSON null as value of optional TranslatedTextField.
Previously, TranslatedTextField.from_native expected a string value.

bug 1079932 - tools/load_webcompat_data.py to sync

Convert tools/load_webcompat_data.py to use sync semantics rather than
load to an empty API semantics.  Includes:
- Handle null IDs (for example, feature.parent for a root feature)
- Handle 'falsey' IDs (such as placeholder 0 for the chrome Browser)
- Switch Resource.to_json_api to default to including sorted fields
- Fix Support.alternate_name_mandatory to Support.alternate_mandatory

bug 1079932 - Add tools/download_data.py

This tool is used to download the data from the API.

bug 1079932 - Add tools/upload_data.py

This tool is used for uploading data from disk versions of JSON API.

bug 1079932 - Refactor logging and API download

- Loggers are now names 'tools.(module name)', and command line tools
install a handler for the 'tools' logger.
- Client.get_resources now .get_resource_collection, drops logger_name
- Collection.load_all now uses Client.get_resource_collection, for
  better logging of massive resource chains.
- Collection.change_original_collection drops logger_name parameter

bug 1079932 - Add docs for data repository

Data is stored at https://github.com/jwhitlock/browsercompat-data.  Can
be moved to Mozilla-owned repo, or a new repo can be created using
tools/download_data.py.  Data and API code is at

bug 1079932 - Improve browse app for Versions

- Display dates as 2014-12-05 w/o hour / minute / second
- Wrap release_notes_uri links in <a></a>

fix bug 1079932 - More ordering in JSON API

- Translated fields - 'en' first, then other languages in alpha order
- Use OrderedDict more to preserve order
- Use unicode more consistantly
- Use JSON serializer in diffs, to preserve field order
- When order doesn't matter, sort related IDs by ID in cache

Merge pull request #17 from jwhitlock/1079932_add_data

Fix bug 1079932 - Add data


4 years ago
Last Resolved: 4 years ago
Resolution: --- → FIXED

Comment 2

3 years ago
Commit pushed to master at https://github.com/mdn/browsercompat

bug 1079932 - Handle more whitespace in Spec2

This line in https://developer.mozilla.org/en-US/docs/Template:Spec2
was breaking tools/load_spec_data.py:

  'Unhandled Promise Rejection': 'ED',

Adjust the regex to handle the optional space in these lines.
You need to log in before you can comment on or make changes to this bug.