[Compat Data] Add data to compat datastore

RESOLVED FIXED

Status

--
enhancement
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: jwhitlock, Unassigned)

Tracking

Details

(Whiteboard: [specification][type:feature])

(Reporter)

Description

4 years ago
What problems would this solve?
===============================
The Compat Data API is only useful if it has data.  Some of this needs to be expertly curated to ensure data quality (browsers, versions, specifications).  Some of the data should be lifted from existing MDN data (features, supports)

Who would use this?
===================
Initially, only MDN staff working directly on the project.  As the data is populated, it will expand to the wider staff and selected volunteers.  Finally, the data will be used by MDN readers, contributors, and other projects.

What would users see?
=====================
Browsers will have a standard name used across compat data, without near-name duplicates.  Version numbering will be consistent.  MDN pages will be validated against curated data before being converted to API-backed tables.

What would users do? What would happen as a result?
===================================================
Domain experts will ensure that new data matches the quality of existing data.  Users will be able to view data and contribute, without degrading data quality.  External developers will be able to build on top of the standardized data set, either through the API or directly with the data.

Is there anything else we should know?
======================================
This will be an iterative process.  Initial data will be stored as JSON in a github project, with contributions through pull requests.  Validation of MDN pages will require adjustments to the MDN pages themselves, to standardize browser names, formatting, etc.  When the contribution UI ships, the API can become the source of truth, with periodic releases of the JSON data.
(Reporter)

Updated

4 years ago
Blocks: 996570
Severity: normal → enhancement

Comment 1

4 years ago
Commits pushed to master at https://github.com/mozilla/web-platform-compat

https://github.com/mozilla/web-platform-compat/commit/200af9d520e8beadd2d00e5090555c747f6e09d6
bug 1079932 - Convert load_spec_data to sync

tools/load_spec_data.py now reads existing data, compares it to data
loaded from SpecName and Spec2, determines new, updated, deleted, and
unchanged data, confirms with the caller, and then executes the
operations.

Also:
- Adds tools/resources.py, for storing related API data
- Adds tools/tests/test_resources.py, for unit tests
- Adds subset of tools to automated tests and coverage
- Adds support to Client for deletes, updates, and URL generation

https://github.com/mozilla/web-platform-compat/commit/865c8dd032fff00937f2651782016953a6e40561
bug 1079932 - Handle null to TranslatedTextField

Handle passing JSON null as value of optional TranslatedTextField.
Previously, TranslatedTextField.from_native expected a string value.

https://github.com/mozilla/web-platform-compat/commit/9b209c1380e0cb3868211926b4073244b84ace8e
bug 1079932 - tools/load_webcompat_data.py to sync

Convert tools/load_webcompat_data.py to use sync semantics rather than
load to an empty API semantics.  Includes:
- Handle null IDs (for example, feature.parent for a root feature)
- Handle 'falsey' IDs (such as placeholder 0 for the chrome Browser)
- Switch Resource.to_json_api to default to including sorted fields
- Fix Support.alternate_name_mandatory to Support.alternate_mandatory

https://github.com/mozilla/web-platform-compat/commit/08f58e24a2c1597661b1567fabae80eb1d6cf7fc
bug 1079932 - Add tools/download_data.py

This tool is used to download the data from the API.

https://github.com/mozilla/web-platform-compat/commit/5e98504e0a440b821379006482cac15c19298b4a
bug 1079932 - Add tools/upload_data.py

This tool is used for uploading data from disk versions of JSON API.

https://github.com/mozilla/web-platform-compat/commit/334cd4d1ff58f541b1184e81389590e5554b8093
bug 1079932 - Refactor logging and API download

- Loggers are now names 'tools.(module name)', and command line tools
install a handler for the 'tools' logger.
- Client.get_resources now .get_resource_collection, drops logger_name
  parameter.
- Collection.load_all now uses Client.get_resource_collection, for
  better logging of massive resource chains.
- Collection.change_original_collection drops logger_name parameter

https://github.com/mozilla/web-platform-compat/commit/e0f2bd15e984a3a896df56a62dae7e00c5425e46
bug 1079932 - Add docs for data repository

Data is stored at https://github.com/jwhitlock/browsercompat-data.  Can
be moved to Mozilla-owned repo, or a new repo can be created using
tools/download_data.py.  Data and API code is at
http://browsercompat.herokuapp.com.

https://github.com/mozilla/web-platform-compat/commit/bbc22c580110098d81022d707e1268eb02310a6f
bug 1079932 - Improve browse app for Versions

- Display dates as 2014-12-05 w/o hour / minute / second
- Wrap release_notes_uri links in <a></a>

https://github.com/mozilla/web-platform-compat/commit/e9f0a970180bd1a54923f07e0125deaa8fb2f4c7
fix bug 1079932 - More ordering in JSON API

- Translated fields - 'en' first, then other languages in alpha order
- Use OrderedDict more to preserve order
- Use unicode more consistantly
- Use JSON serializer in diffs, to preserve field order
- When order doesn't matter, sort related IDs by ID in cache

https://github.com/mozilla/web-platform-compat/commit/3e401c4f0384dc98995d6969c94282c00a9ca82d
Merge pull request #17 from jwhitlock/1079932_add_data

Fix bug 1079932 - Add data

Updated

4 years ago
Status: UNCONFIRMED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED

Comment 2

3 years ago
Commit pushed to master at https://github.com/mdn/browsercompat

https://github.com/mdn/browsercompat/commit/5c6d31d2efe1d8eb89bea74c5d8faf6b01800fb4
bug 1079932 - Handle more whitespace in Spec2

This line in https://developer.mozilla.org/en-US/docs/Template:Spec2
was breaking tools/load_spec_data.py:

  'Unhandled Promise Rejection': 'ED',

Adjust the regex to handle the optional space in these lines.
You need to log in before you can comment on or make changes to this bug.