1216786 - Split sections into sections and references

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: BrowserCompat, defect)

Description

[John Whitlock [:jwhitlock]]

What did you do?
================
1. Attempted to populate fresh database using tools/upload_data.py

What happened?
==============
There are two section records that refer to the ES1 spec, both with an empty path, but with different notes.  The importer detects a collision, because it uses a combination of specification slug and English path to identify a record.

What should have happened?
==========================
The upload should succeed

Is there anything else we should know?
======================================
The 'number' attribute should also be dropped, since this doesn't appear in the current MDN data. When the API becomes the source of compatibility data, the data structure can be reconsidered.

Comment 1

[John Whitlock [:jwhitlock]]

A better option is adding a resource "Reference", to map the relation of a Feature to a Section:

* A Section has one Specification, has an optional name, path, and number, and the pair (Specification, path) is unique.  Sections are ordered with respect to their Specification
* A Reference has one Section and one Feature, has an optional note, and the pair (Section, Feature) is unique.  References are ordered with respect to their Feature, so that Specifications can be ordered in the display. (re: bug 1183599 - Ordering of Specifications)

As Unicode art (best viewed in Bugzilla's fixed-width presentation):

         Old Resource Structure                         New Resource Structure
======================================       ========================================  

                  Parent                                       Parent
                    ⬆                                            ⬆ 
    Section  ⇇   Feature   ⇉   Support          Reference  ⇇   Feature   ⇉   Support
      ⬇             ⇊            ⬇                ⬇             ⇊            ⬇
Specification    Children      Version           Section      Children       Version
      ⬇                          ⬇                ⬇                          ⬇
  Maturity                     Browser        Specification                  Browser
                                                   ⬇
                                                Maturity

Comment 2

[John Whitlock [:jwhitlock]]

Starting on the solution in comment #1. Because there are database additions and deletions, this will involve at least three PRs and three production pushes. After each push, the migrations will be applied, and the production code has to run both with and without the migration applied.

The first PR add to the database, and includes:
* Include a schema migration to add the new Reference tables.

Once it goes to production, the migration will be applied, and an unused reference table will be available.

The second PR migrates the data and updates the code, and will:
* Add the Reference model
* Add a 'references' resource in the API
* Remove the 'note' attribute from the 'sections' resource
* Remove the 'features' relationship from the 'sections' resource
* Remove the ordered 'sections' relationship from the 'features' resource
* Add a 'references' relationship to 'sections' resources
* Add an ordered 'references' relationship to 'features' resources
* Add 'references' handlers to the instances cache
* Update the tools to expect 'references' resources (download_data.py, upload_data.py)
* Update the documentation fixtures for 'references' resources
* Update and augment the integration tests for 'references' resources
* Add References to the documentation
* Update the importer to populate references
* Add a data migration to populate references from sections

Once it goes to production, the data migration will be run, and the API will start serving the new references resource. The altered data will be downloaded from the API and committed to https://github.com/mdn/browsercompat-data.

The third PR deletes unused data, and will:
* Remove the now unused sortedm2m library
* Include a schema migration to drop the section/features many-to-many table, and drop the section.note column

Once it goes to production, the schema migration will be run, dropping the unused data.

All of the code will be written and ready to go in a feature branch, but will be split into PRs for the purposes of keeping production error free:

https://github.com/mdn/browsercompat/tree/resources_1216786

Comment 3

[John Whitlock [:jwhitlock]]

Updated the title to reflect the actual work that will be done. Also, code changes (PR 2) will include:

* Add 'references' model and relations to data browser
* Add unit tests for new functionality

I'm sure I'm missing steps, those will show up in the code as I find them.

Comment 4

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mdn/browsercompat

https://github.com/mdn/browsercompat/commit/7c3319f0feed4cad1da7a418cd4b3f806b1cdb45
bug 1216786 - No prompts in integration tests

In integration tests, ./manage.py migrate will attempt to detect and
delete stale content types, which breaks automated tests, unless
--no-input is passed.

https://github.com/mdn/browsercompat/commit/0f8440b1db864f69b15edc18935d9347a7d73462
bug 1216786 - Prepare database for references

Add the references table and related resources, but without the related
Django model changes. Also add API permissions for the new references
instances.

This migration needs to be applied in production before merging code
that uses the new table.

https://github.com/mdn/browsercompat/commit/f7132a36d85c825910545dc4dc358a751575cbf3
Merge pull request #111 from mdn/references_step1_1216786

bug 1216786 - Prepare database for references

Comment 5

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mdn/browsercompat

https://github.com/mdn/browsercompat/commit/2dcca5a84d821ee1c0fc333677fa5c6a9c52ca09
bug 1216786 - Add references model

The reference model will eventually replace the M2M relation between
features and sections.

https://github.com/mdn/browsercompat/commit/451f4f2cff4d4c6dcaf9ab9c640f3b51b5d9a89e
bug 1216786 - Add cache functions for References

Add cache loader, serializer, invalidators for References, as well as
caches for the reverse relationships on Features and Sections. Also add
the signal that kicks off cache invalidation on Reference changes.

For this round, do not remove data from the cache.

https://github.com/mdn/browsercompat/commit/0c11bd3a3936f8741723da3ccfb954164230e363
bug 1216786 - Add API endpoints for resources

This code is mostly declarative, which means very few tests change.

https://github.com/mdn/browsercompat/commit/c328a70724665adf49c621f04a4d29c52c701587
bug 1216786 - Update tools for references

Configure the command-line tools library to be able to read and write
reference resources.

https://github.com/mdn/browsercompat/commit/4fd5d5839d318df0d193303274dabc44e14bd746
bug 1216786 - Update other API endpoints for refs

Update feature and section resources to use references instead M2M
relationship. Add historical_resources to the changeset links.  These
API changes require adjusting the documentation test cases and results.

https://github.com/mdn/browsercompat/commit/694334da5be3a739943a01a4baf2be9477b906c9
bug 1216786 - Set references order on Feature

https://github.com/mdn/browsercompat/commit/ef0395fa70ea07fac1a7118809d8a7f4a8535685
bug 1216786 - Add additional references tests

https://github.com/mdn/browsercompat/commit/9836212cddc6e3c93fc1e8696acaf99d5fd4fc05
bug 1216786 - Make cascade deletion tests generic

Move identical cascade deletion test code to a generic TestCase, and run
API-specific tests for each namespace.

https://github.com/mdn/browsercompat/commit/a581d4d4561a8a90911785b11053daf03cbfa6a0
bug 1216786 - Use references in deletion tests

https://github.com/mdn/browsercompat/commit/b8c50336c875b92f78ad994993e80847ff67030c
bug 1216786 - Update docs for Resources

Also, v2/view_features supports a PATCH interface, not a PUT interface.

https://github.com/mdn/browsercompat/commit/f5a9171f54861ba4ddcdc31dc6fe700ecb26b356
bug 1216786 - Discard ending slash in list request

For the browse app, use /api/v1/<resource> instead of
/api/v1/<resource>/, which returns a 410 Gone.

https://github.com/mdn/browsercompat/commit/ee32976e77d7e7115c070afc676c8b8d548ee050
bug 1216786 - Add references to browse app

https://github.com/mdn/browsercompat/commit/fd9e88cdbe0d8f35fc3c1bcc421e79a4e4d569eb
bug 1216786 - Update sample views for resources

https://github.com/mdn/browsercompat/commit/0c31fa48f9569aceb97f5740f58e34e231fe2de2
bug 1216786 - Handle optional names

Instead of empty, unclickable <a> elements, return <em>none</em>.

https://github.com/mdn/browsercompat/commit/3c06922244afdbc166ffc41313674a2d087e3e4a
bug 1216786 - Use references in MDN importer

https://github.com/mdn/browsercompat/commit/f282111995e42a88708552be4a8fc63d1bf7fc91
bug 1216786 - Import empty notes as omitted

Instead of importing empty notes as the English string "", import them
as no note, to reduce translation burden.

https://github.com/mdn/browsercompat/commit/c9397dfce35769b4940211a2a13197240411346f
bug 1216786 - Drop Feature/Section M2M signal

https://github.com/mdn/browsercompat/commit/e8ff90a2d9e73b00c2a0c8189340af543818d1bd
bug 1216786 - Drop instance cache relations, props

* Drop sections from feature instance cache
* Drop features, note from sections instance cache

https://github.com/mdn/browsercompat/commit/985d7516a46c6cf9ec756ebd7d05e3641744a2e1
bug 1216786 - Remove feature/section M2M model

Remove references to the feature-to-section relationship, without
dropping it from the database.

https://github.com/mdn/browsercompat/commit/943874a59a0170a7751adde406003c6238abddcb
bug 1216786 - Migrate data to references

https://github.com/mdn/browsercompat/commit/b70f9036683927ddfce1316fd3274ef9ca19acc7
Merge pull request #112 from mdn/references_step2_1216786

bug 1216786 - Add References for Feature-Section relationships

Comment 6

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mdn/browsercompat

https://github.com/mdn/browsercompat/commit/68c5974cc019b7d37b11575e256fd7814809b441
bug 1216786 - Cleanup transition to references

* Drop M2M relation from features to sections
* Drop note from Section
* Disallow null references list in HistoricalFeatures

django-sortedm2m needs to be retained until migrations are squashed.

https://github.com/mdn/browsercompat/commit/4a0a1eaf113beaa062e42c95f4eb7f5bf39ffe78
Merge pull request #113 from mdn/references_step3_1216786

fix bug 1216786 - Cleanup transition to references