Closed Bug 1399639 Opened 8 years ago Closed 7 years ago

Upgrade Kuma dependencies, Q3 2017 - Q2 2018

Categories

(developer.mozilla.org Graveyard :: Code Cleanup, enhancement)

Product:
developer.mozilla.org Graveyard
Component:
Code Cleanup
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: jwhitlock, Assigned: jwhitlock)

References

Details

Upgrade Kuma's third party libraries to the recent versions, as noted by requires.io: https://requires.io/github/mozilla/kuma/requirements/?branch=master The previous quarter's efforts were tracked in bug 1352492. With Django 1.11 released in April 2017, we need to be more aggressive with updates. We also need to occasionally update requirements for security patches.
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/d0315d83e2ad17782253e7287bcaf157b83add87 bug 1399639: Upgrade urllib3 * urllib3 1.14 → 1.22: Checks certificates for SSL connections. Many bug fixes and features. https://github.com/mozilla/kuma/commit/a38b9b82a5889f613108928be54e62450630d1ea Merge pull request #4425 from jwhitlock/upgrade-urllib3-1399639 bug 1399639: Upgrade urllib3
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/6aa1fe3ca00ef9cf57975c88a22f074ddd013182 bug 1399639: Upgrade newrelic to 2.94.0.79 * newrelic 2.60.0.46 → 2.94.0.79: datastore support (elasticsearch, redis, mysql), tornado and other engine support, bugfixes
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/51a9b4ae66f74273c114a8931df6a49bde06d834 bug 1399639: Upgrade raven (for sentry) to 6.2.1 * raven 5.10.2 → 6.2.1: Breadcrumb support, update Django support, many bug fixes and small features.
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/0f23ea3cab611bfafe8b436bc4c11c433caf16ce bug 1399639: Upgrade django-allauth, dependencies * django-allauth 0.24.1 → 0.33.0: AJAX account management views, easier OAuth2 callback_url customization, Django compatibility, GitHub email query, HMAC email confirmations, better Jinja2 support, easier extensions of authentication backend * oauthlib 1.0.3 → 2.0.4: Sanitation / encoding fixes, OpenID support * requests-oauthlib 0.6.1 → 0.8.0: Improve automatic token refresh Kuma requires some update as well: * The email scope is now included by the default GitHubProvider * The confirmation email uses HMAC instead of storing a record in the database, changing the URL pattern and the tests.
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/6b4c88951e6bd0fc9f7e212a3ba827503ab9a482 bug 1399639: Upgrade flake8 and dependencies Update to current versions as installed in TravisCI: * flake8 3.1.1 → 3.5.0: Suppot current pyflakes, pycodestyle, faster performance, Python 3.6 compat * mccabe 0.5.2 → 0.6.1: Report column number, fixes * pycodestyle 2.1.0 → 2.3.1: add E722 for bare exceptions, other fixes * pyflakes 1.3.0 → 1.6.0: Find files without .py extension, other fixes
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/5cb0a1a00d84dce6d0d01a2fd508c4ebbc282e4d bug 1399639: Update to bleach 2.1.1, deps * bleach 2.0.0 → 2.1.1: Convert control characters to entities, restrict clean and linkify to unicode or utf8-encoded strings. * six 1.10.0 → 1.11.0: with_metaclass fixes, more moves https://github.com/mozilla/kuma/commit/457b87fc433bce4620a43cbef961274b1d65a02b Merge pull request #4489 from jwhitlock/update-bleach-1399639 bug 1399639: Update to bleach 2.1.1, deps
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/530ac68639dcfcd1b025d867c4473d230d80c7a8 Merge pull request #4542 from mozilla/dp_update_newrelic Bug 1399639: update to latest newrelic dep
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/0efa2f0ad6da5d89af207bf492ed7f3b65a5c7f8 bug 1399639: Configure python for node-gyp node-gyp is the compiler for binary node.js packages, needed to compile the optional New Relic native-metrics package. node-gyp requires Python, and looks for it at /usr/bin/python by default. This location doesn't exists, because Ubuntu is transitioning to Python 3 and wants users to be explicit about Python 2 versus Python 3. This PR sets the Python path to the installed Python 2.7 binary. https://github.com/mozilla/kuma/commit/0d7ff43c58fc1a45be8cb2fa63cd0cf023b79c5c Merge pull request #4557 from jwhitlock/node-gyp-1399639 bug 1399639: Configure python for node-gyp
Commits pushed to master at https://github.com/mdn/kumascript https://github.com/mdn/kumascript/commit/a2bc0192e23ec0f98e97520dcd439e4321720572 bug 1399639: Update newrelic to 2.4.0 * newrelic 1.34.0 → 2.4.0: Node 8 support, third-party instrumentation support, native metrics, redacted messagees in High Security Mode. Requires PR mozilla/kuma#4557 to install native metrics. https://github.com/mdn/kumascript/commit/3480c00ceabc089843289daf5c495298be03fb28 Merge pull request #416 from jwhitlock/update-newrelic-1399639 bug 1399639: Update newrelic to 2.4.0
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/3bce301bccd6031182ddb3de4b21ddf7c276f695 bug 1399639: Upgrade django-ratelimit to 1.1.0 * django-ratelimit 0.6 → 1.1.0: Allow requests when cache backend is unavailable, Django 1.9 - 2.0 support
Rather than open a new bug, I'm continuing on this one for Q1 2018. We should be able to catch up with dependencies as part of the Django upgrade.
Assignee: nobody → jwhitlock
Blocks: 1401246
Status: NEW → ASSIGNED
Summary: Upgrade Kuma dependencies, Q3/Q4 2017 → Upgrade Kuma dependencies, Q3 2017 - Q1 2018
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/ed04550f54ebf84cf482fa4ef1c46a53e678d548 bug 1399639: Update requirements docs for links Add a section explaining why our requirements files are so wordy, and make them wordier, to include links to code, documentation, and changes for our third-party libraries. https://github.com/mozilla/kuma/commit/3ad33b61ed3a35baf47eda8698ff19441947a76c bug 1399639: Update django-tidings * django-tidings 1.1 → 2.0: Support Django 1.9 to 2.0, Celery 4.x, non-pickle serializer. https://github.com/mozilla/kuma/commit/fc6b3f50c45a1defe432671058a97b1904d47460 Merge pull request #4660 from jwhitlock/upgrade-django-tidings-1399639 bug 1399639: Make requirements more complex, upgrade django-tidings
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/f1ea75d72c8eec264d067cd8517a3980c297e145 bug 1399639: Update django-tidings 2.0.1 * django-tidings 2.0 → 2.0.1: Bug fix for exception when running asynchronously with Celery.
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/ac480e288999a8f61c084d254105c35529b98d6d bug 1399639: Add shared deps for runtime and tests Add requirements/default_and_test.txt, to collect requirements used in the production, deployment, and test environments. This upgrades production: * requests 2.9.1 → 2.11.1: Reject headers with leading whitespace, proxy support https://github.com/mozilla/kuma/commit/760ea3ca5b4657e382c0a1fdbe916502c8b24370 bug 1399639: Update requests * requests 2.11.1 → 2.18.4: IDNA2008 support, deprecate requests.packages, Response as context manager. Added new constraints: * certifi 2018.1.18 * chardet 3.0.4 * idna 2.6 https://github.com/mozilla/kuma/commit/1f88849acbb840d34e30a143c9f2435bdc4a2b63 Merge pull request #4672 from jwhitlock/update-requests-1399639 bug 1399639: Update requests
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/37ec321ad175189fe2feffb7517b6440f089012c bug 1399639: Update to pyquery 1.2.15, constraints pyquery 1.4 is the current release, but tests (and maybe code) starts breaking at 1.2.16. This commit is limited to updates with no code changes. * pyquery 1.2.11 → 1.2.15: Support :has, fix .val() for textarea and select * lxml 3.5.0 → 4.1.1: Manual charset detection for HTML5, bug fixes * cssselect 0.9.1 → 1.0.3: Fix :nth-*(an+b) pseudo selectors https://github.com/mozilla/kuma/commit/777b53ab7d1d57b424fae6fcd89defc8d120adf5 bug 1399639: Update form testing for user profile In a browser, a selected checkbox submits the value "on", and is omitted from the submitted form when off. pyquery 1.2.15 changes to match jQuery and browser behaviour for .val(). Instead, use .checked to determine if the checkbox is selected. Note: Django accepts falsey values "False", "0", etc. as well, to signify that a checkbox is unchecked. Email is not an editable text field on the profile edit page. The user picks their email or adds a new unverified email on a different page. The profile edit form ignores the submitted email, so it is removed from tests. https://github.com/mozilla/kuma/commit/c0caca407915c49233b80cf6d4dab5fe2cb432ee bug 1399639: Update to pyquery 1.2.17 * pyquery 1.2.15 → 1.2.17: Update .val() handling to match PyQuery https://github.com/mozilla/kuma/commit/20c13663ead86531d0cf27598c490f4af52d9519 bug 1399639: Refactor ExtractCodeSampleTests Convert ExtractCodeSampleTests to pytest standards, include with other Extractor.code_sample tests. https://github.com/mozilla/kuma/commit/676d08e0e45f44a7aa63e5eb64e6feb037b4b196 bug 1399639: Handle empty content in Extractor Test Extractor methods with empty content for document.rendered_html. This can either be None (content is unrendered) or an empty string (latest revision is empty). Only Extractor.css_classnames needs modification to handle this case. https://github.com/mozilla/kuma/commit/7e77863fef1da7c34cac19ea42e0c01e4d8816f0 bug 1399639: Update to pyquery 1.3.0 * pyquery 1.2.17 → 1.3.0: Code cleanup, dropped Python 2.6 and 3.2. Parsing None or an empty string now raises an exception in some circumstances. https://github.com/mozilla/kuma/commit/4133bcc4a7da12fc21d1e253b98cb46e9013bba8 bug 1399639: Refactor AttachmentTemplatesTests Refactor to pytest style, including moving some pytest fixtures into the global conftest.py. https://github.com/mozilla/kuma/commit/74a8b288e37e6a0ee56e8271655fba570bc1f92a bug 1399639: Refactor mock_requests to fixture https://github.com/mozilla/kuma/commit/44944c3f2be440f1792b15430f2aea05780d87ca bug 1399639: Refactor AkismetFormTests Refactor to pytest style, and avoid using Exception.message directly, which was deprecated in Python 2.6. https://github.com/mozilla/kuma/commit/6caa23ab5fcda65522f3daf412fcb833a9d74c66 bug 1399639: Update to pyquery 1.4.0 * pyquery 1.3.0 → 1.4.0: Match Firefox behaviour for text(), which means that whitespace in HTML is replaced with a newline. There is a new parameter 'squash_space' that can be set to False to preserve whitespace. https://github.com/mozilla/kuma/commit/ba6abe21a70d542b344c0271987a8d952d747615 Merge pull request #4678 from jwhitlock/update-pyquery-1399639 bug 1399639: Update pyquery and constraints
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/64e39075dfd8535a5ecc5b6421317b5043f47d81 bug 1399639: Update django-constance to 2.1.0 * django-constance 1.1.2 → 2.1.0: Add compatibility with Django 1.11, custom fields, signal for updates. * django-picklefield 0.3.2 → 1.0.0: Drop old Django support These updates don't appear to require any code changes, but improve compatibility with Django 1.11 and open some possible features in the future. https://github.com/mozilla/kuma/commit/03f125f1c33b8cf30bf5dc6047a515167e3e6250 Merge pull request #4691 from jwhitlock/update-constance-1399639 bug 1399639: Update django-constance to 2.1.0
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/384c4796bc63f8186bf241d8104cb21f0a7203ea bug 1399639: Upgrade to Django 1.8.19 * Django 1.8.18 → 1.8.19: Security release, mitigate denial-of-service attacks against catastrophic backtracking vulnerability in regular expressions. One of the affected functions (truncatechars) is used in the Django admin. https://github.com/mozilla/kuma/commit/32855fc4d16319e10bbbe96928ddc41a9bf54e44 Merge pull request #4700 from jwhitlock/update-django-1.8.19-1399639 bug 1399639: Upgrade to Django 1.8.19
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/3cf7d82d86bef330d6d3971791628b313de8338e bug 1399639: Update to django-statici18n 1.7.0 * django-statici18n 1.1.5 → 1.7.0: Django 1.10 and later support, update Python support, support JSON output format. * django-appconf 1.0.1 → 1.0.2: Use django.utils.six, setup fixes https://github.com/mozilla/kuma/commit/a1e4593046a1b22d0084703626fb3aaaf6a561c6 bug 1399639: Update to django-extensions 2.0.6 * django-extensions 1.6.1→2.0.6: Drop support for Django 1.7 and earlier, add 1.11 and 2.0 support, many fixes and improvements. * typing 3.6.4: New requirement of django-extensions, backport of Python 3.5's type hinting library.
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/7a255d10090b3c302dcdd8578e80bc3497e8c187 bug 1399639: Update to hashin 0.13.0 * hashin 0.11.2 → 0.13.0: Sorted hashes, faster New requirements for this version: * packaging 17.1: Core utilities for Python packages * pip-api 0.0.1: Provides pip API, rather than using private API * pyparsing 2.2.0: Create and execute simple grammars
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/ff71078d61f1ac813e1aa07bc017d283b8a8c0a5 bug 1399639: Use locale name for statici18n path django-statici18n is now using the locale name, such as en_US, rather than the standardized language code, like en-US. Replace that dash with an underscore, and update some utility functions to be clear about locale names versus language codes (which may be important as we retire the custom locale middleware). https://github.com/mozilla/kuma/commit/38d68f6ecaa307fd8be9c5af4e6a8d231cdb17fc Merge pull request #4730 from jwhitlock/statici18n-1399639 bug 1399639: Use locale name for statici18n path
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/f13638a05fc76f4c247be87f15f0446a4e34ce3f bug 1399639: Update to celery 3.1.25, deps Update celery and dependencies to prepare (again) for Celery 4.x update. * celery 3.1.20 → 3.1.25: Support Task protocol 2 from 4.x * billiard 3.3.0.22 → 3.3.0.23: Fix traceback wrapper * kombu 3.0.33 → 3.0.37: Serialization, Redis fixes * django-celery 3.1.17 → 3.2.2: Celery 4 compatibility https://github.com/mozilla/kuma/commit/de1664fafbfbefd6067d950a20799fa0137f154b Merge pull request #4743 from jwhitlock/update-celery-3.1-1399639 bug 1399639: Update to celery 3.1.25, deps
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/b32d87f795b518d079e12378fd7f8e1ed7b3ae42 bug 1399639: Update to django-jinja 2.4.1, Jinja2 * django-jinja 2.2.1 → 2.4.1: Django 1.11 compatibility * Jinja2 2.8 → 2.10: Default policy of "rel=nopener", trimmed trans * MarkupSafe 0.23 → 1.0: Drop Python 2.6 support Update configuration, and tests for rel=nopener.
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/f192e2bab0435fa0442f39e4aa08bdb3477fd08e bug 1399639: Update to django-debreach 1.4.2 * django-debreach 1.4.0 → 1.4.2: Add support for Django 2.0
Commits pushed to master at https://github.com/mdn/kumascript https://github.com/mdn/kumascript/commit/d47912bf19258f37582449383dcbc264cbf6d8c0 bug 1399639: Update to newrelic 4.1.0 * newrelic 2.4.0→4.1.0: Drop support for node v0.10, v0.12, fix security issues, SSL required, bug fixes and library updates. https://github.com/mdn/kumascript/commit/8099821aee08ae0e96f907156d6f0ef9e49329aa Merge pull request #696 from jwhitlock/update-newrelic-1399639 bug 1399639: Update to newrelic 4.1.0
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/53974c0a09aa53e967493c88c8871bc9df93a935 bug 1399639: Update to newrelic 3.2.0.91 * newrelic 2.96.0.80→3.2.0.91: Manadatory SSL, security issues with SQL analysis and tracing APIs, fix memory leaks. https://github.com/mozilla/kuma/commit/df55301e85679e540b01fa9398e62b8bc8d4fc9b Merge pull request #4758 from jwhitlock/newrelic-1399639 bug 1399639: Update to newrelic 3.2.0.91
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/a76f9f659c112fdec28f6783ba1505ef5c42e102 bug 1399639: Update to django-allauth 0.34.0 * django-allauth 0.33.0 → 0.34.0: Security updates for password change workflow.
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/66c5e2a7f1d446c9dcbb9e0a8aa50fba43712259 bug 1399639: Update to dennis 0.9 * dennis 0.7 → 0.9: Drop Python 2.6, remove Django shims, unpin click, other bug fixes. * click 6.6 → 6.7: bug fixes https://github.com/mozilla/kuma/commit/e531cbe97a1f5ef689536dcc7a3f97d2bec9337e Merge pull request #4775 from jwhitlock/update-dennis-1399639 bug 1399639: Update to dennis 0.9
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/4acc052bacf473f5aef9dce8a7a3e257130c9b2f bug 1399639: Upgrade to django-extensions 2.0.7 * django-extensions 2.0.6 → 2.0.7: Fix ./manage.py pipchecker for pip 10
This ended up covering the whole 12 months of updating to Django 1.11. We didn't catch up, about half of the requirements are still out of date.
Summary: Upgrade Kuma dependencies, Q3 2017 - Q1 2018 → Upgrade Kuma dependencies, Q3 2017 - Q2 2018
See Also: → 1467532
Tracking the effort for the next two quarters in bug 1467532, with a focus on Python 3 compatibility.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
See Also: → 1352492
Commit pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/359f6b266674d3f9ff29eaaa3895730684023899 bug 1399639: Upgrade newrelic to 3.2.1.93 * newrelic 3.2.0.91 → 3.2.1.93: Bug fixes for other environments
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.