1694270 - Please update Phabricator to 2021 Week 10

Status: RESOLVED FIXED

(Cloud Services :: Operations: Phabricator, task)

Description

[David Lawrence [:dkl]]

Please update Phabricator to 2021 Week 9

https://secure.phabricator.com/w/changelog/2021.08/
https://secure.phabricator.com/w/changelog/2021.09/
https://secure.phabricator.com/w/changelog/2021.10/

{
  "moz_phabricator_commit": "641fea1d5c4e489c4899b99de9634cd57f824a2c",
  "moz_phabricator_version": "641fea1d5c4e489c4899b99de9634cd57f824a2c",
  "moz_phabricator_source": "https://github.com/mozilla-services/phabricator-extensions",
  "build": "https://circleci.com/gh/mozilla-services/phabricator-extensions/409"
}

Database migrations required: Yes
20210215.changeset.01.phid.sql 9,580 ms
20210215.changeset.02.phid-populate.php 153,880 ms See below.
20210216.index.01.version.sql 8,252 ms
20210216.index.02.epoch.sql 8,085 ms

Documentation regeneration: No

Deploy to phabricator-dev first for testing.

Upgrading / Compatibility

NOTE: This release includes a potentially slow migration, 20210215.changeset.02.phid-populate.php.

This migration adds PHIDs to "DifferentialChangeset" objects to support the new differential.changeset.search API method. A rough estimate of the cost of this migration is 5 minutes per million changesets. You can figure out how many changesets you have with:

mysql> SELECT MAX(id) FROM phabricator_differential.differential_changeset;

For example: if you have 12,000,000 changesets, this migration may take about 60 minutes to apply. The migration can be deferred or run online, but doing so is complex; contact support if you administrate a large install and want more guidance.

Comment 1

[Bob Micheletto [:bobm]]

In dev:

MySQL [(none)]> SELECT MAX(id) FROM phabricator_differential.differential_changeset;
+---------+
| MAX(id) |
+---------+
|   11459 |
+---------+

Comment 2

[Bob Micheletto [:bobm]]

Dev database schema change took ~8 minutes.

Comment 3

[David Lawrence [:dkl]]

Phab QA has been done on dev and has passed. We can move to stage today.

Here are the results from running the SQL on production:

MySQL [(none)]> SELECT MAX(id) FROM phabricator_differential.differential_changeset;
+---------+
| MAX(id) |
+---------+
| 3643470 |
+---------+

According to upstream formula it should take around 15-16 mins to migrate. Your result was ~8 mins for only 11k rows. So according to your results it should take almost 2 days :( So I am not sure what the final result will be. Of course phab-dev is lesser powered than production.

Thoughts? Should we do this for a tree closure window to be sure? I would need to see when the next one is coming up.

Comment 4

[Bob Micheletto [:bobm]]

Stage DB migration, and deployment complete.

Stage does not have nearly the number of changesets as dev or production.

MySQL [(none)]> SELECT MAX(id) FROM phabricator_differential.differential_changeset;
+---------+
| MAX(id) |
+---------+
|     161 |
+---------+

Migration took 8 seconds:

Applying schema adjustments...
Done.
Completed applying all schema adjustments.
 ANALYZE  Analyzing tables...
Done.
 ANALYZED  Analyzed 540 table(s).

Comment 5

[David Lawrence [:dkl]]

Update comment 0 with new JSON data for 2021 Week 9 upstream code. Please update phab-dev with the newer image which contains all fixes from week 8 and 9.

Comment 6

[Bob Micheletto [:bobm]]

Tested the migration against a copy of the production database. It took ~45 minutes to complete. That's much better than our 48 hours worst case, but still not great. Ideally, we would find a way to do these live. Tree-closing window, here we come.

Comment 7

[David Lawrence [:dkl]]

I have updated the info in comment 0 to add a newer version to the next deployment. Please update phabricator-dev with the newer image so QA can be done. Thanks.

Comment 8

[David Lawrence [:dkl]]

QA has passed on phab-dev. We are ready to proceed to phab-stage.

Comment 9

[David Lawrence [:dkl]]

Stage is good. Ready for Saturday prod deployment.

Comment 10

[Bob Micheletto [:bobm]]

# docker exec -it mozphab /bin/sh
~ $ cd phabricator
~/phabricator $ bin/storage status | grep "Not Applied"
phabricator:20210215.changeset.01.phid.sql                default mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306 Not Applied -                sql  /app/phabricator/resources/sql/autopatches/20210215.changeset.01.phid.sql
phabricator:20210215.changeset.02.phid-populate.php       default mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306 Not Applied -                php  /app/phabricator/resources/sql/autopatches/20210215.changeset.02.phid-populate.php
phabricator:20210216.index.01.version.sql                 default mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306 Not Applied -                sql  /app/phabricator/resources/sql/autopatches/20210216.index.01.version.sql
phabricator:20210216.index.02.epoch.sql                   default mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306 Not Applied -                sql  /app/phabricator/resources/sql/autopatches/20210216.index.02.epoch.sql
~/phabricator $ bin/storage upgrade
Before running storage upgrades, you should take down the Phabricator web
interface and stop any running Phabricator daemons (you can disable this
warning with --force).

    Are you ready to continue? [y/N] y

Applying patch "phabricator:20210215.changeset.01.phid.sql" to host "mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306"...
Applying patch "phabricator:20210215.changeset.02.phid-populate.php" to host "mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306"...
Applying patch "phabricator:20210216.index.01.version.sql" to host "mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306"...
Applying patch "phabricator:20210216.index.02.epoch.sql" to host "mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306"...
Storage is up to date. Use "storage status" for details.
Synchronizing static tables...
Verifying database schemata on "mozphab-phabhost-rds-devsvcprod-201709240232.c8twujv3cfsf.us-west-2.rds.amazonaws.com:3306"...


Database                 Table                    Name          Issues
phabricator_differential differential_changeset   key_phid      Missing Key
phabricator_harbormaster harbormaster_buildtarget key_started   Missing Key
phabricator_harbormaster harbormaster_buildtarget key_completed Missing Key
phabricator_harbormaster harbormaster_buildtarget key_created   Missing Key

Found 4 adjustment(s) to apply, detailed above.

You can review adjustments in more detail from the web interface, in Config > Database Status. To better understand the adjustment workflow, see "Managing Storage Adjustments" in the documentation.

MySQL needs to copy table data to make some adjustments, so these migrations may take some time.


    Apply these schema adjustments? [y/N] y

Applying schema adjustments...
Done.
Completed applying all schema adjustments.
 ANALYZE  Analyzing tables...
Done.
 ANALYZED  Analyzed 540 table(s).

~/phabricator $ [root@ip-1/data/mozphab/scripts/phdctl.sh statusb /bin/sh
 NO DAEMONS  There are no running daemon processes.
# /data/mozphab/scripts/phdctl.sh start
Freeing active task leases...
Freed 0 task leases.
Launching daemons:
(Logs will appear in "/app/logs/daemons.log".)

    (Pool: 1) PhabricatorRepositoryPullLocalDaemon
    (Pool: 1) PhabricatorTriggerDaemon
    (Pool: 1) PhabricatorFactDaemon
    (Pool: 16) PhabricatorTaskmasterDaemon

Done.
# /data/mozphab/scripts/phdctl.sh status
PID Command
156 php ./phd-daemon

Comment 11

[David Lawrence [:dkl]]

Sanity checks done on prod and all looks good. Closing. Thanks!