1186341 - add a azure-entities.Entity object function that does a no-change upgrade

Status: RESOLVED WONTFIX

(Taskcluster :: Services, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

We have a great migration support.  We have an upgrade function which runs each time an entity is loaded until changes are saved back to the entity.  I understand that we don't want a load() method to mutate the object permanently, but it would be really nice if we had a function that I could put in a loop that says "Load this entity, run the migration on it and save the new form" without having to modify the object.

The main reason I want this is because it's a real pain to see debugging output from migrations happen so frequently until the object is modified.  I'd like to have debugging output in the migration functions, but having that output every time an entity is loaded is not ideal.  It also means that if the migration function changes over time that you might get different results saved back to azure depending on when the object was first mutated after the schema changes.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

@jhford,
So basically, if we have:
  var myEntity = base.Entity.configure({
    version: 1,
    properties: /* version 1 definition */,
  }).configure({
    version: 2,
    properties: /* version 2 definition */,
    migrate: ...
  });

We should be able to make a stand alone binary in bin/ called bin/migrate.js:
  // load config
  var cfg = base.config(...);

  myEntity.migrateTable();

Then we can run bin/migrate.js as a one-off dyno and delete migration code and version 1 definition
afterwards leaving us with:
  var myEntity = base.Entity.configure({
    version: 2,
    properties: /* version 2 definition */,
  });


Is that what you're thinking?

----
I agree that would be nice, and probably fairly easy to do.
Then we can remove legacy code, I like that...

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #1)
> @jhford,
> So basically, if we have:
>   var myEntity = base.Entity.configure({
>     version: 1,
>     properties: /* version 1 definition */,
>   }).configure({
>     version: 2,
>     properties: /* version 2 definition */,
>     migrate: ...
>   });
> 
> We should be able to make a stand alone binary in bin/ called bin/migrate.js:
>   // load config
>   var cfg = base.config(...);
> 
>   myEntity.migrateTable();
> 
> Then we can run bin/migrate.js as a one-off dyno and delete migration code
> and version 1 definition
> afterwards leaving us with:
>   var myEntity = base.Entity.configure({
>     version: 2,
>     properties: /* version 2 definition */,
>   });
> 
> 
> Is that what you're thinking?

Yep.  The value in not-migrating entities feels very low to me as long as we can do them in a background job.  Once we've had a couple of schema changes (say 3-4), the Entity definitions are going to get really messy if we can't do a full table migration.

> ----
> I agree that would be nice, and probably fairly easy to do.
> Then we can remove legacy code, I like that...

Yay!

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

This is now in the azure-entities library.  The changes would be to add a `migrateTable` method to the Entity class which

 - scans all entitites with version less than the latest version
 - upgrades them to the latest version

You can find examples of scanning in https://github.com/taskcluster/taskcluster-index/blob/7aca33aa31c3237fd850ba7c73ed823e8f6c13f0/src/data.js#L175-L200

The azure-entities code is pretty sophisticated and uses some trickery with JavaScript "subclassing", so please have a look and ask any specific questions that might come up here in the bug.

Comment 4

[Divya Baid]

Can I work on this issue ?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Yes you may!

Comment 6

[Divya Baid]

I am not able to upgrade the entity. Can you please guide me ?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I can't, without any additional information :)

Comment 8

[Kriti Singh]

Hi @dustin. Since I have been going through the code of azure-entities and azure-blob-storage in parallel for the inMemory issue, can I take up this issue in case no one is on it? Thank you.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Divya, were you able to make any progress, or provide more information on what the problem is?  If you're no longer working on this, perhaps it's OK for Kriti to take it?

Comment 10

[Divya Baid]

Yes, She may take up the issue.

Comment 11

[Kriti Singh]

@dustin so I am on this. For running tests on azure-entities, I need to use the inmemory version, right? Although(maybe I am doing it wrong), when I pass the account, tablename and credentials in the options of Entity.setup of src/entity.js file, I get the error as 'Provide accountname, accountkey, sas, etc.) and thus the tests don't run due to insufficient credentials. So how do I run the tests? Please guide me. Thank you!

Comment 12

[Kriti Singh]

@dustin so I am on this. For running tests on azure-entities, I need to use the inmemory version, right? Although(maybe I am doing it wrong), when I pass the account, tablename and credentials in the options of Entity.setup of src/entity.js file, I get the error as 'Provide accountname, accountkey, sas, etc.' and thus the tests don't run due to insufficient credentials. So how do I run the tests? Please guide me. Thank you!

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

For testing azure-entities, you do need credentials.  Other projects that *use* azure-entities can use the inMemory version.

We can get you credentials in two ways: you can sign up for a "free tier" in Azure, and set up a storage account.  Or, we could send you a "jungle" account credential for the Taskcluster.  For that, we will need your GPG key; the easiest way to set one of those up is with a keybase.io account.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

We'll be migrating away from azure-entities, so there's no sense in adding features.