1277933 - Reset ETL Processing

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Kyle Lahnakoski [:ekyle]]

The timezone change has caused record duplication in the ES cluster.  A reset should work, but we should verify the script will work

Comment 1

[Kyle Lahnakoski [:ekyle]]

What version (hash) is running in production now?

Comment 2

[Kyle Lahnakoski [:ekyle]]

Maybe it would be nice to remove the timezone corrections the ETL performs, since they are no longer corrections.

Comment 3

[Kendall Libby [:fubar] (he/him)]

(In reply to Kyle Lahnakoski [:ekyle] from comment #1)
> What version (hash) is running in production now?

looks like a945380a8906f3e83d2a9277e0311a1c2e0abc9b

Comment 4

[Kyle Lahnakoski [:ekyle]]

nope, that does not exist in my repo, was there a local change applied?

Comment 5

[Kyle Lahnakoski [:ekyle]]

ETL can be reset with the `--reset` flag, but looking through the code it seems the indexes must be deleted before a reset is done.

Comment 6

[Kyle Lahnakoski [:ekyle]]

OK, I reviewed the old master branch.  I am reasonably convinced the reset procedure will just plain work.  No need to update the code to the ?more recent? master on Github.

:fubar,  

Please delete the public_bugs and public_comments indexes and run the public ETL with the --reset flag.  Same goes for the private bugs.  I am happier if the cluster is completely wrong rather then this "sneaky" wrong it currently exhibits. 

If you want to do this while on a vidyo call, I am available whenever you are.

Comment 7

[Kendall Libby [:fubar] (he/him)]

(In reply to Kyle Lahnakoski [:ekyle] from comment #4)
> nope, that does not exist in my repo, was there a local change applied?

no local changes.

commit a945380a8906f3e83d2a9277e0311a1c2e0abc9b
Author: Kyle Lahnakoski <[e-mail]>
Date:   Sun Mar 16 23:59:05 2014 -0400

    Can successfully delete from ES: Tests added to confirm.
    https://bugzilla.mozilla.org/show_bug.cgi?id=983685

    Also, various accelerations, smaller batch sizes, and more logging.

    Squashed commit of the following:

    commit a4ae2f651b6325d80c0c7119a9dfb85c4df8c3f2
    commit 6e121196e6f8cb433b76d63465a8462f600c5ff2
    commit efe5bf8899427e7ab85c6adfd2a83a83264d5a04
    ...

looks like https://github.com/klahnakoski/Bugzilla-ETL/commit/c14aa450038701649db9924e233dd7e1368c4052



(In reply to Kyle Lahnakoski [:ekyle] from comment #6)
> Please delete the public_bugs and public_comments indexes and run the public
> ETL with the --reset flag.  Same goes for the private bugs.  I am happier if
> the cluster is completely wrong rather then this "sneaky" wrong it currently
> exhibits. 
> 
> If you want to do this while on a vidyo call, I am available whenever you
> are.

I vaguely remember that resets weren't difficult or problematic, other than how long they took. Will kick 'em off now.

Comment 8

[Kendall Libby [:fubar] (he/him)]

private_bugs was reset yesterday and :ekyle says it looks good, so kicking off public_bugs now.

Comment 9

[Kendall Libby [:fubar] (he/him)]

should be all set. please let me know if that's not the case.