982857 - Public BZ-ETL acting odd

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Kyle Lahnakoski [:ekyle]]

The public ETL is not deleting private bugs.  The code responsible for deleting does not appear to be run.

1) Please update the public settings.json so that elasticsearch log has a different "type" (aka name) than the private ETL.  Right now they are both set to "bz_etl", so their logs are overlapping.  Here is what my staging server looks like. 

> {
>   "log_type": "elasticsearch",
>   "host": "http://klahnakoski-es.corp.tor1.mozilla.com",
>   "index": "debug",
>   "type": "public_etl"
> }

2) We should see the public logs emitting "get all private bug ids" (or "problem getting private bugs" if it fails).  We should also see the occasional "Ensure the following private bugs are deleted".  Neither of these are showing.  This could happen if the public settings.json file has "allow_private_bugs" set to true.  - But that makes no sense.   Please verify the public ETL script is running the latest code.

Thanks

Comment 1

[Kendall Libby [:fubar] (he/him)]

Fixed logging; private is now logging as private_etl, and public is public_etl. Should be in ES as of the 10:10 EST run.


public definitely does NOT have allow_private_bugs enabled:
etl1.bugs.scl3# grep private public_settings.json 
        "allow_private_bugs":false


Version of running code is:
commit bc4c6117836ebc914daf7bbc294e5639102bfd68
Merge: 93feb05 5a2727b
Author: Kyle Lahnakoski <[e-mail]>
Date:   Wed Mar 12 08:30:27 2014 -0400

Also...
2014-03-13 13:30:01.687389 - Timer start: get all private bug ids
2014-03-13 13:30:03.331550 - Timer end  : get all private bug ids (took 1.55 sec
onds)
2014-03-13 13:30:05.380884 - Ensure the following private bugs are deleted:
        [979724]
2014-03-13 13:30:05.450101 - Ensure the following private bugs are deleted:
        [982562, 982590]

Although I'm seeing those logs repeated (i.e. same bug numbers), so it looks like it's finding them, but isn't removing them.

Comment 2

[Kyle Lahnakoski [:ekyle]]

Thanks Kendall.  I can now see the same in the structured logs.  I have been able to confirm the problem on my dev machine.