1402938 - Port Heka plugins to the new logging 2.0 Hindsight CEP

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task, P1)

Description

[Mike Trinkala [:trink]]

Current Heka instance: https://heka-fxa-us-west-2.prod.mozaws.net/

The discussion with jrgm yielded the following decisions:
1) Analysis plugins that are commented out below are not processing any data and will not be ported.
2) The ROC/MWW alerts were too noisy and are currently being ignored; they will not be ported.  Only the data and graph output will remain for direct inspection or roll-ups on other dashboards (via HTTPS).
2a) The URLs of the graphs/data will change (new host and some plugin renaming)
3) All FXA outputs on this instance were for archival purposes. The logging 2.0 pipeline now handles archiving so these will not be ported:

Analysis:
FxaAccountCreationAbuse 	
FxaAccountLogin 	
FxaActiveDailyUsers 	
# FxaAuthBouncerMySQL 	
# FxaAuthDbMemory 	
FxaAuthDbMySQL 	
FxaAuthHTTPStatus 	
FxaAuthMemory 	
FxaAuthRateLimit 	
FxaAuthRateLimit60Days 	
FxaAuthRequestTime 	
FxaAuthResend 	
FxaBasicMetrics 	
FxaBrowserIdHTTPStatus 	
FxaBrowserIdRelyingParties 	
FxaContentHTTPStatus 	
FxaContentOauthErrors 	
FxaContentSignupMetrics 	
FxaContentSignupMetrics-fmd 	
FxaContentSignupMetrics-marketplace 	
FxaContentSignupMetrics-pocket_mobile 	
FxaContentSignupMetrics-pocket_web 	
# FxaContentSignupMetrics-setup 	
FxaContentSignupMetrics-sync 	
FxaCreates1Day 	
FxaCreates30Day 	
FxaDailyActiveUsers 	
FxaMonthlyActiveUsers 	
FxaOauthHTTPStatus 	
FxaProfileHTTPStatus 	
# FxaSandbox 	
# FxaSandbox-BadMetricsRequest 	
# FxaSandbox-FxaContentSignupMetricsCategorized 	
FxaSync11Migrations 

Output:
FxaAuthBouncerFileOutput 	
FxaAuthDbFileOutput 	
FxaAuthFileOutput 	
FxaAuthNginxErrorFileOutput 	
FxaAuthNginxFileOutput 	
FxaAuthProfileMsgQFileOutput 	
FxaBasicMetricsFileOutput 	
FxaBrowserIdFileOutput 	
FxaBrowserIdNginxErrorFileOutput 	
FxaBrowserIdNginxFileOutput 	
FxaContentFileOutput 	
FxaContentNginxErrorFileOutput 	
FxaContentNginxFileOutput 	
FxaCustomsFileOutput 	
FxaEmail 	
FxaFlow 	
FxaLastActiveTimestamp 	
FxaNetworkDeviceUsage 	
FxaOauthFileOutput 	
FxaOauthNginxErrorFileOutput 	
FxaOauthNginxFileOutput 	
FxaPiiScrubberOutput 	
FxaProfileFileOutput 	
FxaProfileNginxErrorFileOutput 	
FxaProfileNginxFileOutput 	
FxaRetention

Comment 1

[Mike Trinkala [:trink]]

# Blocked (:jrgm is this data going to be made available?)
- FxaBrowserIdHTTPStatus - no Kinesis stream
- FxaBrowserIdRelyingParties - no Kinesis stream
- FxaContentOauthErrors - no data

# Incompatible
- fxa_basic_metrics - this would have to be redesigned as it consumes data from other analysis plugins (this type of looping is not allowed in Hindsight). To perform that type of aggregation it would have to become an output plugin or we would have to change the topology to chain together multiple Hindsights.  I recommend turning it into an output plugin.

# Potential Future Improvements
- fxa_abuse - use the expiring cuckoo filter to track more IP's; switch to the streaming_algorithms running stats (or better yet p2 and use the median instead of the mean)
- fxa_account - should add a preservation_version cfg variable
- fxa_auth_memory - mww alert was still enabled in the old version but was not ported;; should add a preservation_version cfg variable
- fxa_auth_mysql - should add a preservation_version cfg variable
- fxa_auth_rate_limit - mww alert was still enabled in the old version but was not ported; should add a preservation_version cfg variable
- fxa_auth_request_time - roc alert was still enabled in the old version but was not ported; should add a preservation_version cfg variable
- fxa_auth_resend - roc alert was still enabled in the old version but was not ported; should add a preservation_version cfg variable
- moz_logging_http_status - roc alert was still enabled in the old version but was not ported

# Added
FxaAuthDbMemory - Inactive in Heka (incorrect message matcher), enabled in Hindsight

Comment 3

[Mike Trinkala [:trink]]

Final summary (this bug will be close as fixed if no feedback is given (new specific bugs can be opened if necessary)).  Please comment if you aren't satisfied with the following resolutions:

* FxaAccountCreationAbuse has a title output only, no data (Heka and Hindsight).  It will be removed. 

* FxaBasicMetrics - In Heka 6 of the 7 outputs have had no new data since Sep 10, 2016.  ADU is the only active JSON output, it will not be ported.

* FxaBrowserID* plugins have no Kinesis data source.  They will be removed.

* FxaContentSignupMetrics* has no Fields[events] array in the new data source.  Since the schema has changed these plugins have no output. They will be removed.

* FxaContentOauthErrors has no Heka output since Aug 6 2017 (no current Hindsight output either).  It will be removed.


The following analysis plugins have been ported and are producing equivalent output:
FxaAccountLogin
FxaDailyActiveUsers
FxaAuthDbMemory
FxaAuthDbMySQL
FxaAuthHttpStatus
FxaAuthMemory
FxaAuthRateLimit
FxaAuthRateLimit60Days
FxaAuthRequestTime
FxaAuthResend
FxaContentHttpStatus
FxaCreates1Day
FxaCreates30Day
FxaDailyActiveUsers
FxaMonthlyActiveUsers
FxaOauthHTTPStatus
FxaProfileHTTPStatu
FxaSync11Migrations

Comment 4

[Mike Trinkala [:trink]]

This has been deployed for several months now, closing.
https://logging-cep.prod.mozaws.net/