Closed Bug 869648 Opened 7 years ago Closed 6 years ago

[logparser] Support several destinations

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mcote, Unassigned)

References

Details

Attachments

(1 file)

The logparser will have to write to both the prod and dev ES instances.  It will also have to write to the metrics instance until we are ready to switch away from it.
Blocks: 848834
Should be applied on top of the patch in bug 869712.
Assignee: nobody → mcote
Status: NEW → ASSIGNED
Attachment #747475 - Flags: review?(jgriffin)
The shell scripts will also have to be updated; however, since we will be changing the servers at least a couple times in the near future, I think we should modify the generated script (logparser/bin/savelogs-gen.sh) by hand for now and check in an updated version when we finally stop using the metrics cluster.
Blocks: 870559
Comment on attachment 747475 [details] [diff] [review]
Support multiple ES servers

Review of attachment 747475 [details] [diff] [review]:
-----------------------------------------------------------------

::: logparser/logparser.py
@@ +49,5 @@
>      self.delete_file = delete_file
>      self.logger = logger
> +    if not es_servers:
> +      es_servers = DEFAULT_ES_SERVERS[:]
> +    self.es_servers = es_servers

or more compactly:

self.es_servers = es_servers or DEFAULT_ES_SERVERS[:]

@@ +145,4 @@
>  
>      if 'testsuites' in data:
>        for ts_index, ts in enumerate(data.get('testsuites', [])):
> +        for tg in testgroups:

It seems like we're repeating 'for tg in testgroups' unnecessarily.  Can't we just do:

testgroups = [...]
for tg in testgroups:
  # everything goes here
  # ...
  tg.submit()

@@ +166,4 @@
>  
>          for tp_index, tp in enumerate(ts.get('passes', [])):
> +          for tg in testgroups:
> +            tg.add_test_pass(doc_type='testpasses_' + testgroup.testsuites[-1].testsuite,

s/testgroup/tg
Attachment #747475 - Flags: review?(jgriffin) → review+
Fixed both.  Also added "[default: localhost:9200]" to the --es-server help.  We're repeating that default in a few places now, but meh for another day.

https://hg.mozilla.org/automation/logparser/rev/abada2fd0fab

We should deploy this as soon as possible to minimize the amount of data we need to copy over.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
So are we ready to deploy this and add an --es-server elasticsearch-zlb.webapp.scl3.mozilla.com:9200 argument to logparser's startup script?
I believe so.  The necessary flow was created in bug 869627.  We should probably wait on adding the dev es server until we figure out an expiration mechanism.

Just keep an eye on it after deploying--or you can wait until tomorrow morning when we're both around.
I'll do it tomorrow AM, just in case!
Deployed, and verified that it's now writing data to both the metrics and the IT prod clusters.
You need to log in before you can comment on or make changes to this bug.