935065 - Losing connections to Vertica

Status: RESOLVED WORKSFORME

(Data & BI Services Team :: DB: MySQL, task)

Description

[Annie Elliott]

[08:54am] aelliott: Database error 0x80004005: [Vertica][VerticaDSII] (20) An error occurred during query execution: server closed the connection unexpectedly
[08:54am] aelliott: 	This probably means the server terminated abnormally
[08:54am] aelliott: 	before or while processing the request.
[08:54am] aelliott: Unable to create extract
[08:55am] sheeri: how long did the query run for?
[08:55am] sheeri: can you try running it again?
[08:55am] aelliott: I wager about 20-30 min
[08:55am] aelliott: sure thing
[08:55am] sheeri: stage or production?
[08:55am] aelliott: prod
[08:55am] aelliott: vertica.metrics.scl3.mozilla.com
[08:55am] sheeri: it could be that there's a time limit on queries maybe?
[08:55am] aelliott: noooooooo
[08:55am] aelliott: we run gargantuan pulls
[08:56am] aelliott: they've run over night before just fine
[08:56am] aelliott: this is really odd behaviour
[08:56am] aelliott: just 40M rows
[08:57am] aelliott: starting
[08:57am] aelliott: ugh
[08:57am] aelliott: Database error 0x80004005: [Vertica][VerticaDSII] (10) An error occurred during query preparation: no connection to the server
[08:57am] aelliott: Unable to create extract
[08:57am] aelliott: Database error 0x80004005: [Vertica][VerticaDSII] (10) An error occurred during query preparation: no connection to the server
[08:57am] aelliott: vpn connection is fine
[09:13am] sheeri: OK, let me check the dbs
[09:13am] aelliott: ok
[09:14am] sheeri: hrm, all the prod servers I can connect to
[09:14am] sheeri: so they're all on
[09:15am] sheeri: what's the query?
[09:15am] sheeri: also, can you make a bug?
[09:15am] aelliott: ok
[09:15am] aelliott: what kind of bug?
[09:16am] aelliott: (it was just the canned tableau query with a where clause added)
[09:17am] • sheeri checks the load balancer
[09:17am] sheeri: server operations:database
[09:17am] sheeri: in the mozilla.org product

Here is the canned tableau query, to which I had added the where clause. It had pulled back in excess of 12M rows last time I peeked before it crapped out.

SELECT "adi_dimensional_by_date"."_year_quarter" AS "_year_quarter",
  "adi_dimensional_by_date"."bl_date" AS "bl_date",
  "adi_dimensional_by_date"."product" AS "product",
  "adi_dimensional_by_date"."v_prod_major" AS "v_prod_major",
  "adi_dimensional_by_date"."prod_os" AS "prod_os",
  "adi_dimensional_by_date"."v_prod_os" AS "v_prod_os",
  "adi_dimensional_by_date"."channel" AS "channel",
  "adi_dimensional_by_date"."locale" AS "locale",
  "adi_dimensional_by_date"."continent_code" AS "continent_code",
  "adi_dimensional_by_date"."cntry_code" AS "cntry_code",
  "adi_dimensional_by_date"."tot_requests_on_date" AS "tot_requests_on_date",
  "adi_dimensional_by_date"."distro_name" AS "distro_name",
  "adi_dimensional_by_date"."distro_version" AS "distro_version"
FROM "public"."adi_dimensional_by_date" "adi_dimensional_by_date"
WHERE lower(product) = 'thunderbird' AND
bl_date >= '2012-01-01'

Comment 1

[Annie Elliott]

aelliott: i restarted vpn and everything
[09:37am] aelliott: I am going to try again
[09:38am] aelliott: 1M
[09:41am] aelliott: 4M
[09:44am] aelliott: 8M
[09:50am] aelliott: 16M
[09:58am] aelliott: 26M
[10:08am] aelliott: 38M
[10:10am] aelliott: 40M - getting close to the end here
[10:14am] sheeri: yay
[10:14am] • sheeri crosses fingers
[10:15am] aelliott: 46M
[10:17am] aelliott: 46.333M D-O-N-E
[10:17am] aelliott: I shall close the ticket
[10:17am] sheeri: YAY!