1623183 - Postgres Connection termination kills services

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The queue pod keeps dying with

Error: Connection terminated unexpectedly                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    at Connection.<anonymous> (/app/node_modules/pg/lib/client.js:252:9)
    at Object.onceWrapper (events.js:417:28)
    at Connection.emit (events.js:311:20)
    at Connection.EventEmitter.emit (domain.js:482:12)
    at TLSSocket.<anonymous> (/app/node_modules/pg/lib/connection.js:133:10)
    at TLSSocket.emit (events.js:323:22)
    at TLSSocket.EventEmitter.emit (domain.js:482:12)
    at endReadableNT (_stream_readable.js:1204:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

a) it shouldn't die and
b) why are we getting this

unfortunately, that's all it logs before it dies (well, it's a monitor.error report).

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I can repro this locally by running a simple script that just holds a client open, and killing the pg server.

events.js:288
      throw er; // Unhandled 'error' event
      ^

Error: Connection terminated unexpectedly
    at Connection.<anonymous> (/home/dustin/p/taskcluster/node_modules/pg/lib/client.js:252:9)
    at Object.onceWrapper (events.js:417:28)
    at Connection.emit (events.js:311:20)
    at Socket.<anonymous> (/home/dustin/p/taskcluster/node_modules/pg/lib/connection.js:133:10)
    at Socket.emit (events.js:323:22)
    at endReadableNT (_stream_readable.js:1204:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
Emitted 'error' event on Client instance at:
    at connectedErrorHandler (/home/dustin/p/taskcluster/node_modules/pg/lib/client.js:199:10)
    at Connection.<anonymous> (/home/dustin/p/taskcluster/node_modules/pg/lib/client.js:268:9)
    at Object.onceWrapper (events.js:417:28)
    [... lines matching original stack trace ...]
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

shell returned 1

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

The script got a little more complicated! The error occurs when the connection dies while the client is checked out of the pool, so I added a sleep:

const tcdb = require('taskcluster-db');                                                                                                                                                                                                                                                                                       

(async () => {
  const db = await tcdb.setup({
    readDbUrl: 'postgresql://postgres@localhost/postgres',
    writeDbUrl: 'postgresql://postgres@localhost/postgres',
    serviceName: 'foo',
  });

  while (1) {
    try {
      await db._withClient('read', async client => {
        console.log(new Date());
        try {
          const res = await client.query('select 1 as x');
          console.log(res.rows[0].x);
        } catch (err) {
          console.log(`err ${err}`);
        }
        await new Promise(r => setTimeout(r, 1000));
        try {
          const res = await client.query('select 2 as x');
          console.log(res.rows[0].x);
        } catch (err) {
          console.log(`err 2 ${err}`);
        }
      });
    } catch (err) {
      console.log(`err outer ${err}`);
    }
  }
})().then(console.log, console.log);

With https://github.com/taskcluster/taskcluster/pull/2531 in place, this script succeeds (well, it gets the expected errors when the server is killed, but it handles them without exiting).