805996 - Upgrade to 9.2 PostgreSQL for socorro-github jenkins

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Selena Deckelmann :selenamarie :selena]

We are ready to start running our jenkins environment for socorro-github against a 9.2 postgresql database. This may affect all the socorro-* instances if they are running on distinct systems. 

I'm unfamiliar with the jenkins environment, so help with getting this to the right people would be much appreciated.

Comment 1

[Shyam Mani [:fox2mike]]

Sure thing :) Webops owns jenkins now.

Comment 2

[Justin Dow [:jabba]]

http://jabba.pastebin.mozilla.org/1889312 here is a patch to the postgres2 module to support specific versions.

What it currently doesn't support is upgrading from one version to the next.. Anybody have any good ideas on that? Because of how the version number is in the data path, things could get tricky here.

Let me know if that patch is good to push. I think it should be a noop for existing installations.

Comment 3

[Ben Kero [:bkero]]

Looks good to me.  Do we have those other repos defined anywhere though?

Comment 4

[Justin Dow [:jabba]]

Yeah, I added them to the mrepo module and the yum_repos module today. They should mirror over night tonight, so I'll check in the morning that things are working properly there.

Comment 5

[Jason Thomas [:jason]]

r+ from me for patch

Comment 6

[Brandon Burton [:solarce]]

There is a Postgres instance running on the Jenkins box, :selena can you please confirm that's the your tests are using and that the data here is ephemeral?

:jabba, in theory I should be good to use the module and repo changes right?

Comment 7

[Justin Dow [:jabba]]

I'm still testing that the upgrade will work. I think I need to add some more logic to make it upgrade. I'm working on this right now.

Comment 8

[Justin Dow [:jabba]]

I've committed the patch. What this does is allow you to specify a specific version of postgres to manage. Currently though, because of the way the postgres RPMs are set up, taking a 9.0 host and specifying 9.2 will install postgres9.2 and initialize a new database in /var/lib/pgsql/9.2/ and won't actually migrate the data. I'm not sure the best way to automate moving the data to the new directory.. Any thoughts on that?

Comment 9

[Selena Deckelmann :selenamarie :selena]

Per IRC conversation: automating the data migration is likely out of scope. There are some changes to our particular schema that have to be run either after a migration, or as part of a new database load. And a different project, with a different schema, may not need the same migration code.

Comment 10

[Brandon Burton [:solarce]]

We're discussing requirements on IRC, 9.0 + 9.2, and confirming what the jenkins job use for credentials so we can plan out how to make these changes

Comment 11

[Brandon Burton [:solarce]]

The couple files we may need to tweak

https://github.com/mozilla/socorro/blob/master/scripts/build.sh
https://github.com/mozilla/socorro/blob/master/socorro/unittest/config/commonconfig.py.dist

I will review the puppet module and plan out things on Mon

Comment 12

[Selena Deckelmann :selenamarie :selena]

(In reply to Brandon Burton [:solarce] from comment #11)

> I will review the puppet module and plan out things on Mon

Awesome! Is there anything I can do to help? We have three small issues with tests remaining in our 9.2 compatibility updates, and should have those resolved very shortly.

Comment 13

[Selena Deckelmann :selenamarie :selena]

Just checking in. We've resolved all our issues with the tests, and would love to get this updated.

Comment 14

[Brandon Burton [:solarce]]

I should have the new 9.2 node done by the end of the day and will then work with you to get an account, load data, and run some tests.

Comment 15

[Brandon Burton [:solarce]]

Ran into what may be a bug in the postgres2 module, :atoll was kind enough to whip up a patch that should fix it, just need to get a r? from :jabba, which should happen tomorrow

I'll update the bug once I have things ready, sorry for the delay

Comment 16

[Brandon Burton [:solarce]]

I am working on what should fix this

Comment 17

[Matt Pressman [:mpressman]]

so it looks like an already existing PGDATA or -D datadir in /var/lib/pgsql/9.2/data already existed and wasn't empty prior to creating it with initdb. When initdb is run with an existing datadir it will not create/overwrite existing files and will fail:

initdb: directory "/var/lib/pgsql/9.2/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/pgsql/9.2/data" or run initdb
with an argument other than "/var/lib/pgsql/9.2/data".

Comment 18

[Matt Pressman [:mpressman]]

I've migrated the data from jenkins1.dmz.phx1 to the new host and the service is now running

Comment 19

[Matt Pressman [:mpressman]]

One more point, I needed to install the postgresql92-contrib rpm.

Comment 20

[Selena Deckelmann :selenamarie :selena]

Also, need to add the jenkins webhost to pg_hba.conf allows -- I think :solarce is taking care of this...

Comment 21

[Brandon Burton [:solarce]]

This is mostly working, we need to do a side-by-side on jenkins1. for libs to build psycopg2, will work on this next week

Comment 22

[Selena Deckelmann :selenamarie :selena]

:solarce -- I talked this over with :rhelmer, and we think that just upgrading to 9.2 libs is fine on Jenkins. 

The libs are designed to be backward compatible for libraries. So as long as the libs aren't being shared with a 9.0 instance of PostgreSQL, we'd be fine. I wasn't aware of how the systems were set up, so my suggestions about libraries were made out of an abundance of caution. 

If we're *only* supporting libraries on that system and not instances of the database, it's totally ok to only have the 9.2 libraries.

Comment 23

[Brandon Burton [:solarce]]

Ok, we should be good to do a rebuild of 29 for stage tomorrow

[[e-mail]1 run]# /usr/pgsql-9.2/bin/pg_config
BINDIR = /usr/pgsql-9.2/bin
DOCDIR = /usr/share/doc/pgsql
HTMLDIR = /usr/share/doc/pgsql
INCLUDEDIR = /usr/pgsql-9.2/include
PKGINCLUDEDIR = /usr/pgsql-9.2/include
INCLUDEDIR-SERVER = /usr/pgsql-9.2/include/server
LIBDIR = /usr/pgsql-9.2/lib
PKGLIBDIR = /usr/pgsql-9.2/lib
LOCALEDIR = /usr/pgsql-9.2/share/locale
MANDIR = /usr/pgsql-9.2/share/man
SHAREDIR = /usr/pgsql-9.2/share
SYSCONFDIR = /etc/sysconfig/pgsql
PGXS = /usr/pgsql-9.2/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--disable-rpath' '--prefix=/usr/pgsql-9.2' '--includedir=/usr/pgsql-9.2/include' '--mandir=/usr/pgsql-9.2/share/man' '--datadir=/usr/pgsql-9.2/share' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/usr/lib64' '--with-openssl' '--with-pam' '--with-krb5' '--with-gssapi' '--with-includes=/usr/include' '--with-libraries=/usr/lib64' '--enable-nls' '--with-ossp-uuid' '--with-libxml' '--with-libxslt' '--with-ldap' '--with-system-tzdata=/usr/share/zoneinfo' '--sysconfdir=/etc/sysconfig/pgsql' '--docdir=/usr/share/doc' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et' 'CPPFLAGS= -I/usr/include/et'
CC = gcc
CPPFLAGS = -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include
CFLAGS = -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv
CFLAGS_SL = -fpic
LDFLAGS = -L/usr/lib64 -Wl,--as-needed
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgport -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lcrypt -ldl -lm 
VERSION = PostgreSQL 9.2.1