1363579 - Migrate bmoweb container to centos 7

Status: RESOLVED INVALID

(MozReview Graveyard :: Testing / Development Environment, enhancement)

Description

[Gregory Szorc [:gps]]

Currently, the bmoweb container is running Ubuntu 14.04. The reason we didn't use CentOS 6 several years ago was it was old and required extra work to get all the Docker bits working. This is before a lot of the Docker complexity in v-c-t came into existence and was a time when the Docker integration in v-c-t was simple and turnkey.

According to dylan, we should be able to move the BMO Docker container to CentOS 7. https://moz-devservices-bmocartons.s3.amazonaws.com/bmo_centos7/vendor.tar.gz is the URL for cartons.

Comment 1

[Dylan Hardison [:dylan] (he/him)]

This might serve as a sort of example. https://github.com/mozilla-bteam/docker-bmo-base/tree/centos7
I've built a version of bmo-ci based on it and that passed almost all the tests.

Comment 2

[Gregory Szorc [:gps]]

Attachment: docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

For hysterical raisins, the bmoweb Docker image was previously
Ubuntu 14.04. The BMO team has to maintain a one-off carton for
Ubuntu 14.04 and this is a waste of their time. So this commit
converts the Docker images to CentOS 7.

While invasive, the port is pretty straightforward.

Swapping Ubuntu for CentOS required changing a bunch of packages
around.

CentOS 7 doesn't have a decent Python 3.5+ package built-in, so we
configure the IUS repo and install Python 3.6 from it. And, there
is no built-in python-mysql-connector package for Python 3.5. So
we install pip and install a different MySQL client from there.
This required modifying some Python in entrypoint.py.

The httpd configuration has been simplified by consolidating the
prefork config into the BMO config file.

The path to socket files is different in CentOS, so we update the
MySQL socket path throughout.

Since we invalidate the Docker image, we remove code in
entrypoint.py to `git checkout` some files. We haven't patched these
files in quite a long time since upstream did equivalent work.

Review commit: https://reviewboard.mozilla.org/r/137796/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137796/

Comment 3

[Gregory Szorc [:gps]]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137796/diff/1-2/

Comment 4

[Dylan Hardison [:dylan] (he/him)]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

https://reviewboard.mozilla.org/r/137796/#review141078

I think you can get rid of some of the potentially-fragile config updating code by using the scripts dkl wrote a few quarters ago.

Also seeing "require all granted" made me realize our httpd config files are not compatible with newer apache. I filed this as bug 1363736.

::: testing/docker/builder-bmoweb/Dockerfile:137
(Diff revision 2)
>  
>  ENTRYPOINT ["/bmoweb_entrypoint.py"]
>  CMD ["/usr/bin/supervisord"]
>  
>  # Temporarily pin commit until bug 1311165 is resolved.
>  ENV BMO_COMMIT 2f310fba9800f9d106ad8090cbbfb628c53599a1

Since bug 1312735 landed, you don't need to do this
if you set BUGZILLA_UNSAFE_AUTH_DELEGATION.

::: testing/docker/builder-bmoweb/bugzilla.conf:20
(Diff revision 2)
>    ServerName localhost
>    DocumentRoot "${BUGZILLA_HOME}/bugzilla"
>    <Directory "${BUGZILLA_HOME}/bugzilla">
>      DirectoryIndex index.cgi
>      Options Indexes FollowSymLinks ExecCGI
>      AllowOverride All

Just a note, this line is not needed. .htaccess files are loaded at startup time and ignored subsequently.

::: testing/docker/builder-bmoweb/checksetup_answers.txt:10
(Diff revision 2)
>   $answer{'db_check'}  = 1;
>   $answer{'db_mysql_ssl_ca_file'}     = '';
>   $answer{'db_mysql_ssl_ca_path'}     = '';
>   $answer{'db_mysql_ssl_client_cert'} = '';
>   $answer{'db_mysql_ssl_client_key'}  = '';
>   $answer{'create_htaccess'} = '';

I'm not sure why this is '', but setting it to '1' will more accurately reflect production.

::: testing/docker/builder-bmoweb/entrypoint.py:148
(Diff revision 2)
>  
>  if reset_database and not fresh_database:
>      print(subprocess.check_output(mysql_args, input=b'DROP DATABASE bugs;'))
>      fresh_database = True
>  
>  # Workaround for bug 1152616.

You can remove this comment, I WONTFIX'd that bug.

::: testing/docker/builder-bmoweb/entrypoint.py:160
(Diff revision 2)
>  
>      # We may not always need -I. When introduced, upstream assumed the modules
>      # were already in the default path.
>      args = [
>          'perl',
>          '-I', j(b, 'lib'),

the -I line is not needed, and is in fact useless: bugzilla doesn't store its own modules under lib/ like a sane application. Instead, it is a hold-over for the pre-local-libification... the short of it is that scripts/generate_bmo_data.pl handles its include path.

::: testing/docker/builder-bmoweb/entrypoint.py:162
(Diff revision 2)
>      # were already in the default path.
>      args = [
>          'perl',
>          '-I', j(b, 'lib'),
>          j(b, 'scripts', 'generate_bmo_data.pl'),
>          '[e-mail]',

replace this with admin_email

::: testing/docker/builder-bmoweb/entrypoint.py:171
(Diff revision 2)
>  
>      with open(j(h, 'checksetup.done'), 'a'):
>          pass
>  
>  # The base URL is dynamic at container start time. Since we don't always run
>  # checksetup.pl (because it adds unacceptable container start overhead), we

Is this still true?

time perl checksetup.pl --no-templates --no-permissions &>/dev/null
perl checksetup.pl --no-templates --no-permissions &> /dev/null  1.30s user 0.18s system 29% cpu 5.016 total

::: testing/docker/builder-bmoweb/entrypoint.py:174
(Diff revision 2)
>  with open(j(b, 'data', 'params'), 'w') as fh:
>      for line in params_lines:
>          if "'urlbase' =>" in line:
>              fh.write("           'urlbase' => '" + bmo_url + "',\n")
>          elif "'mail_delivery_method' =>" in line:
>              fh.write("           'mail_delivery_method' => 'Test',\n")
>          elif "'auth_delegation' =>" in line:
>              fh.write("           'auth_delegation' => 1,\n")
>          else:
>              fh.write(line)
>  

I suggest replacing this code:

```python
def set_param(key, value):
    cc([j(b, "scripts/update_params.pl"), key, value], cwd = b)

set_param('urlbase', bmo_url)
set_param('mail_delivery_method', 'Test')
set_param('auth_delegation', '1')
```

::: testing/docker/builder-bmoweb/entrypoint.py:187
(Diff revision 2)
>  with open(j(b, 'localconfig'), 'w') as fh:
>      def write_variable(k, v):
>          fh.write("$%s = '%s';\n" % (k, v))
>  
>      for line in localconfig_lines:
>          if line.startswith('$db_user'):
>              write_variable('db_user', db_user)
>          elif line.startswith('$db_pass'):
>              write_variable('db_pass', db_pass)
>          elif line.startswith('$db_name'):
>              write_variable('db_name', db_name)
>          # The default memory limit is not sufficient to run the BMO
>          # configuration. Bump it up.
>          elif line.startswith('$apache_size_limit'):
>              fh.write('$apache_size_limit = 700_000;\n')
>          else:
>              fh.write(line)

Similarly, you can replace this code with something like:

```python
def set_localconfig(key, value):
    cc([j(b, "scripts/update_localconfig.pl"), key, value], cwd = b)

set_localconfig('db_user', db_user)
set_localconfig('db_pass', db_pass)
set_localconfig('db_name', db_name)
```

You can ignore the apache size limit, we set a reasonable value now (and we ignore values that are too small.

::: testing/docker/builder-bmoweb/run-apache.sh:5
(Diff revision 2)
>  #!/bin/bash
>  
>  set -e
>  
> -. /etc/apache2/envvars
> +LANG=C

If bmoweb needs to use auth delegation, setting BUGZILLA_UNSAFE_AUTH_DELEGATION=1 will allow http:// callback URIs.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

https://reviewboard.mozilla.org/r/137796/#review141078

Thank you for finding everything wrong with the existing bmoweb container!

As much as I'd like to fix things, I think migrating to CentOS 7 and fixing the configuration are unrelated and should be done separately. Maybe in the same bug. But at least as seaparate commits. I don't want to bite off too much and risk breaking things.

Comment 6

[:glob ✱]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

https://reviewboard.mozilla.org/r/137796/#review154288

the bmoweb image currently isn't being built:
bmoweb> Checking for LWP::Protocol::https (6.07)      found v6.06
bmoweb> *** Installation aborted. Read the messages above. ***

this is likely an issue with the vendor carton, but it blocks my ability to verify these changes work.
(time passes) yup.

dylan - https://moz-devservices-bmocartons.s3.amazonaws.com/bmo_centos7/vendor.tar.gz contains LWP::Protocol::https 6.06 instead of 6.07

::: testing/docker/builder-bmoweb/Dockerfile:137
(Diff revision 2)
>  
>  ENTRYPOINT ["/bmoweb_entrypoint.py"]
>  CMD ["/usr/bin/supervisord"]
>  
>  # Temporarily pin commit until bug 1311165 is resolved.
>  ENV BMO_COMMIT 2f310fba9800f9d106ad8090cbbfb628c53599a1

i agree with dylan about this issue - this should be fixed now to ensure we're running the revision the carton packages were built against.

Comment 7

[Dylan Hardison [:dylan] (he/him)]

Sigh, one day I won't have to maintain dependencies for 5 different flavors... 
I've kicked off a rebuild of bmo_centos7, it should be built shortly.

Comment 8

[Dylan Hardison [:dylan] (he/him)]

https://github.com/mozilla-bteam/carton-bundles/compare/7f5003c10913...aa9957cb4bce

the bmo_centos7 tarball should be working again. I'll make sure to do checking against this variation moving forward.

Comment 9

[Dylan Hardison [:dylan] (he/him)]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

https://reviewboard.mozilla.org/r/137796/#review154288

That is fixed now, btw.

Comment 10

[:glob ✱]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

https://reviewboard.mozilla.org/r/137796/#review154288

thanks dylan; checksetup is no longer failing.  (it's failing after that in vct code, working on that now).

Comment 11

[:glob ✱]

Comment on attachment 8866178 [details]
docker: convert bmoweb Docker image to CentOS 7 (bug 1363579);

https://reviewboard.mozilla.org/r/137796/#review154952

/var/lib/bugzilla/bugzilla/scripts/issue-api-key.pl is failing: Can't locate Bugzilla.pm in @INC (@INC contains: . lib local/lib/perl5 /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5)

it looks like either the cwd needs to be set to /var/lib/bugzilla/bugzilla (which doesn't appear to be easy with the docker api), or PERL5LIB needs to set:
PERL5LIB=/var/lib/bugzilla/bugzilla:/var/lib/bugzilla/bugzilla/local/lib/perl5

Comment 12

[BMO Automation]

MozReview is now obsolete. Please use Phabricator instead. Closing this bug.