1188008 - Need test instance to for feedback/testing of upcoming merge of upstream master and BMO

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[David Lawrence [:dkl]]

We will need a VM or actual machine for testing the upcoming merge of upstream trunk and our BMO customizations. The system can/should be identical to the current bugzilla-dev.allizom.org system as far as resources. Definitely not as powerful as production. I am not fussed as far as naming goes. Maybe bugzilla-dev-merge.allizom.org? This system can be decommissioned once we have rolled out the merge to production so this is not a permanent thing.

Thanks
dkl

Comment 1

[Kendall Libby [:fubar] (he/him)]

will this need its own db?

Comment 2

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #1)
> will this need its own db?

Yes. Either as another DB on a current development system or its own separate DB server. The idea is we would take a recent sanitized copy of production, import it to the new DB, and then run checksetup.pl on it to bring the schema up to date with the merged code base.

dkl

Comment 3

[David Lawrence [:dkl]]

The branch has been pushed to git.m.o now \o/ 

http://git.mozilla.org/?p=webtools/bmo/bugzilla.git;a=shortlog;h=refs/heads/upstream-merge [github]

Please use this branch for building out the test system. We should use a sanitized copy of the database similar to what is on bugzilla-dev as well and test the upgrade process of the when checksetup.pl is ran. Worked locally so hopefully nothing blows up.

I will file bugs for TaskCluster, etc. to get it added to our CI system.

dkl

Comment 4

[David Lawrence [:dkl]]

let me know if there is something more we need for this. do you have an estimate on how much longer this will be?

thanks
dkl

Comment 5

[David Lawrence [:dkl]]

friendly ping

Comment 6

[Kendall Libby [:fubar] (he/him)]

working on setting up puppet to install perl modules with cpanm (or something similar)

Comment 7

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #6)
> working on setting up puppet to install perl modules with cpanm (or
> something similar)

The script in bug 1243051 might be helpful later on for this. We would need to get the output from the current production system and check the file into git.

dkl

Comment 8

[Kendall Libby [:fubar] (he/him)]

ok, after some digging I'm going to have to reverse direction on using a cpanfile. unless it's easy to have BMO look elsewhere for modules, other than the system perl. too many other things in RHELland use the system perl, and overwriting things with rpms and/or cpanm is a disaster waiting to happen. 

however, I believe part of my issue with the earlier foray into making RPMs was conflicts with modules installed from rpmforge-extras. turns out the repo config for rpmforge-extras was changed over time to exclude perl modules, but we've still got a bunch installed from said repo, and they like to cause conflicts. 

have removed said modules from merge-test1 and will build new RPMs to cover what's missing. doing this for dev/stage/prod (e.g. bug 1231778) will be ... fun.

Comment 9

[David Lawrence [:dkl]]

https://drive.google.com/open?id=0Bwv9CxBr_bzWeWpIWVU2M1lQeEk

I have uploaded a vendor.tar.gz to google drive that you can now use install the dependencies needed.

I created with a Centos6 system with the following rpms installed initially:

ImageMagick-devel \
expat-devel \
gd-devel \
gmp-devel \
httpd-devel \
mod_perl \
mysql-community-devel \
mysql-community-server \
openssl \
openssl-devel \                                                                                            
perl-core

carton instuctions:

1. cd /data/bugzilla/www/bugzilla.allizom.org
2. tar zxvf /path/to/vendor.tar.gz
3. vendor/bin carton install --cached --deployment

We can commit cpanfile.snapshot once we get close to deployment.

Let me know how this works.

dkl

Comment 10

[Kendall Libby [:fubar] (he/him)]

Gave it a whirl and run into some errors:

perl-Module-CoreList needs to be installed to even run carton; easy fix. Also needed gcc-c++ for Encode::Detect.

more problematic is the tarball is missing IPC::Cmd, which causes all sorts of follow on issues. I can work around by installing perl-IPC-Cmd, but if you can include it that'd be great.

XML::SAX::Writer also appears to be missing.

lastly, but least importantly, the install throws a *lot* of the following:
/bin/tar: Ignoring unknown extended header keyword `SCHILY.dev'
/bin/tar: Ignoring unknown extended header keyword `SCHILY.ino'
/bin/tar: Ignoring unknown extended header keyword `SCHILY.nlink'

Comment 11

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #10)
> Gave it a whirl and run into some errors:
> 
> perl-Module-CoreList needs to be installed to even run carton; easy fix.
> Also needed gcc-c++ for Encode::Detect.
> 
> more problematic is the tarball is missing IPC::Cmd, which causes all sorts
> of follow on issues. I can work around by installing perl-IPC-Cmd, but if
> you can include it that'd be great.
> 
> XML::SAX::Writer also appears to be missing.

Thanks will update my build scripts.

> lastly, but least importantly, the install throws a *lot* of the following:
> /bin/tar: Ignoring unknown extended header keyword `SCHILY.dev'
> /bin/tar: Ignoring unknown extended header keyword `SCHILY.ino'
> /bin/tar: Ignoring unknown extended header keyword `SCHILY.nlink'

That is a tar incompatibility with OSX from what I have read and not harmful.

Do we have a S3 bucket in our AWS cluster for this eventually to live?

dkl

Comment 12

[Kendall Libby [:fubar] (he/him)]

I'll set up an S3 bucket in our devservices account and give you guys write access to it.

Comment 13

[David Lawrence [:dkl]]

https://drive.google.com/open?id=0Bwv9CxBr_bzWeUZnSW1BV3l6ems

Updated file for Centos6 uploaded.

dkl

Comment 14

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #12)
> I'll set up an S3 bucket in our devservices account and give you guys write
> access to it.

Once we start putting these in S3 should we start versioning them with a datestamp or something similar?

dkl

Comment 15

[Kendall Libby [:fubar] (he/him)]

(In reply to David Lawrence [:dkl] from comment #14)
> Once we start putting these in S3 should we start versioning them with a
> datestamp or something similar?


possibly; S3 provides some versioning mechanic, but I haven't looked at it closely yet.

will give the new tarball a try.

Comment 16

[Kendall Libby [:fubar] (he/him)]

missing pre-reqs for IPC::Cmd:

-> FAIL Couldn't find module or a distribution Locale::Maketext::Simple
-> FAIL Couldn't find module or a distribution Module::Load::Conditional
-> FAIL Couldn't find module or a distribution Params::Check (0.20)

Additionally, it's relying on these pre-installed rpms:
perl-Test-Simple-0.98-1.el6.rfx.noarch
perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64

perl-Test-Simple should be in the tarball; rpmforge-extra (.rfx.) rpms are what we want to get away from.

Comment 17

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #16)
> missing pre-reqs for IPC::Cmd:
> 
> -> FAIL Couldn't find module or a distribution Locale::Maketext::Simple
> -> FAIL Couldn't find module or a distribution Module::Load::Conditional
> -> FAIL Couldn't find module or a distribution Params::Check (0.20)
> 
> Additionally, it's relying on these pre-installed rpms:
> perl-Test-Simple-0.98-1.el6.rfx.noarch
> perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64
> 
> perl-Test-Simple should be in the tarball; rpmforge-extra (.rfx.) rpms are
> what we want to get away from.

As mentioned in IRC, please verify that you install perl-core from RHEL6 before installing the tarball.

dkl

Comment 18

[Kendall Libby [:fubar] (he/him)]

can do, however this WILL cause issues when we want to use this on current production.

to wit, there is a conflict between perl-IO-Compress-*, which perl-core wants, and perl-IO-Compress, which is currently installed from rpmforge-extras. removing it requires ripping out a whole bunch of other perl modules and replacing them. and reinstalling them might run into an entirely different set of issues.


much better this time around, only one error:
Searching GSSAPI on mirror index /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.txt ...
-> FAIL Finding GSSAPI (0) on mirror index /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.txt failed.
-> FAIL Couldn't find module or a distribution GSSAPI

Comment 19

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #18)
> can do, however this WILL cause issues when we want to use this on current
> production.
> 
> to wit, there is a conflict between perl-IO-Compress-*, which perl-core
> wants, and perl-IO-Compress, which is currently installed from
> rpmforge-extras. removing it requires ripping out a whole bunch of other
> perl modules and replacing them. and reinstalling them might run into an
> entirely different set of issues.

I suppose I could get you to dump a list of installed rpms as well as yum repo configurations so that I could recreate the same package environment locally. I did not plan for having to co-exist with other custom installed packages.

Also I assume we cannot rebuild each webhead one by one after we roll out the merge to not have all of the extra rpms installed and just rely on the core list provided with the vendor tarball installed? Probably a huge task. Will be easier for AWS at least.

> much better this time around, only one error:
> Searching GSSAPI on mirror index
> /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.
> txt ...
> -> FAIL Finding GSSAPI (0) on mirror index
> /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.
> txt failed.
> -> FAIL Couldn't find module or a distribution GSSAPI

Interesting as it installs for me locally without error. What happens if you do:

cd /path/to/bugzilla
cpanm -l local --notest GSSAPI

dkl

Comment 20

[Dylan Hardison [:dylan] (he/him)]

fubar, can you attach all the output of ./vendor/bin/carton install --cached --deployment?
Something is really weird, GSSAPI isn't required by anything (it is recommended by Authen::SASL) but
as it isn't in the tarball, it should be in the list of things carton attempts to install.

Thanks!

Comment 21

[Kendall Libby [:fubar] (he/him)]

(In reply to David Lawrence [:dkl] from comment #19)
> 
> I suppose I could get you to dump a list of installed rpms as well as yum
> repo configurations so that I could recreate the same package environment
> locally. I did not plan for having to co-exist with other custom installed
> packages.

Well, we're not going to have a clean environment to install into when we finally get around to doing this for prod, unless we specifically set out to rebuild the entire environment. In which case, it'll probably be Centos7, which changes everything again.
But we can figure that out as we go...

I think the biggest wrench is understanding how deploys happen, and thus how I expect this *should* go: code is rsynced from bugzillaadm to each node, which is triggered by running the deploy scripts. We *could* have nodes just install the tarball themselves, checking for changes on a regular basis, but if there's ever a case where we have to coordinate a deploy with a package change we'll be in trouble. 

So, the carton install is thus happening on the admin node, and will be synced out (or maybe use NFS; I'm still on the fence) to the nodes. But the admin node will always be a dirty environment, at least until all of dev/stage/prod are using cartons and we can clean/rebuild it. I can probably clean up some of it now, but it will likely mean stopping cronjobs for the duration (an hour or two, maybe?). Something to mull over.


> Also I assume we cannot rebuild each webhead one by one after we roll out
> the merge to not have all of the extra rpms installed and just rely on the
> core list provided with the vendor tarball installed? Probably a huge task.
> Will be easier for AWS at least.

Remember it's not just webheads and, in particular, there's only one push node. And it'd be quite a bit of work with the MOC's help - it's a total of 13 hosts, assuming memcache nodes don't need changing. And I say above, I wouldn't be inclined to rebuild as RHEL6 so, another wrench.

We *could* do it, particularly since it carries a number of benefits, but it would be a long slog.


> Interesting as it installs for me locally without error. What happens if you
> do:
> 
> cd /path/to/bugzilla
> cpanm -l local --notest GSSAPI

bash: cpanm: command not found

:-P


(In reply to Dylan William Hardison [:dylan] from comment #20)
> fubar, can you attach all the output of ./vendor/bin/carton install --cached
> --deployment?
> Something is really weird, GSSAPI isn't required by anything (it is
> recommended by Authen::SASL) but
> as it isn't in the tarball, it should be in the list of things carton
> attempts to install.

The actual output from carton is useless, imo:

[...]
Successfully installed SOAP-Lite-1.19
Successfully installed XMLRPC-Lite-0.717
Successfully installed Net-SMTP-SSL-1.03
! Couldn't find module or a distribution GSSAPI
Successfully installed XML-SAX-Base-1.08
Successfully installed XML-NamespaceSupport-1.11
[...]

But the cpanm build log shows it happening as a dependency for perl-ldap:

Checking configure dependencies from META.json
- HTTP::Status      ...loaded. (6.11)
- JSON              ...loaded. (2.90)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Net::LDAP
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have LWP::MediaTypes 6 ... Yes (6.02)
Checking if you have Authen::SASL 2.00 ... Yes (2.16)
Checking if you have GSSAPI 0 ... No
Checking if you have HTTP::Status 6 ... Yes (6.11)
Checking if you have HTTP::Negotiate 6 ... Yes (6.01)
Checking if you have File::Compare 0 ... Yes (1.1006)
Checking if you have Digest::HMAC_MD5 0 ... Yes (1.01)
Checking if you have XML::SAX::Base 0 ... No
Checking if you have Digest::MD5 2.35 ... Yes (2.39)
Checking if you have File::Path 2.07 ... Yes (2.07_03)
Checking if you have Time::Local 0 ... Yes (1.1901)
Checking if you have Convert::ASN1 0.2 ... Yes (0.27)
Checking if you have IO::Socket::SSL 1.54 ... Yes (2.024)
Checking if you have IO::File 1.14 ... Yes (1.14)
Checking if you have ExtUtils::MakeMaker 6.64 ... Yes (7.10)
Checking if you have URI::ldap 1.1 ... Yes (1.71)
Checking if you have Text::Soundex 0 ... Yes (3.03)
Checking if you have Test::More 0.9301 ... Yes (1.001014)
Checking if you have HTTP::Response 6 ... Yes (6.11)
Checking if you have MIME::Base64 2.1 ... Yes (3.08)
Checking if you have XML::SAX::Writer 0 ... No
Checking if you have File::Basename 0 ... Yes (2.77)
Checking if you have JSON 0 ... Yes (2.90)
Checking if you have LWP::Protocol 0 ... Yes (6.15)
Checking if you have LWP 6 ... Yes (6.15)
==> Found dependencies: GSSAPI, XML::SAX::Base, XML::SAX::Writer
Searching GSSAPI on mirror index /data/www/bugzilla-merge.allizom.org/local/cach
e/modules/02packages.details.txt ...
-> FAIL Finding GSSAPI (0) on mirror index /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.txt failed.
-> FAIL Couldn't find module or a distribution GSSAPI

Comment 22

[Kendall Libby [:fubar] (he/him)]

Tried installing the first tarball again to see what would happen.

GSSAPI and XML::SAX::Writer both bombed as part of installing perl-ldap:
-> FAIL Finding GSSAPI (0) on mirror index /data/www/bugzilla-merge.allizom.org/local/cache/modu
les/02packages.details.txt failed.
-> FAIL Finding XML::SAX::Writer (0) on mirror index /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.txt failed.

Additionally, Math-Pari is doing something really weird:

Unpacking Math-Pari-2.010808.zip
Copying Math-Pari-2.010808.zip to /data/www/bugzilla-merge.allizom.org/local/cache/authors/id/I/IL/ILYAZ/modules/Math-Pari-2.010808.zip
Entering Math-Pari-2.010808
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.10)
Configuring Math-Pari-2.010808
Running Makefile.PL
Did not find GP/PARI build directory around.
Non-interactive session, autofetching...

Getting GP/PARI from ftp://megrez.math.u-bordeaux.fr/pub/pari/unix/
-> FAIL Timed out (> 60s). Use --verbose to retry.
-> N/A
-> FAIL Configure failed for Math-Pari-2.010808. See /home/klibby/.cpanm/work/1457623799.8391/build.log for details.

I don't understand why it's trying to fetch anything when it's RIGHT THERE.

Comment 23

[David Lawrence [:dkl]]

New Centos6 vendorball uploaded for testing.

https://drive.google.com/open?id=0Bwv9CxBr_bzWYjZPWEp0enN4S00

Comment 24

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #22)
> Tried installing the first tarball again to see what would happen.
> 
> GSSAPI and XML::SAX::Writer both bombed as part of installing perl-ldap:
> -> FAIL Finding GSSAPI (0) on mirror index
> /data/www/bugzilla-merge.allizom.org/local/cache/modu
> les/02packages.details.txt failed.
> -> FAIL Finding XML::SAX::Writer (0) on mirror index
> /data/www/bugzilla-merge.allizom.org/local/cache/modules/02packages.details.
> txt failed.
> 
> Additionally, Math-Pari is doing something really weird:
> 
> Unpacking Math-Pari-2.010808.zip
> Copying Math-Pari-2.010808.zip to
> /data/www/bugzilla-merge.allizom.org/local/cache/authors/id/I/IL/ILYAZ/
> modules/Math-Pari-2.010808.zip
> Entering Math-Pari-2.010808
> Checking configure dependencies from META.json
> Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.10)
> Configuring Math-Pari-2.010808
> Running Makefile.PL
> Did not find GP/PARI build directory around.
> Non-interactive session, autofetching...
> 
> Getting GP/PARI from ftp://megrez.math.u-bordeaux.fr/pub/pari/unix/
> -> FAIL Timed out (> 60s). Use --verbose to retry.
> -> N/A
> -> FAIL Configure failed for Math-Pari-2.010808. See
> /home/klibby/.cpanm/work/1457623799.8391/build.log for details.
> 
> I don't understand why it's trying to fetch anything when it's RIGHT THERE.

Please try the latest tarball and see if it is better. If you still get the GSSAPI error try installing it manually and see what the error is exactly.

wget https://raw.githubusercontent.com/miyagawa/cpanminus/master/cpanm -O /usr/local/bin/cpanm
chmod a+x /usr/local/bin/cpanm
cd /path/to/bugzilla
cpanm -l local -v GSSAPI

Thanks
dkl

Comment 25

[Kendall Libby [:fubar] (he/him)]

Are you sure that's a *new* tarball? diff says it's the same as older ones.

anyways, looks like it's failing on tests, though why that didn't show up before I can't say:

PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/1constants.t ..... ok     
t/2status.t ........ ok   
t/checkoids.t ...... ok     
t/importnames.t .... ok   
t/indicatemechs.t .. 1/5 # KRB5 old Mechtype, Kerberos 5, SPNEGO
t/indicatemechs.t .. ok   
t/inquire_cred.t ... 1/6 
#   Failed test '$cred1->inquire_cred($name, $lifetime, $cred_usage, $oidset'
#   at t/inquire_cred.t line 47.
# inquire_cred()    lifetime:	0 seconds
# inquire_cred()    credusage:	 0 (GSS_C_BOTH)
	(in cleanup) oidset has no value.
# Looks like you failed 1 test of 6.
t/inquire_cred.t ... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/6 subtests 
t/pod.t ............ skipped: Test::Pod 1.00 required for testing POD
t/test.t ........... ok     
t/zbugfixes.t ...... 1/3 # 
# 
#  if you want to run tests that do a realworld *use* of your GSSAPI
#  start a kinit and try to run
# 
# ./examples/getcred_hostbased.pl 
# 
t/zbugfixes.t ...... ok   

Test Summary Report
-------------------
t/inquire_cred.t (Wstat: 256 Tests: 6 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1
Files=9, Tests=148,  1 wallclock secs ( 0.05 usr  0.02 sys +  0.26 cusr  0.04 csys =  0.37 CPU)
Result: FAIL
Failed 1/9 test programs. 1/148 subtests failed.
make: *** [test_dynamic] Error 255
FAIL

Comment 26

[Kendall Libby [:fubar] (he/him)]

looks like GSSAPI test fail isn't anything new: https://rt.cpan.org/Public/Bug/Display.html?id=73293

Is it possible to have carton skip running tests?

Comment 27

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #25)
> Are you sure that's a *new* tarball? diff says it's the same as older ones.
> 
> anyways, looks like it's failing on tests, though why that didn't show up
> before I can't say:
> 
> PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM"
> "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0,
> 'blib/lib', 'blib/arch')" t/*.t
> t/1constants.t ..... ok     
> t/2status.t ........ ok   
> t/checkoids.t ...... ok     
> t/importnames.t .... ok   
> t/indicatemechs.t .. 1/5 # KRB5 old Mechtype, Kerberos 5, SPNEGO
> t/indicatemechs.t .. ok   
> t/inquire_cred.t ... 1/6 
> #   Failed test '$cred1->inquire_cred($name, $lifetime, $cred_usage, $oidset'
> #   at t/inquire_cred.t line 47.
> # inquire_cred()    lifetime:	0 seconds
> # inquire_cred()    credusage:	 0 (GSS_C_BOTH)
> 	(in cleanup) oidset has no value.
> # Looks like you failed 1 test of 6.
> t/inquire_cred.t ... Dubious, test returned 1 (wstat 256, 0x100)
> Failed 1/6 subtests 
> t/pod.t ............ skipped: Test::Pod 1.00 required for testing POD
> t/test.t ........... ok     
> t/zbugfixes.t ...... 1/3 # 
> # 
> #  if you want to run tests that do a realworld *use* of your GSSAPI
> #  start a kinit and try to run
> # 
> # ./examples/getcred_hostbased.pl 
> # 
> t/zbugfixes.t ...... ok   
> 
> Test Summary Report
> -------------------
> t/inquire_cred.t (Wstat: 256 Tests: 6 Failed: 1)
>   Failed test:  4
>   Non-zero exit status: 1
> Files=9, Tests=148,  1 wallclock secs ( 0.05 usr  0.02 sys +  0.26 cusr 
> 0.04 csys =  0.37 CPU)
> Result: FAIL
> Failed 1/9 test programs. 1/148 subtests failed.
> make: *** [test_dynamic] Error 255
> FAIL

Ok new vendor tarball build on bugzillaadm itself, so please try it out.

https://drive.google.com/open?id=0Bwv9CxBr_bzWWnFRWXhZUVJVSUk

dkl

Comment 28

[Kendall Libby [:fubar] (he/him)]

No DateTime, and/or ParseXS, is unhappy (same whether I run it on bugzillaadm or merge-test1):

Building DateTime-1.26
cp lib/DateTime.pm blib/lib/DateTime.pm
cp lib/DateTime/Helpers.pm blib/lib/DateTime/Helpers.pm
cp lib/DateTime/Duration.pm blib/lib/DateTime/Duration.pm
cp lib/DateTime/PP.pm blib/lib/DateTime/PP.pm
cp lib/DateTime/Infinite.pm blib/lib/DateTime/Infinite.pm
cp lib/DateTime/PPExtra.pm blib/lib/DateTime/PPExtra.pm
cp lib/DateTime/LeapSecond.pm blib/lib/DateTime/LeapSecond.pm
Running Mkbootstrap for DateTime ()
chmod 644 "DateTime.bs"
"/usr/bin/perl" "/usr/share/perl5/ExtUtils/xsubpp"  -typemap "/usr/share/perl5/ExtUtils/typemap"
  DateTime.xs > DateTime.xsc && mv DateTime.xsc DateTime.c
Undefined subroutine &ExtUtils::ParseXS::errors called at /usr/share/perl5/ExtUtils/xsubpp line 
44.
make: *** [DateTime.c] Error 255


I almost wonder if it would just be easier to make a new perl-DateTime rather than finding more ways to break the carton...

Comment 29

[David Lawrence [:dkl]]

new vendor tarball

https://drive.google.com/open?id=0Bwv9CxBr_bzWTXNqRk9qdWJ4RnM

dkl

Comment 30

[Kendall Libby [:fubar] (he/him)]

Same error:

"/usr/bin/perl" "/usr/share/perl5/ExtUtils/xsubpp"  -typemap "/usr/share/perl5/ExtUtils/t
ypemap"  DateTime.xs > DateTime.xsc && mv DateTime.xsc DateTime.c
Undefined subroutine &ExtUtils::ParseXS::errors called at /usr/share/perl5/ExtUtils/xsubp
p line 44.
make: *** [DateTime.c] Error 255
-> FAIL Installing DateTime failed. See /home/klibby/.cpanm/work/1458824516.22819/build.log for details. Retry with --force to force install it.

Comment 31

[David Lawrence [:dkl]]

(In reply to Kendall Libby [:fubar] from comment #30)
> Same error:
> 
> "/usr/bin/perl" "/usr/share/perl5/ExtUtils/xsubpp"  -typemap
> "/usr/share/perl5/ExtUtils/t
> ypemap"  DateTime.xs > DateTime.xsc && mv DateTime.xsc DateTime.c
> Undefined subroutine &ExtUtils::ParseXS::errors called at
> /usr/share/perl5/ExtUtils/xsubp
> p line 44.
> make: *** [DateTime.c] Error 255
> -> FAIL Installing DateTime failed. See
> /home/klibby/.cpanm/work/1458824516.22819/build.log for details. Retry with
> --force to force install it.

Ok so chicken and egg issue :( Please do the following instead as a workaround to see if it works:

# curl -L https://cpanmin.us/ -o /usr/local/bin/cpanm && chmod +x /usr/local/bin/cpanm
# cd /path/to/bugzilla
# cpanm -l local --quiet --notest ExtUtils::ParseXS
# tar xvf /path/to/vendor.tar.gz
# perl -Ilocal/lib/perl5 $CARTON install --cached --deployment

dkl

Comment 32

[Kendall Libby [:fubar] (he/him)]

Because I'm a rebel, I actually did:

# tar zxvf /path/to/vendor.tar.gz
# cpanm -l local --quiet --notest ExtUtils::ParseXS
# ./vendor/bin/carton install --cached --deployment

and it worked! zomg. we've spent so long on this bit, I'm not sure what happens next. -.-

Comment 33

[Kendall Libby [:fubar] (he/him)]

deploy script made and added to bugzillaadm, though it's not currently running out of cron (and I haven't tested it, but it's not greatly changed from the existing ones).

I have done a manual update and deploy so the latest code (and carton) is on merge-test1.

set up zeus pools, VIPs and DNS, and you should be able to access it at https://bugzilla-merge.allizom.org/


will need to add a cron job for running this automatically, plus add in any carton updating if we want to do that (I have an s3 bucket ready we can use to store it)

Comment 34

[David Lawrence [:dkl]]

Attachment: rpm_list.diff

diff of rpms from bugzillaadm and bugzilla-merge

Comment 35

[David Lawrence [:dkl]]

Ok. So missed one very big point in rollout planning for all of this. Upstream Bugzilla master just switched not too long ago to require perl 5.14.0 or newer to run. Currently RHEL/Centos6 has perl 5.10.1 which will not work. Over the last meeting or so we decided to not require upgrading to RHEL7 (perl 5.16) before doing the merged code rollout to production. So we have 3 choices:

1) Wait to do the merge rollout until we can update all of our systems to RHEL7. This will require quite a bit of work and coordination with other operations personnel from what I understand. We need to do this eventually anyway but would likely push the merge rollout back quite a bit.

2) Back out the upstream patch that changes the code to require perl 5.14.0 or newer everywhere. This would be the easiest thing to do in the short term. There is possibilities of unicode issues popping up that are fixed in newer versions of Perl. This would also put us out of sync with upstream and may cause conflicts and more development work in future code merges.

3) Install 3rd party rpms of newer versions of Perl/HTTPD/mod_perl onto our production systems bringing us closer to modern versions of Perl, etc.

https://www.softwarecollections.org/en/scls/rhscl/perl516/

I have done this locally in docker and is working best I can tell. Here are the steps I took:

# yum -y install centos-release-scl yum-utils
# yum-config-manager --enable rhel-server-rhscl-6-rpms
# yum -y install httpd24 perl516-mod_perl perl516-mod_perl-devel perl516-perl-ExtUtils-Manifest perl516-perl-core
# ln -s /opt/rh/httpd24/root/usr/sbin/httpd /usr/sbin/httpd
# ln -s /opt/rh/httpd24/root/etc/httpd /etc/httpd
# mv /usr/bin/perl /usr/bin/perl.back
# ln -s /opt/rh/perl516/root/usr/bin/perl /usr/bin/perl

And then install of the BMO perl modules as normal into the local of the document root. As you can see it is sort of hacky and could cause unforseen issues in itself down the road.

I need to know what others thoughts are on this?

dkl

Comment 36

[David Lawrence [:dkl]]

(In reply to David Lawrence [:dkl] from comment #35)
> Ok. So missed one very big point in rollout planning for all of this.
> Upstream Bugzilla master just switched not too long ago to require perl
> 5.14.0 or newer to run. Currently RHEL/Centos6 has perl 5.10.1 which will
> not work. Over the last meeting or so we decided to not require upgrading to
> RHEL7 (perl 5.16) before doing the merged code rollout to production. So we
> have 3 choices:
> 
> 1) Wait to do the merge rollout until we can update all of our systems to
> RHEL7. This will require quite a bit of work and coordination with other
> operations personnel from what I understand. We need to do this eventually
> anyway but would likely push the merge rollout back quite a bit.
> 
> 2) Back out the upstream patch that changes the code to require perl 5.14.0
> or newer everywhere. This would be the easiest thing to do in the short
> term. There is possibilities of unicode issues popping up that are fixed in
> newer versions of Perl. This would also put us out of sync with upstream and
> may cause conflicts and more development work in future code merges.
> 
> 3) Install 3rd party rpms of newer versions of Perl/HTTPD/mod_perl onto our
> production systems bringing us closer to modern versions of Perl, etc.
> 
> https://www.softwarecollections.org/en/scls/rhscl/perl516/
> 
> I have done this locally in docker and is working best I can tell. Here are
> the steps I took:
> 
> # yum -y install centos-release-scl yum-utils
> # yum-config-manager --enable rhel-server-rhscl-6-rpms
> # yum -y install httpd24 perl516-mod_perl perl516-mod_perl-devel
> perl516-perl-ExtUtils-Manifest perl516-perl-core
> # ln -s /opt/rh/httpd24/root/usr/sbin/httpd /usr/sbin/httpd
> # ln -s /opt/rh/httpd24/root/etc/httpd /etc/httpd
> # mv /usr/bin/perl /usr/bin/perl.back
> # ln -s /opt/rh/perl516/root/usr/bin/perl /usr/bin/perl
> 
> And then install of the BMO perl modules as normal into the local of the
> document root. As you can see it is sort of hacky and could cause unforseen
> issues in itself down the road.
> 
> I need to know what others thoughts are on this?
> 
> dkl

Ok. I blame my current head cold I have. I now see that we actually already did #2 and reverted the upstream patch. I was using a new RHEL6 dev environment I built based on what we now have in Taskcluster. I did a new checkout of upstream master and noticed it would error out when running checksetup.pl and panicked thinking this was going to happen for our merge branch. But upon verification it doesn't. I will probably need to so some hacking of my dev environment to also be able to do upstream work and BMO work on the same environment. Or use two different environments, one RHEL6 and one RHEL7 (yuck). Can we make a plan to update our systems to RHEL7 soon? :)

dkl

Comment 37

[Kendall Libby [:fubar] (he/him)]

fwiw...

#3 might be ok to do, but I think it's likely that we'd run into issues with IT puppet and other things that depend on perl (nagios, yum). Not saying it can't be done, just that there are hidden dangers there.

As as far as centos 7 is concerned we're still finding, and fixing, random issues and getting used to it, but it's not out of the realm of possibility. Or maybe taking it out of the data center entirely, is a possibility...

Comment 38

[David Lawrence [:dkl]]

Ok. So I think we can close this bug out as the test instance has been working well for a bit now. We can address the perl version issue once we have a deployment plan for RHEL7 and we can un-revert the upstream patch. We will open new bugs for any further issues related to the test instance.

dkl