1128832 - file error - recursion into 'global/header.html.tmpl' && The MySQL server is running with the --read-only option

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[:glob ✱]

after today's push i'm seeing a lot of the following:

  file error - recursion into 'global/header.html.tmpl'

at /usr/lib64/perl5/Template/Context.pm line 500
	Template::Context::throw(...) called at /usr/lib64/perl5/Template/Context.pm line 385
	Template::Context::process(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Template/Context.pm line 45
	Bugzilla::Template::Context::process(...) called at /usr/lib64/perl5/Template/Service.pm line 94
	eval {...} called at /usr/lib64/perl5/Template/Service.pm line 91
	Template::Service::process(...) called at /usr/lib64/perl5/Template.pm line 66
	Template::process(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Template.pm line 674
	Bugzilla::Template::process(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Error.pm line 102
	Bugzilla::Error::_throw_error(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Error.pm line 195
	Bugzilla::Error::ThrowCodeError(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/DB.pm line 181
	Bugzilla::DB::_handle_error(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Token.pm line 457
	Bugzilla::Token::_create_token(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Token.pm line 65
	Bugzilla::Token::issue_api_token(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Template.pm line 1044
	Bugzilla::Template::__ANON__(...) called at /data/www/bugzilla.mozilla.org/template/en/default/global/header.html.tmpl line 244
	eval {...} called at /data/www/bugzilla.mozilla.org/template/en/default/global/header.html.tmpl line 18
	Template::Provider::__ANON__(...) called at /usr/lib64/perl5/Template/Document.pm line 151
	eval {...} called at /usr/lib64/perl5/Template/Document.pm line 149
	Template::Document::process(...) called at /usr/lib64/perl5/Template/Context.pm line 351
	eval {...} called at /usr/lib64/perl5/Template/Context.pm line 321
	Template::Context::process(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Template/Context.pm line 45
	Bugzilla::Template::Context::process(...) called at /data/www/bugzilla.mozilla.org/template/en/default/list/list.html.tmpl line 1
	eval {...} called at /data/www/bugzilla.mozilla.org/template/en/default/list/list.html.tmpl line 18
	Template::Provider::__ANON__(...) called at /usr/lib64/perl5/Template/Document.pm line 151
	eval {...} called at /usr/lib64/perl5/Template/Document.pm line 149
	Template::Document::process(...) called at /usr/lib64/perl5/Template/Context.pm line 351
	eval {...} called at /usr/lib64/perl5/Template/Context.pm line 321
	Template::Context::process(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Template/Context.pm line 45
	Bugzilla::Template::Context::process(...) called at /usr/lib64/perl5/Template/Service.pm line 94
	eval {...} called at /usr/lib64/perl5/Template/Service.pm line 91
	Template::Service::process(...) called at /usr/lib64/perl5/Template.pm line 66
	Template::process(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Template.pm line 674
	Bugzilla::Template::process(...) called at /data/www/bugzilla.mozilla.org/buglist.cgi line 1108
	ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::data_www_bugzilla_2emozilla_2eorg_buglist_2ecgi::handler(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204
	eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204
	ModPerl::RegistryCooker::run(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 170
	ModPerl::RegistryCooker::default_handler(...) called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 31
	ModPerl::Registry::handler(...) called at /data/www/bugzilla.mozilla.org/mod_perl.pl line 143
	Bugzilla::ModPerl::ResponseHandler::handler(...) called at -e line 0
	eval {...} called at -e line 0

Comment 1

[:glob ✱]

the root cause appears to be:

> Bugzilla::DB::_handle_error(...) called at /data/www/bugzilla.mozilla.org/Bugzilla/Token.pm line 457

it's failing to insert into the tokens table, and because of the broken error handling it's failing to report on it.



my guess is we're trying to update the database while pointing to the shadow db; but we'll need the ThrowUserError disabled before i can tell for sure.  (even with a shadow-db configured on stage, i can't reproduce it there).

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   5142db3..58c6e30  master -> master
To ssh://[e-mail]/webtools/bmo/bugzilla.git
   58c6e30..205a5d3  master -> master

Comment 2

[:glob ✱]

and now:

[22:52:43] Adding foreign key: user_api_keys.user_id -> profiles.userid...
[22:52:43] DBD::mysql::db do failed: Can't write; duplicate key in table '#sql-1e11_33f9f0' [for Statement "ALTER TABLE user_api_keys ADD 
[22:52:43]      CONSTRAINT fk_user_api_keys_user_id_profiles_userid FOREIGN KEY (user_id)
[22:52:43]      REFERENCES profiles(userid)
[22:52:43]       ON UPDATE CASCADE ON DELETE CASCADE"] at Bugzilla/DB.pm line 680.
[22:52:43] checksetup.pl exited with return code 255!

bz_schema and the schema itself for user_api_keys is identical between stage, production, and my local dev box.  i'm flummoxed as to why this is failing only on production.

Comment 3

[:glob ✱]

ah, there's a minor difference in bz_schema, user_api_keys.user_id's references has 'created' set to 0.

    'user_id',
    {
        'NOTNULL' => 1,
        'REFERENCES' => {
            'COLUMN' => 'userid',
            'DELETE' => 'CASCADE',
            'TABLE' => 'profiles',
            'created' => 0
        },
        'TYPE' => 'INT3'
    },

i'll throw together a script which manually tweaks that on production.


i suspect this failed because..

[21:24:40] Adding new column 'cf_rank' to the 'bugs' table...
[21:26:07] Creating IT permission grant bug for new field 'cf_rank'...Error creating IT bug for new field cf_rank: There is no component named 'Server Operations: Database' in the
[21:26:07] 'mozilla.org' product.
[21:26:07] 
[21:26:07] Error: There is no component named 'Server Operations: Database' in the
[21:26:07] 'mozilla.org' product.
[21:26:08] 
[21:26:08] Adding foreign key: user_api_keys.user_id -> profiles.userid...
[21:26:08] checksetup.pl complete.

perhaps the bug creation failure put checksetup in weird state with respects to transactions, resulting in the schema changes being made, but changes bz_schema not being committed.

Comment 4

[:glob ✱]

23:53:40 #2 [error] DBD::mysql::db do failed: The MySQL server is running with the --read-only option so it cannot execute this statement [for Statement "INSERT INTO tokens (userid, issuedate, token, tokentype, eventdata) VALUES (?, NOW(), ?, ?, ?)"] (Bugzilla/Token.pm:457)

i'll fix this and then try to see what's happening with stage's shadow_db.

Comment 5

[:glob ✱]

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   205a5d3..5426a2b  master -> master

Comment 6

[:glob ✱]

looks like stage _is_ set up correctly -- i had some stale api_tokens in the tokens table.  after nuking the whole tokens table i was able to reliable reproduce the issue on stage.

Comment 7

[:glob ✱]

it appears the change in comment 5 fixed the issue.
that fix is a hack, and may cause other issues.
i've raised bug 1128853 upstream to get a better fix.

i've raised bug 1128856 upstream for the file recursion issue.

i updated the product/component for the dba bug creation.