781522 - Change config of SSS on stage to use HBase stage

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[[DEACTIVATED] Adrian Gaudebert]

Currently SocorroSearchService on stage pulls crashes out of our production HBase servers. As our stage HBase servers now have full data replication (and more as we constantly index fake crashes there), we need to pull crashes from stage HBase instead of prod.

The file to change is /usr/local/SocorroSearchService/conf/hbase-site.xml on hosts socorro-es[1,2].stage.webapp.phx1.mozilla.com. The property "hbase.rootdir" needs to change from its current value "hdfs://hp-admin01.phx1.mozilla.com:8020/hbase" to whatever we use for stage. 

That's my current problem, I don't know the URL of our stage HBase. I heard about "gsgw1000.metrics.phx1.m.c" and "hp-node70.phx1.mozilla.com" but I couldn't find any clear documentation on that. CC'ing :aphadke and :rhelmer as they may know that.

Comment 1

[Brandon Burton [:solarce]]

:aphadke / :rhelmer, can one of you advise on the hostname to use for HBase?

Thanks

Comment 2

[Laura Thomson :laura]

tmary is the right person to answer.

Comment 3

[Laura Thomson :laura]

I also found this VIP 
socorro-thrift-stage.zlb.phx1.mozilla.com
in
https://mana.mozilla.org/wiki/display/websites/crash-stats.mozilla.com+%28Socorro%29#crash-statsmozillacomSocorro-HBase

tmary should be able to confirm.

Comment 4

[T [:tmary] Meyarivan]

For HBase/HDFS config information, please login into the Socorro specific GW (gsgw1000.metrics.phx1.mozilla.com) and lookup local configs (/etc/hbase/conf, /etc/hadoop/conf)

Re VIP etc, IT/Infra would be able to confirm whether it points to the correct IPs etc - IIUC this would be the same VIP as used by the staging processors 

--

Comment 5

[Brandon Burton [:solarce]]

(In reply to T [:tmary] Meyarivan from comment #4)
> For HBase/HDFS config information, please login into the Socorro specific GW
> (gsgw1000.metrics.phx1.mozilla.com) and lookup local configs
> (/etc/hbase/conf, /etc/hadoop/conf)
> 
> Re VIP etc, IT/Infra would be able to confirm whether it points to the
> correct IPs etc - IIUC this would be the same VIP as used by the staging
> processors 
> 
> --

What's the gateway to use for staging?

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

According to this page https://intranet.mozilla.org/Cluster_info_phx_prd_cluster04 and what we use in our stage config, it seems the URL to use is the following:

hdfs://hp-node61.phx1.mozilla.com:8020/hbase

Thanks rhelmer and tmary, and please let me know if I'm wrong here.

Comment 7

[T [:tmary] Meyarivan]

(In reply to Adrian Gaudebert [:adrian] from comment #6)
> According to this page
> https://intranet.mozilla.org/Cluster_info_phx_prd_cluster04 and what we use
> in our stage config, it seems the URL to use is the following:
> 
> hdfs://hp-node61.phx1.mozilla.com:8020/hbase
> 

IMHO it would be simpler to copy the configs from hp-node61:/etc/hbase/conf instead of editing them in place

--

Comment 8

[[DEACTIVATED] Adrian Gaudebert]

(In reply to T [:tmary] Meyarivan from comment #7)
> IMHO it would be simpler to copy the configs from hp-node61:/etc/hbase/conf
> instead of editing them in place

Mmm I'm not sure what you mean, but the goal here is to change the value of one config key on our stage ElasticSearch hosts. Precise instructions: 

on hosts: socorro-es[1,2].stage.webapp.phx1.mozilla.com
in file: /usr/local/SocorroSearchService/conf/hbase-site.xml
in key: hbase.rootdir
replace: hdfs://hp-admin01.phx1.mozilla.com:8020/hbase
with: hdfs://hp-node61.phx1.mozilla.com:8020/hbase

Comment 9

[[DEACTIVATED] Adrian Gaudebert]

To be even clearer, we're not changing the config of HBase, we are changing the config of SocorroSearchService (doc: https://mana.mozilla.org/wiki/display/websites/Socorro+Search+Service ). I am not aware of another way of changing that config, if there is any please let me know.

Comment 10

[Corey Shields [:cshields]]

Adrian, tmary says you have the relevant address to change to, let me know what line(s) need to change and what to change them to.

Comment 11

[[DEACTIVATED] Adrian Gaudebert]

We're not using SocorroSearchService anymore, this is invalid.