407050 - Nutch blocking bugs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Fred Wenzel [:wenzel]]

There are a number of Nutch-related, blocking bugs in the MDC component (will connect them to this bug shortly).

After being unable to find a trace of Nutch code anywhere in our repositories, I believe Nutch is entirely in IT's domain.

Thus the three bugs probably need to be fixed by IT; please let me know if this assessment is right. Work to be done for all three of them includes changing Nutch filter settings, and possibly upgrading to the latest version.

Comment 1

[Jeremy Baron]

Have you seen http://svn.mozilla.org/projects/developer.mozilla.org/trunk/extensions/SpecialNutch.php  ? (and the other couple nutch pages in that dir)

Comment 2

[Fred Wenzel [:wenzel]]

Yes I have, thanks! Yet the bugs specifically cover the indexing, not the querying side of Nutch, as far as I can tell right now.

(adding bug dependencies)

Comment 3

[Aravind Gottipati [:aravind]]

You are right.  The nutch search instance for MDC is managed by us.  There was however a development nutch instance setup for Andrei and maybe others in the docs team.  I gave them access to the nutch instance there etc.  Not sure how actively they have been using it.

Any updates to nutch should probably be tested there first?

Comment 4

[Fred Wenzel [:wenzel]]

(In reply to comment #3)
> There was
> however a development nutch instance setup for Andrei and maybe others in the
> docs team.  I gave them access to the nutch instance there etc.  Not sure how
> actively they have been using it.
> 
> Any updates to nutch should probably be tested there first?

I believe so too! So please give me access to the dev instance of Nutch too and let me know how to get to it. Thanks!

Comment 5

[Aravind Gottipati [:aravind]]

dm-devnutch01 is your box.  It was the search instance being used in the development instance of MDC.  You should be able to login to this box as yourself (fwenzel) after bouncing off the vpn box.

Once you login to this box, "sudo -H -u nutch -c /bin/bash"  Should get you to the nutch environment.

Regarding the code itself.  We currently use stock nutch (release 0.8) and tomcat (release 5.5.23).  Beyond that I *think* we just use some basic nutch crawl and indexing.  Once you are ready to upgrade nutch code, please check it into some kind of source control system, so I can pull from there.  This way, the next person looking at it will have a better idea of what/where to look.

Comment 6

[Aravind Gottipati [:aravind]]

Also, can we close this bug at this point, since there is not much for us to do at this point.  Please open a new bug or re-open this with some pointers on where we can get the code.

Comment 7

[Michael Morgan [:morgamic]]

Yes, thanks Aravind that's what we were looking for.

Comment 8

[Fred Wenzel [:wenzel]]

Yup, thanks Aravind.

I have another problem: The box you mentioned seems to be down: When going through the VPN, I get:
ssh: connect to host dm-devnutch01.mozilla.org port 22: Connection refused

when I jump off the jumphost, I don't get any error, SSH just waits silently.

Please check if the box is up and running.

Comment 9

[Aravind Gottipati [:aravind]]

Sorry, try sm-devnutch01.  I had it in the wrong network originally.