362152 - code for grabbing/cleaning set of web pages.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[alice nodelman [:alice] [:anode]]

I've created a set of two scripts to handle grabbing/cleaning web pages.  The first script collects the web pages using wget and the second (heavily based upon a script by Darin Fisher) ensures that all links within the collected pages have been localized.

The code works best using this custom built wget with css parser.
http://www.mail-archive.com/[e-mail]/msg09142.html
While it will collect web pages using a vanilla wget install, the pages collected will be missing most or all of their css.  This information would probably be best covered in a README.

This code belongs in mozilla/testing.  Possibly in something like mozilla/testing/tools/grabber.

Comment 1

[alice nodelman [:alice] [:anode]]

Attachment: script for grabbing web page sets

Comment 2

[alice nodelman [:alice] [:anode]]

Attachment: script to clean web pages

Comment 3

[Robert Helmer [:rhelmer]]

Comment on attachment 246860 [details]
script to clean web pages

might want to do this one in perl if you are going to call out to perl so much, think it'd solve the perf problem.

Comment 4

[(not currently active) Ted Mielczarek]

Nice to see someone using my hacked wget :)

Comment 5

[Rob Campbell [:rc] (:robcee)]

(In reply to comment #4)
> Nice to see someone using my hacked wget :)

It seems like a useful thing to have. :)

(In reply to comment #3)
> might want to do this one in perl if you are going to call out to perl so
> much, think it'd solve the perf problem.

I agree. This should probably be rewritten in perl rather than calling it a bunch of times and evaluating a single expression. Other than that, great stuff.

Where should we put this? In mozilla/testing/performance/tools? In a more general mozilla/testing/tools directory?

Comment 6

[alice nodelman [:alice] [:anode]]

Attachment: getpages.sh attempt #2

I combined the two scripts together and reduced the multiple perl invocations to a single perl command.  I think that this makes the code cleaner, it should also have better performance.

Comment 7

[Robert Helmer [:rhelmer]]

Comment on attachment 246999 [details]
getpages.sh attempt #2

yes, much better.

Comment 8

[alice nodelman [:alice] [:anode]]

Attachment: getpages.sh attempt #3

Bug fix for getpages.sh.  Before the script couldn't handle cleaning downloaded files with spaces in their names (ie, "+ addimg +.html") - use of xargs -0 fixes this issue.

Comment 9

[Robert Helmer [:rhelmer]]

Landed on trunk:

RCS file: /cvsroot/mozilla/testing/tools/grabber/getpages.sh,v
done
Checking in getpages.sh;
/cvsroot/mozilla/testing/tools/grabber/getpages.sh,v  <--  getpages.sh
initial revision: 1.1
done

Comment 10

[Adam Guthrie]

Can this be marked as FIXED now?

Comment 11

[Rob Campbell [:rc] (:robcee)]

yes!

Comment 12

[(not currently active) Ted Mielczarek]

The wget patch mentioned in comment 0 totally landed on wget trunk a while ago. Just thought I'd mention that!