Perl: Downloading Content

Had to download 250 resources from an existing site this weekend. First, I used Firebug to get a list of all the requests, and then parsed it into a list of URLs for only the resources I was interested in (jpeg, png, js css and xml). Using my tiny module of perl utility methods, I was able to accomplish all of this in about five minutes.

Once you have a list of URLS for the resources, pulling them down is no problem at all.

#/usr/bin/perl

use BWright::FNP;
use BWright::Mechanize;

$resource_path = 'C:pathDesktopresources';
$resource_path .= '\';

@resources = &readFile('C:pathDesktopresources_list.txt');

foreach $resource (@resources) {
    $fileName = &getFileName($resource);
    if (&getURL($resource) == 1) {
    $binMode = "";
    if ($fileName =~ /.(jpg|png|gif)/ig) {
        $binMode = "binMode";
    }
        &writeFile($resource_path.$fileName, &getContent(), $binMode);
    }
}

exit;