URL checker. 
Author Message
 URL checker.

Hi,

I have a database with URL's in it. Is there a script which can check if the
URL's are still valid??

The database looks something like this:

|Site name|Site description and keywords|Site URL

Thanks for your help, would be apreciated!!

Victor




Sun, 26 Aug 2001 03:00:00 GMT  
 URL checker.

Quote:

> Hi,

> I have a database with URL's in it. Is there a script which can check if the
> URL's are still valid??

> The database looks something like this:

> |Site name|Site description and keywords|Site URL

Here is ripped functionality from my TEXT 2 HTML conversion Perl program
that also checks the URL links in the document. You need LIB WWW.

Don't mind the eval, because this is part of the "on demand functions".

jari

        eval "use LWP::UserAgent";

        my $ua      = new LWP::UserAgent;
        my $request = new HTTP::Request( 'GET', $url );
        my $obj     = $ua->request( $request );

        if ( $obj->is_success )
        {
            0;
        }
        else
        {
            #  There is error

            my $txt = $obj->message;
            ...
        }



Sun, 26 Aug 2001 03:00:00 GMT  
 URL checker.
[F'up narrowed; comp.lang.perl is long dead]

: I have a database with URL's in it. Is there a script which can check if the
: URL's are still valid??

If there isn't, write one -- it's not at all hard to do (if you can perl,
and use the libwww-perl (or LWP for short)).

cu,
  Martin
--
                        | Martin Vorlaender | VMS & WNT programmer

 Microsoft wants        |       http://www.pdv-systeme.de/users/martinv/



Sun, 26 Aug 2001 03:00:00 GMT  
 URL checker.

Quote:


>> Hi,

>> I have a database with URL's in it. Is there a script which can check if the
>> URL's are still valid??

>> The database looks something like this:

>> |Site name|Site description and keywords|Site URL

>Here is ripped functionality from my TEXT 2 HTML conversion Perl program
>that also checks the URL links in the document. You need LIB WWW.

>Don't mind the eval, because this is part of the "on demand functions".

>jari

>    eval "use LWP::UserAgent";

>    my $ua      = new LWP::UserAgent;
>    my $request = new HTTP::Request( 'GET', $url );

That line needs to be changed to
        my $request = new HTTP::Request( 'HEAD', $url );
which will check to see that the document exists WITHOUT downloading it.

Quote:
>    my $obj     = $ua->request( $request );

>    if ( $obj->is_success )
>    {
>        0;
>    }
>    else
>    {
>        #  There is error

>        my $txt = $obj->message;
>        ...
>    }

--
INWAP.COM is Joe Smith, Sally Smith and our cat Murdock.
(The O'Hallorans and their cats moved to http://www.tyedye.org/ Nov-98.)
See http://www.inwap.com/ for PDP-10, "ReBoot", "Shadow Raiders"/"War Planets"


Sun, 26 Aug 2001 03:00:00 GMT  
 URL checker.

Quote:


> >       my $ua      = new LWP::UserAgent;
> >       my $request = new HTTP::Request( 'GET', $url );

> That line needs to be changed to
>         my $request = new HTTP::Request( 'HEAD', $url );
> which will check to see that the document exists WITHOUT downloading it.

HEAD is not the total answer because there are still servers
that do not understand it.  it the HEAD fails, revert to GET.  HEAD
can only tell you that a URL has something behind it. it can't tell
you that it doesn't, necessarily.

--
brian d foy                    
CGI Meta FAQ <URL:http://www.smithrenaud.com/public/CGI_MetaFAQ.html>



Sun, 26 Aug 2001 03:00:00 GMT  
 URL checker.

The April 1997 issue of Linux Journal (pg 84) has an article including a

http://w3.one.net/~jweirich/lj/).  It is a fairly comprehensive script.

Hope this helps.

Quote:
>Hi,

>I have a database with URL's in it. Is there a script which can check if the
>URL's are still valid??

>The database looks something like this:

>|Site name|Site description and keywords|Site URL

>Thanks for your help, would be apreciated!!

>Victor





Sun, 26 Aug 2001 03:00:00 GMT  
 URL checker.


askoczur> The April 1997 issue of Linux Journal (pg 84) has an article including a

askoczur> http://w3.one.net/~jweirich/lj/).  It is a fairly comprehensive script.

And for my WebTechniques column, I now have four successively improved
link verification scripts... see them progress at
<URL:http://www.stonehenge.com/merlyn/WebTechniques/>.

print "Just another Perl hacker,"

--
Name: Randal L. Schwartz / Stonehenge Consulting Services (503)777-0095
Keywords: Perl training, UNIX[tm] consulting, video production, skiing, flying

Web: <A HREF="http://www.stonehenge.com/merlyn/">My Home Page!</A>
Quote: "I'm telling you, if I could have five lines in my .sig, I would!" -- me



Mon, 27 Aug 2001 03:00:00 GMT  
 
 [ 7 post ] 

 Relevant Pages 

1. URL checker.

2. URL->Text->URL

3. convert escape url to unescaped url

4. URL - How can I tell from within PERL if a URL is Up

5. Attachment checker ?

6. duplicate checker and remove from array?

7. spell checker...

8. Puzzling DTD parsing error with XML::Checker

9. Perl spell checker

10. ncc - News Consistency Checker

11. rfc822 address checker

12. open source spell checker for German

 

 
Powered by phpBB® Forum Software