Getting files off remote servers 
Author Message
 Getting files off remote servers

Is it possible to get a Perl cgi script to open up a file that's located on
a different server?  Say I want my script to fetch a html page located on a
different site, and skim through it to get some information, or check to
see the last time it was updated - is this possible?

And is it possible to do this if the other page is dynamically generated
rather than hard-coded?  I want my script to call a cgi program on someone
else site and treat the resulting response the same way I would a file
opened on my own server - can I do this?

(Just as an aside, this isn't intended for any nefarious pupose, like
stealing someone's pages, or gathering email addresses for spam)

Thanks

Stuart Carroll

--
Posted via CNET Help.com
http://www.*-*-*.com/



Wed, 02 Jul 2003 00:30:09 GMT  
 Getting files off remote servers


Quote:
>Is it possible to get a Perl cgi script to open up a file that's located on
>a different server?  Say I want my script to fetch a html page located on a
>different site, and skim through it to get some information, or check to
>see the last time it was updated - is this possible?

        use LWP::Simple;
        my $html = get('http://perl.plover.com/perl.html');

Quote:
>And is it possible to do this if the other page is dynamically generated
>rather than hard-coded?

Not only is it possible, but there is no way for your program to tell
the difference.

rd
($p{$_})&6];$p{$_}=/ ^$P/ix?$P:close$_}keys%p}p;p;p;p;p;map{$p{$_}=~/^[P.]/&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];sleep rand(2)if/\S/;print



Wed, 02 Jul 2003 00:49:08 GMT  
 Getting files off remote servers

Remember, in Perl, easy things are easy, and hard things are possible
(mostly ;) --from the Perl Cook Book

The easiest way is to use LWP, as far as your requested file
is accessible from a web server's document path:

use LWP::UserAgent;

my $ua = LWP::UserAgent->new(agent => "$ENV{'SCRIPT_NAME'} V$VERSION");
my $request = HTTP::Request->new('GET', 'http://localhost');
my $response = $ua->request($request);
print "Content-type: text/plain\n\n";
unless ($response->is_success) {
    print $response->error_as_HTML . "\n";
    print "Could not connect to server\n";
    exit(1);

Quote:
}

my $Result = $response->content(); # content without HTTP header
print $Result;

If your need is to fetch a file beyond the web servers document
root, you have to hack one more CGI script on the remote machine
which slurps the requested file and prints its content to STDOUT.

Check out also from CPAN:
Net::FTP

State-of-the-art would be to install rsync (http://www.rsync.org),
which would be the most performant and secure solution anyway....

--Reto



Quote:
>Is it possible to get a Perl cgi script to open up a file that's located
>on a different server?  Say I want my script to fetch a html page
>located on a different site, and skim through it to get some
>information, or check to see the last time it was updated - is this
>possible?

>And is it possible to do this if the other page is dynamically generated
>rather than hard-coded?  I want my script to call a cgi program on
>someone else site and treat the resulting response the same way I would
>a file opened on my own server - can I do this?

>(Just as an aside, this isn't intended for any nefarious pupose, like
>stealing someone's pages, or gathering email addresses for spam)

>Thanks

>Stuart Carroll



Wed, 02 Jul 2003 01:06:09 GMT  
 Getting files off remote servers

Quote:

> Is it possible to get a Perl cgi script to open up a file that's located on
> a different server?  Say I want my script to fetch a html page located on a
> different site, and skim through it to get some information, or check to
> see the last time it was updated - is this possible?

Using many different methods, yes, this is possible.

Quote:
> And is it possible to do this if the other page is dynamically generated
> rather than hard-coded?  I want my script to call a cgi program on someone
> else site and treat the resulting response the same way I would a file
> opened on my own server - can I do this?

The method to use here is LWP. Check Cpan. When you see a question
about getting a Web page from a Web server, think, "LWP".

Quote:
> (Just as an aside, this isn't intended for any nefarious pupose, like
> stealing someone's pages, or gathering email addresses for spam)

I hope not. However, let me tell you my feelings on that.
Programming is a lot like fire, talk radio, guns, or what the
occultists say about magic. It is not the tool that is good or evil,
only the person using it. That is to say, you can use a tool towards
benevolent ends or towards malevolent ends, and can use it in
benevolent ways or malevolent ways to reach those ends, but the tools
themselves are blameless.

Chris

--
Christopher E. Stith

Parking for people we like only. All other vehicles will be vandalized.



Wed, 02 Jul 2003 02:17:31 GMT  
 Getting files off remote servers

Quote:

>Programming is a lot like fire, talk radio, guns, or what the
>occultists say about magic. It is not the tool that is good or evil,
>only the person using it. That is to say, you can use a tool towards
>benevolent ends or towards malevolent ends, and can use it in
>benevolent ways or malevolent ways to reach those ends, but the tools
>themselves are blameless.

In some sense that is quite true indeed.  Yet in others.. let me give
an example: A major reason, why spam isn't even more of a problem than
it already is, is that no-one capable of actually writing *good* tools
for spamming and address harvesting feels any inclination to do so.

Yes, we do have Mail::Bulkmail, but I haven't really heard of any of
the actual spam tools using it.  Mostly because you'd still need a
programmer to write the user interface for the canned tool.

It's actually quite amazing.  There are well written viruses out
there, but no smart address harvesters.  How do I know it?  First,
because I haven't heard of any, but also because I haven't *ever*
encountered any myself.

There are address harvesters that search the WWW.  There are those
that grep news overview lists.  Despite widespread paranoia, I've
never heard of or witnessed an address being harvested from the body
of a Usenet post, or from any other header except "From:".

And at least most of the news harvesters seem to use broken regexen
for finding the addresses.  Ironically, I know this because using

I've also heard reports that any address with punctuation other than
. or _ in it is virtually never harvested intact.

I just recently had a piece of spam evade my filters for the first

still haven't heard back from the maintainers of the relay they used
-- I guess I ought to see if it's still open, and nominate them for
the RBL if it is.

--
Ilmari Karonen - http://www.sci.fi/~iltzu/
"If that's a reasonable definition, I prefer unreasonable ones."
                    -- Simon van Dongen in rec.arts.sf.science



Wed, 02 Jul 2003 07:32:12 GMT  
 Getting files off remote servers

Quote:

> I just recently had a piece of spam evade my filters for the first

> still haven't heard back from the maintainers of the relay they used
> -- I guess I ought to see if it's still open, and nominate them for
> the RBL if it is.

Don't bother - someone already did that for you :)  MAPS and ORBS
are liberating in that way (the diligence of a few benefits us all).

--
Joe Schaefer



Wed, 02 Jul 2003 08:01:01 GMT  
 Getting files off remote servers

Quote:


>> I just recently had a piece of spam evade my filters for the first

>> still haven't heard back from the maintainers of the relay they used
>> -- I guess I ought to see if it's still open, and nominate them for
>> the RBL if it is.

>Don't bother - someone already did that for you :)  MAPS and ORBS
>are liberating in that way (the diligence of a few benefits us all).

Actually, they closed the relay after I wrote that.  But the same
spammer is still out there, and sent me another copy over another
relay.  Oh well, that one's been forwarded to the postmasters along
the chain as well.  He ought to be LARTed soon enough.

--
Ilmari Karonen -- http://www.sci.fi/~iltzu/
"Get real!  This is a discussion group, not a helpdesk.  You post
 something, we discuss its implications.  If the discussion happens to
 answer a question you've asked, that's incidental." -- nobull in clpm



Wed, 02 Jul 2003 20:18:50 GMT  
 
 [ 7 post ] 

 Relevant Pages 

1. Getting Path variable from remote NT server

2. Getting File stats on a remote file

3. Getting Remote File Size

4. HELP! Getting Remote File Size

5. getting file fro ma remote computer ?

6. getting remote file sizes

7. getting the results from a remote cgi script to a text file

8. Open Off-Server-File

9. milli-second and html file on remote server

10. search a file on a remote server

11. Reading files on a remote server ????????

12. pl reads files on remote server?

 

 
Powered by phpBB® Forum Software