HTTP::Request: how to download to a file 
Author Message
 HTTP::Request: how to download to a file

My code downloads a large amount of data using something like:

  my $url = " http://www.*-*-*.com/ ;;
  my $ua = LWP::UserAgent->new();
  my $res =
    $ua->request(POST $url,
                 [ field1 => "foo",
                   field2 => "bar",
                   field3 => "frobozz ]);

  if($res->is_success) {
    print SOME_FILE $res->content;
  }

This works OK, but it takes a lot of memory ($res->content gets to be
huge).  Is there a way to have the request method dump the response's
content directly to a file as it comes in?

Thanks!

ellen



Mon, 21 Jun 2004 03:00:06 GMT  
 HTTP::Request: how to download to a file


Quote:

> My code downloads a large amount of data using something like:

>   my $url = "http://www.some.site/some_cgi_script.pl";
>   my $ua = LWP::UserAgent->new();
>   my $res =
>     $ua->request(POST $url,
>            [ field1 => "foo",
>    field2 => "bar",
>    field3 => "frobozz ]);

>   if($res->is_success) {
>     print SOME_FILE $res->content;
>   }

> This works OK, but it takes a lot of memory ($res->content gets to be
> huge).  Is there a way to have the request method dump the response's
> content directly to a file as it comes in?

> Thanks!

> ellen

Hi ellen,

Take a look at the lwpcook document you have 2 choices:
You can instruct the library to write the document content to a file (second
$ua->request() argument is a file name) or
you can process the document as it arrives (second $ua->request() argument
is a code reference):

try something like this:
my $savefile ='path to file';
my $req =  POST 'http://www.some.site/some_cgi_script.pl',
                 [field1 => 'foo', field2 => 'bar', field3 => 'frobozz' ];
my $ua = LWP::UserAgent->new();
open (OUT,">$savefile") or die "Can't open $savefile:$!\n";
my $request = $ua->request($req,
                                sub {

                   $bytes_received += length($chunk);
                   unless (defined $expected_length) {
                      $expected_length = $request->content_length || 0;
                   }
                   if ($expected_length) {
                        printf STDERR "%d%% - ",
                                  100 * $bytes_received / $expected_length;
                   }
                   print STDERR "$bytes_received bytes received\n";
                   print OUT $chunk;
               });
close (OUT);

I hope this helps

Ron



Mon, 21 Jun 2004 21:39:29 GMT  
 HTTP::Request: how to download to a file

Quote:

> My code downloads a large amount of data using something like:

>   my $url = "http://www.some.site/some_cgi_script.pl";
>   my $ua = LWP::UserAgent->new();
>   my $res =
>     $ua->request(POST $url,
>                  [ field1 => "foo",
>                    field2 => "bar",
>                    field3 => "frobozz ]);

>   if($res->is_success) {
>     print SOME_FILE $res->content;
>   }

> This works OK, but it takes a lot of memory ($res->content gets to be
> huge).  Is there a way to have the request method dump the response's
> content directly to a file as it comes in?

I would suggest you RTFM.  If you did that, you would discover you could
do:

my $res = $ua->request(POST( $url, [
    field1 => "foo",
    field2 => "bar",
    field3 => "frobozz
]), "filename");

--
Klein bottle for rent - inquire within.



Tue, 22 Jun 2004 03:59:26 GMT  
 HTTP::Request: how to download to a file

Quote:
>I would suggest you RTFM.  If you did that, you would discover you could
>do:
>my $res = $ua->request(POST( $url, [
>    field1 => "foo",
>    field2 => "bar",
>    field3 => "frobozz
>]), "filename");

i had tried this already, exactly as in the FM:

  my $res = $ua->request($req, '/tmp/filename');

but it fails with the message:

  <HTML>
  <HEAD><TITLE>An Error Occurred</TITLE></HEAD>
  <BODY>
  <H1>An Error Occurred</h1>
  404 Can't chdir to tmp
  </BODY>
  </HTML>

the FM says nothing about this.

e.



Tue, 22 Jun 2004 21:40:56 GMT  
 HTTP::Request: how to download to a file

Quote:


> >I would suggest you RTFM.  If you did that, you would discover you
> >could do:

> >my $res = $ua->request(POST( $url, [
> >    field1 => "foo",
> >    field2 => "bar",
> >    field3 => "frobozz
> >]), "filename");

> i had tried this already, exactly as in the FM:

>   my $res = $ua->request($req, '/tmp/filename');

> but it fails with the message:

>   <HTML>
>   <HEAD><TITLE>An Error Occurred</TITLE></HEAD>
>   <BODY>
>   <H1>An Error Occurred</h1>
>   404 Can't chdir to tmp
>   </BODY>
>   </HTML>

> the FM says nothing about this.

If you're getting an error like this, then you've constructed your req
incorrectly.  An html message like this is from the http server, not
from how your client chose to deal with the data the server sent.

There should be no significant difference between:
    open FILE, "/tmp/filename";
    print FILE $ua->request($req)->content;
and:
    $ua->request($req, "/tmp/filename");
and:
    open FILE, "/tmp/filename";
    $ua->request($req, sub { print FILE $_[0] } );

Except of course that the first would take more memory.
I'll admit, this doesn't mean that there *won't* be any difference, just
that there shouldn't be one.

--
DATA COMPRESSION: What you get when you squish an android



Sat, 26 Jun 2004 11:06:03 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. http client file download?

2. Download files using HTTP / Perl

3. file download via http

4. http client file download?

5. http file download module

6. perl module for http download of file

7. Download a file via HTTP

8. Downloading a file w/an HTTP address

9. http client file download?

10. file name in http plain text download?

11. download file via http

12. http client file download?

 

 
Powered by phpBB® Forum Software