Capturing STDOUT and STDERR and preserving order? 
Author Message
 Capturing STDOUT and STDERR and preserving order?

Hi,

I've been dabbling a bit with scripts designed to run a script on a local
machine and report the results in a web page. IPC::Open3 allows me to
capture STDOUT and STDERR (and thus display STDERR differently, say in bold
or italics), but I haven't found a way yet of reliably reading from the two
filehandles in the correct order. Any ideas?

Sam



Tue, 28 Sep 2004 00:03:48 GMT  
 Capturing STDOUT and STDERR and preserving order?

Quote:

> Hi,

> I've been dabbling a bit with scripts designed to run a script on a
> local machine and report the results in a web page. IPC::Open3 allows
> me to capture STDOUT and STDERR (and thus display STDERR differently,
> say in bold or italics), but I haven't found a way yet of reliably
> reading from the two filehandles in the correct order. Any ideas?

Here's a bit of untested code which might do what you want:

#!/usr/local/bin/perl -w
use strict;
use IO::Select;
print "Content-type: text/html\n\n";
print "<html><head><title>Output of prog</title><head><body>\n";
print "<pre>";
my ($pid, $o, $e);
$pid = eval { open3(my($i), $o, $e, "prog") } or
   print("<b>Error spawning prog: $!</b></pre></html>"), exit;
my $sel = IO::Select->new($o, $e);
my ($obuf, $ebuf);
while( $sel->handles ) { for my $r ( $sel->can_read ) {
   my $buf = $r == $e ? \$ebuf : \$obuf;
   my ($beg, $end) = $r == $e ? (qw(<i> </i>)) : ("","");
   my $n = sysread( $r, $$buf, 8192, length $$buf );
   unless( defined($n) ) {
      print "<b>Error in sysread: $!</b></pre></html>";
      exit;
   } elsif( $n ) {
      while( $$buf =~ s/(.*)\n// ) {
          my $line = $1;
          s/&/&amp;/g, s/</&lt;/g for $line;
          print $beg, $line, $end, "\n";
      }
   } else {
      $sel->remove($r);
      close $r; # beware! remove must be before close
      s/&/&amp;/g, s/</&lt;/g for $$buf;
      print $beg, $line, $end, "\n" if length $$buf;
   }

Quote:
} } # end for loop, end while loop.

if( waitpid( $pid, 0 ) ) {{ # double braces so last will work.
   my ($sig, $ret) = ( $? & 0xFF, $? >> 8 );
   print("<b>prog died from signal $sig<\b>"), last if $sig;
   print("<b>prog exited with code $ret<\b>"), last if $ret;
Quote:
}} else {

   print("<b>Error in waitpid: $!</b>");
Quote:
}

print "</pre></html>\n";
__END__

NB: Windows select only works with sockets... but if you're not on
windows, this should all work ok, barring typoes and thinkoes :)

--
print reverse( ",rekcah", " lreP", " rehtona", " tsuJ" )."\n";



Tue, 28 Sep 2004 10:13:17 GMT  
 Capturing STDOUT and STDERR and preserving order?

Quote:

> I've been dabbling a bit with scripts designed to run a script on a local
> machine and report the results in a web page. IPC::Open3 allows me to
> capture STDOUT and STDERR (and thus display STDERR differently, say in bold
> or italics), but I haven't found a way yet of reliably reading from the two
> filehandles in the correct order. Any ideas?

Use select (or IO::Select) to find out which of STDOUT, STDERR have
data ready for reading.  It still won't come out "in the correct
order".  Even without buffering (on STDOUT, if not disabled with $|=1)
to{*filter*}things up, I believe there is no way of guaranteeing the same
sequencing.  select doesn't help you too much here; if your program
spends too much time capturing STDERR (say) after one call to select,
after the next call it may find both STDOUT and STDERR with data for
reading, and be unable to determine which one came first.

--
Ariel Scolnicov        | http://www.*-*-*.com/ ~ariels

72 Pinhas Rosen St.    |Tel: +972-3-7658117      "fast, good, and cheap;
Tel-Aviv 69512, ISRAEL |Fax: +972-3-7658555       pick any two!"



Tue, 28 Sep 2004 16:37:30 GMT  
 Capturing STDOUT and STDERR and preserving order?

[sample code]

Thanks a lot for that - I hadn't even considered IO::Select. Serves me
right.

A few comments. First of all:

Quote:
>   my $n = sysread( $r, $$buf, 8192, length $$buf );
>   unless( defined($n) ) {
>     print "<b>Error in sysread: $!</b></pre></html>";
>      exit;
>  } elsif( $n ) {
>     while( $$buf =~ s/(.*)\n// ) {
>         my $line = $1;
>         s/&/&amp;/g, s/</&lt;/g for $line;
>         print $beg, $line, $end, "\n";
>     }

The problem here is the the buffer keeps on getting filled up, so you'd have
to say
 my $newstuff= substr($buf, -$n);
 # print $newstuff
otherwise you get the old output printed out over and over.

Secondly, the program being called has to call
 STDOUT->autoflush(1);
 STDERR->autoflush(1);
or you have to wait for one of the streams to be finished before you can
read the other one.

Last of all, if output comes too fast you get it in the wrong order
*anyway*. The occasional judicious sleep statement may be in order (a simple
test program I ran worked fine if I added use Time::HiRes and the occasional
Time::HiRes::sleep(0.01))

Thanks again for all the help.



Tue, 28 Sep 2004 22:30:14 GMT  
 Capturing STDOUT and STDERR and preserving order?

Quote:

> I've been dabbling a bit with scripts designed to run a script on a local
> machine and report the results in a web page. IPC::Open3 allows me to
> capture STDOUT and STDERR (and thus display STDERR differently, say in bold
> or italics), but I haven't found a way yet of reliably reading from the two
> filehandles in the correct order. Any ideas?

Sorry, but at least on UNIX that can't be done. At least not reliably. There
are a couple of reasons for this. First, if the process producing the output
is using the stdio library the stdout stream will normally by line buffered
when open on a tty while stderr will be unbufferred. This means that a
sequence like this

    fprintf( stdout, "some text" );
    fprintf( stderr, "start of error message" );
    fprintf( stdout, ", more text\n" );
    fprintf( stderr, ", end of error text\n" );

will result in two calls to write() for the stderr stream but only one call to
write() for stdout.

Even if that isn't an issue there is no guarantee the process reading the data
will run immediately after each write to stdout and stderr. Thus it is
possible for the data producer to write some text to stderr, then write some
text to stdout, then a context switch to occur. When your reading process next
runs it will find data on both streams and there isn't any way for it to
determine the temp{*filter*}ordering of the data without the cooperation of the
data producing process (e.g., by timestamping the data).



Tue, 28 Sep 2004 12:50:54 GMT  
 Capturing STDOUT and STDERR and preserving order?

Quote:


> [sample code]

> Thanks a lot for that - I hadn't even considered IO::Select. Serves me
> right.

> A few comments. First of all:
> >   my $n = sysread( $r, $$buf, 8192, length $$buf );
> >   unless( defined($n) ) {
> >     print "<b>Error in sysread: $!</b></pre></html>";
> >      exit;
> >  } elsif( $n ) {
> >     while( $$buf =~ s/(.*)\n// ) {
> >         my $line = $1;
> >         s/&/&amp;/g, s/</&lt;/g for $line;
> >         print $beg, $line, $end, "\n";
> >     }

> The problem here is the the buffer keeps on getting filled up,

Not unless \n never appears.  Or unless you made an error when copying
and pasting, and accidentally transformed s/(.*)\n// into m/(.*)\n/.

Quote:
> so you'd have to say
>  my $newstuff= substr($buf, -$n);

That's what the s/// does.  Each time you call s///, it removes one
line, and stores it in $1.

Quote:
>  # print $newstuff
> otherwise you get the old output printed out over and over.

> Secondly, the program being called has to call
>  STDOUT->autoflush(1);
>  STDERR->autoflush(1);
> or you have to wait for one of the streams to be finished before you
> can read the other one.

No.  That's a misdiagnosis.  With IO::Select, you can read any of the
streams *as soon as it has data*.  You don't have to wait for either
stream to finish before you can read from it or the other.

However, if the program you're reading from buffers its streams, then
*that data isn't actually output anything* until an explicit flush
occurs, which only occurs when the buffer fills up, or the stream is
closed.

Having the child process turn on autoflush for stdout and stderr
essentially disables that buffering.

Quote:
> Last of all, if output comes too fast you get it in the wrong order
> *anyway*.

Define the right order :)
The problem is, io takes time.  While you're reading from one handle,
the process could be writing to it, or to the other handle.

Think of the child process as a friend sitting on top of the stairs,
dropping slinkies down the left or right side of the staircase.  You can
grab all the ones that land on the left, then the ones that land on your
right.  If your friend is dropping them infrequently enough, then you
can be sure that you grab them in the same order they were dropped.  But
if he drops them really fast, then inevitably some will pile up, and you
will get some out of order.

Quote:
> The occasional judicious sleep statement may be in order (a
> simple test program I ran worked fine if I added use Time::HiRes and
> the occasional Time::HiRes::sleep(0.01))

That will work.  Although you don't need Time::HiRes to do short
delays... you could do select('','','',0.01);

--
print reverse( ",rekcah", " lreP", " rehtona", " tsuJ" )."\n";



Sat, 02 Oct 2004 12:38:24 GMT  
 
 [ 6 post ] 

 Relevant Pages 

1. MSaccess DB security

2. jrfpas04.zip J.R. Ferguson's Pascal Library

3. How to redirect STDOUT,STDERR and keep the message order

4. capturing stdout and stderr?

5. Capturing both STDOUT and STDERR from `ls` ?

6. Probs w/ redirection and capture of STDERR/STDOUT

7. SRC: capturing stdout and stderr differently

8. Capturing STDOUT/STDERR

9. graph-mode in Turbo-Pascal 7.0

10. How to find unknown structure of databasefile??????

11. P7 autoinc field ok?

12. preserve order in hash for multiple levels

 

 
Powered by phpBB® Forum Software