spawning unix command processes which run in parallel and simultaneously 
Author Message
 spawning unix command processes which run in parallel and simultaneously

HI, I am learning to code with python and I encounter a technical problem.

I want to spawn a few unix command processes from a python script so that I
can get the output from the unix command processes and use them in python.  

But I don't want to wait for the completion of one process at a time and
then start another.  I want the python script to start the processes
simultaneously and monitor whether anyone of them have finished and if so,
get the output from the finished one.

It seems to me that the os and popen2 module is useful in this respect but
I am not sure how to do it.

Is it possible to do it in python?  would be grateful if anyone can give me
a reference to any documentation.

Thanks



Thu, 27 Nov 2003 18:23:38 GMT  
 spawning unix command processes which run in parallel and simultaneously


Quote:
>HI, I am learning to code with Python and I encounter a technical
>problem.

Hi and welcome.

Quote:
>I want to spawn a few unix command processes from a python script so
>that I can get the output from the unix command processes and use
>them in python.
>But I don't want to wait for the completion of one process at a time
>and then start another.  I want the python script to start the
>processes simultaneously and monitor whether anyone of them have
>finished and if so, get the output from the finished one.
>It seems to me that the os and popen2 module is useful in this
>respect but I am not sure how to do it.

You can do what you want using popen (maybe popen2) and select. I'll
append a simple example. Unfortunately, with pipes and subprocesses,
once you go beyond a deliberately simple example, things can get
messy. Unless you and -- much harder to control -- your subprocesses
get buffering just right, programs that want to be interactive can
block waiting for input that you think you've sent and waiting for you
to read output that you haven't seen.

Happily, other folks have written code to deal with this nuisance.

The Vaults of Parnassus (www.vex.net/parnassus/) is a very valuable
resource. If you have a look in the Tools/Extensions section there
you'll find several modules inspired by Tcl's Expect (the canonical
program-that-controls-programs). I don't know which of them is best
but there aren't so many that it would be hard to try them out. Tim
O'Malley's event loop that's there looks like it might be of use to
you too.

Regards,
Matt

import random
import os
import select
import string

def getCommand():
  return "sleep %i;echo %s" % \
    (random.randint(0,10),random.choice(string.hexdigits))

def main():
  readPipes=[]
  for count in range(5):
    readPipes.append(os.popen(getCommand()))
  while 1:
    # Could put a timeout here if we had something else to do
    readable,writable,errors=select.select(readPipes,[],[])
    for p in readable:
      print p.read()
      readPipes.remove(p)
      os.wait() # Don't want zombies
    if len(readPipes)==0:
      break

if __name__=="__main__":
  main()



Fri, 28 Nov 2003 02:41:14 GMT  
 spawning unix command processes which run in parallel and simultaneously
Yes, os.popen and friends are what you want.

I recommend trying stuff in the interpreter; it's more fun that reading
documentation, and usually more instructive, and more accurate.

The documentation seems to have the wrong argument order for popen2, popen3,
and popen4, where the optional bufsize and more arguments are swapped
(should read "popen2 (cmd[, mode [, bufsize]])").

The documentation is in the os module under "6.1.2 File Object Creation"
presumably because it creates file like objects.  I would have expected it
to be in "6.1.5 Process Management", or at least a cross-reference there.

By the way, it seems to act strangely in PythonWin:

Quote:
>>> f =os.popen(r'C:\junk\test.exe')

Traceback (innermost last):
  File "<interactive input>", line 1, in ?
WindowsError: [Errno 2] The system cannot find the file specified

The same happens for any command line string, but it works fine in the
console version of python.

Quote:
----- Original Message -----

Newsgroups: comp.lang.python

Sent: Sunday, June 10, 2001 3:23 AM
Subject: spawning unix command processes which run in parallel and
simultaneously

> HI, I am learning to code with Python and I encounter a technical problem.

> I want to spawn a few unix command processes from a python script so that
I
> can get the output from the unix command processes and use them in python.

> But I don't want to wait for the completion of one process at a time and
> then start another.  I want the python script to start the processes
> simultaneously and monitor whether anyone of them have finished and if so,
> get the output from the finished one.

> It seems to me that the os and popen2 module is useful in this respect but
> I am not sure how to do it.

> Is it possible to do it in python?  would be grateful if anyone can give
me
> a reference to any documentation.

> Thanks

> --
> http://mail.python.org/mailman/listinfo/python-list



Thu, 27 Nov 2003 19:47:54 GMT  
 spawning unix command processes which run in parallel and simultaneously
Hoi,

Windows Win32 (NT version only) uses named pipe whose file names must look
like //./pipe/YOUR_FILE_NAME_HERE in order to be recognized correct (note
that I did not try it)

--
---
Kind regards, best wishes,
francis meyvis

Quote:
> Yes, os.popen and friends are what you want.

> I recommend trying stuff in the interpreter; it's more fun that reading
> documentation, and usually more instructive, and more accurate.

> The documentation seems to have the wrong argument order for popen2,
popen3,
> and popen4, where the optional bufsize and more arguments are swapped
> (should read "popen2 (cmd[, mode [, bufsize]])").

> The documentation is in the os module under "6.1.2 File Object Creation"
> presumably because it creates file like objects.  I would have expected it
> to be in "6.1.5 Process Management", or at least a cross-reference there.

> By the way, it seems to act strangely in PythonWin:

> >>> f =os.popen(r'C:\junk\test.exe')
> Traceback (innermost last):
>   File "<interactive input>", line 1, in ?
> WindowsError: [Errno 2] The system cannot find the file specified

> The same happens for any command line string, but it works fine in the
> console version of python.

> ----- Original Message -----

> Newsgroups: comp.lang.python

> Sent: Sunday, June 10, 2001 3:23 AM
> Subject: spawning unix command processes which run in parallel and
> simultaneously

> > HI, I am learning to code with Python and I encounter a technical
problem.

> > I want to spawn a few unix command processes from a python script so
that
> I
> > can get the output from the unix command processes and use them in
python.

> > But I don't want to wait for the completion of one process at a time and
> > then start another.  I want the python script to start the processes
> > simultaneously and monitor whether anyone of them have finished and if
so,
> > get the output from the finished one.

> > It seems to me that the os and popen2 module is useful in this respect
but
> > I am not sure how to do it.

> > Is it possible to do it in python?  would be grateful if anyone can give
> me
> > a reference to any documentation.

> > Thanks

> > --
> > http://mail.python.org/mailman/listinfo/python-list



Fri, 28 Nov 2003 21:07:00 GMT  
 spawning unix command processes which run in parallel and simultaneously

Quote:
Bufu writes:

 > The documentation seems to have the wrong argument order for popen2, popen3,
 > and popen4, where the optional bufsize and more arguments are swapped
 > (should read "popen2 (cmd[, mode [, bufsize]])").
 >
 > The documentation is in the os module under "6.1.2 File Object Creation"
 > presumably because it creates file like objects.  I would have expected it
 > to be in "6.1.5 Process Management", or at least a cross-reference there.

  Thanks for pointing this out; I've made the appropriate corrections
and additions to the Library Reference.  The updates will be part of
the manuals for Python 2.1.1 and 2.2.
  And please don't hesitate to CC: additional suggestions to be me at

SourceForge:

    http://sourceforge.net/tracker/?atid=105470&group_id=5470

  -Fred

--
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Digital Creations



Fri, 28 Nov 2003 23:24:53 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. defunct processes on UNIX server with spawn telnet

2. RUN and wait for spawned process

3. Spawn process running in the background

4. Is it possible to spawn the unix command screen

5. What tcl command to spawn process

6. expect: Matching patterns from multiple spawns simultaneously

7. Running commands in parallel...

8. How to invoke VI function in LabView running on Windows from a unix process in Solaris

9. processing unix command output lists

10. Running a Unix command from Ada

11. running unix commands from insdie expect script

12. running actual unix commands

 

 
Powered by phpBB® Forum Software