CMUCL: concurrent programming questions 
Author Message
 CMUCL: concurrent programming questions

I'd like to start a lightweight concurrent process, and then be able
to kill it if it takes more than its allotted time or memory. Is this
possible? How? I want to do this programmatically (not with "/bin/kill
-9"). Thanks.


Wed, 09 Nov 2005 15:39:38 GMT  
 CMUCL: concurrent programming questions

Quote:
>>>>> On 24 May 2003 00:39:38 -0700, Michael Park ("Michael") writes:

 Michael> I'd like to start a lightweight concurrent process, and then be able
 Michael> to kill it if it takes more than its allotted time or memory. Is this
 Michael> possible? How? I want to do this programmatically (not with "/bin/kill
 Michael> -9"). Thanks.

I don't know the specific answer to how this is done in CMUCL.

If you're using the multiprocessing facilities available from
the Lisp scheduler (which may, or more likley may not correspond
to native threads or some kind of OS processes), then you can control
and monitor the process) in terms of starting, stopping, etc.  
You could time out such a process by using a timer, or another
such "process", or having specified a timeout when you started.
Usually such "processes" are really part of the single Lisp process,
so asking how much memory it's using is nonsensical.  

If you're using a facility provided in Lisp to access OS process,
then there should be an interface that will do what you want.
To monitor the memory usage or time limit of such a process,
you might use an "in-Lisp" "process" as above, to make "foreign
function calls" (directly, or through a Lisp interface library).

For example, I don't see any problem with implementing a fancy
program that combines the features of Unix "top" and "kill"
in Lisp.  The details of how you do the process control would
be approximately the same as in any other programming language.

Multiprocessing is not a standard part of Common Lisp,
although most of them have it, so the details will vary.

I am sure someone will come along and answer your CMUCL question;
hope this was useful in general.



Wed, 09 Nov 2005 21:26:56 GMT  
 CMUCL: concurrent programming questions

Quote:

>  I'd like to start a lightweight concurrent process, and then be able
>  to kill it if it takes more than its allotted time or memory. Is this
>  possible? How? I want to do this programmatically (not with "/bin/kill
>  -9"). Thanks.

It seems that CMUCL only supports multiprocessing on x86 platforms.
By default it is non-preemptive, which means that each thread must
yield the control back to the scheduler every once in a while.

Multi-processing functions are part of the MP package.
Have a look at the source code in file multi-proc.lisp
(I don't think there is documentation anywhere else).
There is also an example of use in function start-lisp-connection-listener
(same file)

In summary, you start a threan using MAKE-PROCESS.
The thread must pause from time to time, using function such as
PROCESS-YIELD or PROCESS-WAIT-UNTIL-FD-USABLE or PROCESS-WAIT-WITH-TIMEOUT
(there is a couple more)

You can monitor a process' run time using PROCESS-REAL-TIME and
PROCESS-RUN-TIME. You can kill a process using DESTROY-PROCESS.
Finally there is a function called WITH-TIMEOUT which may do
what you want, but....

I haven't used these facilities very much so there may things to be carful
about, of which I am not aware. Do a Google search for "with-timeout" is
this newsgroup, I seem to remember that with-timeout causes problem when
the thread needs to use unwind-protect.

--
Eric Daniel

-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----==  Over 80,000 Newsgroups - 16 Different Servers! =-----



Thu, 10 Nov 2005 05:24:03 GMT  
 CMUCL: concurrent programming questions

Quote:

> It seems that CMUCL only supports multiprocessing on x86 platforms.
> By default it is non-preemptive, which means that each thread must
> yield the control back to the scheduler every once in a while.

What do you mean with "by default". Does that mean there is an option
to run CMUCL in a preemtive mode?

--
Gisle S?lensminde  
Computational biology unit, University of Bergen, Norway

Biology easily has 500 years of exciting problems to work on. (Donald Knuth)



Fri, 11 Nov 2005 15:27:10 GMT  
 CMUCL: concurrent programming questions

  mp> I'd like to start a lightweight concurrent process, and then be
  mp> able to kill it if it takes more than its allotted time or
  mp> memory. Is this possible? How? I want to do this
  mp> programmatically (not with "/bin/kill -9").

you can use MP:PROCESS-RUN-TIME to know how much time a lisp-level
process has been running, and MP:DESTROY-PROCESS to terminate it.
There isn't any easy way to determine how much heap space a thread is
responsible for, however.

If you really want to enforce quota-like controls over lisp programs
that weren't written with this in mind and that you don't trust, you
will be better off using a CMUCL instance per activity, and using the
operating system's introspective abilities to determine CPU time and
memory use and enfore scheduling fairness. At startup, around half of
CMUCL's memory image (the "read-only space") can be shared between
multiple instances.

It's also worth noting that Dan Barlow's work on native threads for
SBCL on Linux/x86 will lead to much better multiprocessing support
than the user-space implementation in CMUCL. You can participate in
that work by beta-testing or by sending him a CD/book/postcard.

--
Eric Marsden                          <URL:http://www.laas.fr/~emarsden/>



Fri, 11 Nov 2005 19:33:35 GMT  
 CMUCL: concurrent programming questions
Hello!


Quote:
>[...]
>It's also worth noting that Dan Barlow's work on native threads for
>SBCL on Linux/x86 will lead to much better multiprocessing support
>than the user-space implementation in CMUCL. You can participate in
>that work by beta-testing or by sending him a CD/book/postcard.

How light/heavyweight are those thread implementations in comparison?

I think that 1:1 mapping of high level threads to OS threads is
quite expensive, and I'd really like something like Erlang wrt
concurrency (and perhaps another few goodies), and like (Common)
Lisp wrt all the rest.

Kind regards,

Hannah.



Fri, 11 Nov 2005 20:46:51 GMT  
 CMUCL: concurrent programming questions

Quote:
> Hello!


>>[...]

>>It's also worth noting that Dan Barlow's work on native threads for
>>SBCL on Linux/x86 will lead to much better multiprocessing support
>>than the user-space implementation in CMUCL. You can participate in
>>that work by beta-testing or by sending him a CD/book/postcard.

> How light/heavyweight are those thread implementations in comparison?

> I think that 1:1 mapping of high level threads to OS threads is
> quite expensive, and I'd really like something like Erlang wrt
> concurrency (and perhaps another few goodies), and like (Common)
> Lisp wrt all the rest.

There lies the standard dilemma with threads, in a whole pile of
places; do you want:
 a) To use the "OS threading" system, or
 b) To use a "platform-non-specific" threading system, or
 c) To use a "managed-within-local-process" threading system.

They provide three different sets of trade-offs.  

If you play the "requires-OS-support" game, you can get some
multiprocessing concurrency gains, which, on an SMP-like system, may
be valuable.  Option c) is unlikely to provide _any_ benefit of that
sort, but has the separate merit of allowing "really cheap threads" of
the Erlang sort, although I suspect it might be better to term those
"coroutines."
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://www.ntlug.org/~cbbrowne/lisp.html
Out of my mind. Back in five minutes.



Sat, 12 Nov 2005 00:52:31 GMT  
 CMUCL: concurrent programming questions

Quote:


> > It seems that CMUCL only supports multiprocessing on x86 platforms.
> > By default it is non-preemptive, which means that each thread must
> > yield the control back to the scheduler every once in a while.

>  What do you mean with "by default". Does that mean there is an option
>  to run CMUCL in a preemtive mode?

Hmm, actually this isn't excatly the way I thought.

There is this function:

;;; Start-Sigalrm-Yield  --  Internal
;;;
;;; Start a regular interrupt to switch processes. This may not be a
;;; good idea yet as the CMUCL code is not too interrupt safe.
;;;

It causes a SIGALRM to be sent regularly to the process.
When the signal is caught the signal handle simply calls
PROCESS-YIELD.

As the comments say it may break CMUCL. I don't know if the
author knew for a fact that it would break, or didn't know
what would happen.

--
Eric Daniel

-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----==  Over 80,000 Newsgroups - 16 Different Servers! =-----



Sat, 12 Nov 2005 05:35:25 GMT  
 CMUCL: concurrent programming questions

Quote:

>[Eric Marsden]
>>It's also worth noting that Dan Barlow's work on native threads for
>>SBCL on Linux/x86 will lead to much better multiprocessing support
> How light/heavyweight are those thread implementations in comparison?

Nobody's benchmarked, as far as I know.  My motivation for a 1:1 model
is

 * much simpler to deal with foreign code that may block (stream IO
   in SBCL is already kind of baroque and has strange bugs; why make
   it harder?)

 * take advantage of SMP machines

 * and the killer: there are many more Linux kernel hackers being paid
   to improve the efficiency of kernel threads than there are of me.
   Even though userland threads potentially can be scheduled more
   efficiently, the number of performance wizards in each court
   strongly suggests that OS-level scheduling is the way to go in
   practice

Quote:
> I think that 1:1 mapping of high level threads to OS threads is
> quite expensive, and I'd really like something like Erlang wrt

Thread creation is expensive in that each thread requires a binding
stack, a control stack, etc etc.  Thread switch costs about as much as
a process context switch, less the tlb flushes because they all have
the same memory maps.  The current implementation is limited to 8192
threads because that's all we can get in an LDT (the new NPTL stuff in
linux 2.6 can fix this, but we don't take advantage of it yet).  

Quote:
> concurrency (and perhaps another few goodies), and like (Common)
> Lisp wrt all the rest.

I don't think the CMUCL model is what you're looking for here either,
that said.  I've not used Erlang, but my impression is that Erlang
programmers use threads for pretty much anything they feel like, and I
doubt that CMUCL's threads (which wind/unwind the binding stack on
each switch) are quite up to this either.  But someone who knows
Erlang (Luke Gorrie, if he reads cll) would be more qualified to
comment here.

-dan

--

   http://www.cliki.net/ - Link farm for free CL-on-Unix resources



Sat, 12 Nov 2005 08:08:59 GMT  
 CMUCL: concurrent programming questions
Hello!


Quote:
>[...]
>There lies the standard dilemma with threads, in a whole pile of
>places; do you want:
> a) To use the "OS threading" system, or
> b) To use a "platform-non-specific" threading system, or
> c) To use a "managed-within-local-process" threading system.
>They provide three different sets of trade-offs.  
>If you play the "requires-OS-support" game, you can get some
>multiprocessing concurrency gains, which, on an SMP-like system, may
>be valuable.  Option c) is unlikely to provide _any_ benefit of that
>sort, but has the separate merit of allowing "really cheap threads" of
>the Erlang sort, although I suspect it might be better to term those
>"coroutines."

I'd prefer a combination of a + c, passed upwards with a (mostly)
portable interface.

With a m:n threading scheme, you could use a number n of OS threads
(e.g. the number of CPUs, or even a few more, so you schedule
potentially-blocking FFI calls on separate OS threads so the rest
of the program can continue to run), with a varying number m of
threads as seen from the provided interface (i.e. programmer's view).
Thus, you can reduce the memory overhead of threads quite a bit
without losing all the advantages of OS threads.

By the way, I disagree with you trying to call Erlang's "processes"
(as they call them) coroutines. As far as I know the latter, they
rely on programmers' exact annotations of control transfer, i.e. they
are non-preemptive, often the control transfer statements even name
the coroutine to run next. In Erlang, processes are preempted and
you don't have to code any transfer of control. So they are best described
as - threads - they *can* access some shared state like ets or in-memory
mnesia tables. Just that the commonly used programming patterns prefer
message passing over direct sharing of state.

Kind regards,

Hannah.



Tue, 29 Nov 2005 01:21:02 GMT  
 
 [ 10 post ] 

 Relevant Pages 

1. Concurrent Object-Oriented Programming and Eiffel

2. User Interfaces, Concurrent Functional Programming

3. Support for concurrent programming in Lisp

4. Language for Concurrent Programming - Not Java

5. An animated concurrent programming language for kids

6. Concurrent programming

7. books on concurrent programming paradigm

8. Concurrent Programming Languages

9. Integrating concurrent & O-O programming

10. seeking information: concurrent programming

11. request for concurrent ada programs

12. Integrating concurrent & O-O programming

 

 
Powered by phpBB® Forum Software