J/APL memory use [was Re: SV: Space efficient sorting of arrays] 
Author Message
 J/APL memory use [was Re: SV: Space efficient sorting of arrays]

I've noticed that in Dyalog APL, even though you specify
a workspace size, the interpreter apparently only uses
this number to check that there is this much memory
available before starting the session - it does not
necessarily allocate and use this much.

(J, on the other hand, appears to grab increasing
amounts of virtual memory as it needs to and is
not so good about releasing it: sometimes, when
working with large arrays, I find I must save my
work and restart J in order to continue.)

Perhaps someone versed in Dyalog internals could
better explain this but I'm basing my theory on
observed behavior of the system.  To whit, I have
some background processes written in Dyalog that
start at system startup and stay up while I'm
logged in.  Even though I run with 65M workspaces,
the (Windows NT) Task Manager shows the dyalogrt.exe
for these processes consuming only 5 or 6 M.

However, once, during testing a new version of this
process, I started about 15 copies without realizing
it (because my usual shutdown was failing to kill
the process) and the 16th or so copy refused to start,
citing lack of memory.  There was sufficient memory
at this point to run a 6M process but not a 65M one.

So, my guess is that the Dyalog workspace allocation
represents more of a programmer limit on the
application's "high-water" mark - maximum memory
use expected - rather than an initial allocation.

I lack the experience with APL2000 to guess how
it works in this regard.

On another issue Tomas raises, that of multiple
processes requiring multiple copies of a large
array, I am currently working with memory-sharing
between J and Dyalog.  I've gotten it to work but
am still ironing out the details of interprocess
communication and array conversion.

I intend to post this once I've got a good version,
hopefully within a month or so.

Devon McCormick


Quote:

> David, do J and K use Windows memory by some "flexible"
> concept? In APL, you
> have to tell the interpreter in advance how much memory you
> want to allocate
> for the session, for example 100 MB. APL will then reserve
> 100 MB regardless
> of how much it in fact uses internally.

> The reason for asking is that i figured you would need twice
> that amount if
> you pass an array between J and K, i.e. *if* both J and K use
> memory on the
> same, pre-allocation basis, that amount would have to exist
> twice within the
> operating system. And i could imagine that the same happens
> just because of
> the large arrays themselves; if you have a 100 MB array in J
> and pass it on
> to K, won't there be a need of totally 200 MB of simultaneous
> storage? Then
> we have the same problem as when using APL alone; there is
> the original
> array and the temporary copy of it.

> ...unless you pass the array by looping and reduce it's size
> in the giving
> end while increasing it's size in the receiving end? Or use
> shared data
> between different OS threads, which is a *rather* tricky
> thing.

> In addition, i could imagine that the OS itself may need a
> relatively big
> memory allocation, if you pass the data through OLE.

> In fact i suspect that APLWin consumes memory *very* stupidly
> in this
> particular case, creating three instances of the array, so in
> that sense you
> are right.
> / Tomas




> > > It is indeed trivial to roll in and roll out. But - as
> far as I can
> see - it
> > > does not solve the original problem which was posed to
> the newsgroup by
> Ric
> > > Sherlock.

> > The original statement from the original Post was:

> > > Now I'm trying to sort large arrays (>100,000 rows) and
> am running into
> > > WSFULL problems.

> > And it seemed to me that if I was having trouble with that,
> all other
> > things being equal, I'd simply roll-out, sort, and roll-in,
> and I'd expect
> > it to cost me only a few seconds for the scale mentioned.

> > > Maybe I'm a bit slow today. Perhaps your point is
> actually "If you
> wanted to
> > > go there, you wouldn't be starting from here"?

> > > :-)

> > Perhaps it's me that's slow. But at least it isn't my
> computer languages!

__________________________________________________
Do You Yahoo!?
Send your FREE holiday greetings online!
http://www.*-*-*.com/


Wed, 26 May 2004 06:16:51 GMT  
 J/APL memory use [was Re: SV: Space efficient sorting of arrays]
[snip all but most recent post]
That does sound nice, but maybe Win98 handles the entire thing differently,
as i am unable to get the same results. Win98 allows you to open a number of
system resource tracking windows and to put an update poll interval of for
example one second. In fact, those windows do reflect exactly the expected
situation when starting Dyalog APL; the amount of physically allocated
memory increases exactly with the WS size. But i could very well imagine,
even expect, that NT might give APL a better starting point for memory
allocation.

Of course the natural next question would be: If APL only allocates 6 MB
even though one said 65 MB, what happens if APL then needs the 65 MB but
there is no longer such an amount left in Windows... Then the 65 is no
longer guaranteed.

I would very much like to hear once you got memory sharing between OS
processes to work. That would really power up a combined APL/J or APL/APL
environment in some extreme cases or in a multi-processor NT server
solution. I remember having heard someone using TCP/IP through loopback to
achieve such an exchange, but that would not provide the access to shared
data.

I even found the post i sent a year ago when arguing that shared memory
would actually be an option to consider seriously. Extract below:
---
I am trying to say that there may (and must!) be ways for different
processes in the OS to access a shared area of memory, in order to avoid
streaming the data out and then back in to another process in the very same
computer through winsock loopback.

Memory mapped files may be a good alternative, as you say. And indeed there
ARE methods for sharing memory in NT too. Here's an excerpt from an article
about page tables at
http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

"For Windows NT, a better implementation than the scheme described above was
chosen for sharing memory. Rather than having multiple PTEs point to the
same physical page, another layer of page tables was put in place
exclusively for shared memory. When two or more processes share a page of
memory, an additional structure called a prototype page-table entry is used
to reference the shared page. Each process's PTE points to the prototype
PTE, which, in turn, points to the actual shared page."
---
/ Tomas



Quote:
> I've noticed that in Dyalog APL, even though you specify
> a workspace size, the interpreter apparently only uses
> this number to check that there is this much memory
> available before starting the session - it does not
> necessarily allocate and use this much.

> (J, on the other hand, appears to grab increasing
> amounts of virtual memory as it needs to and is
> not so good about releasing it: sometimes, when
> working with large arrays, I find I must save my
> work and restart J in order to continue.)

> Perhaps someone versed in Dyalog internals could
> better explain this but I'm basing my theory on
> observed behavior of the system.  To whit, I have
> some background processes written in Dyalog that
> start at system startup and stay up while I'm
> logged in.  Even though I run with 65M workspaces,
> the (Windows NT) Task Manager shows the dyalogrt.exe
> for these processes consuming only 5 or 6 M.

> However, once, during testing a new version of this
> process, I started about 15 copies without realizing
> it (because my usual shutdown was failing to kill
> the process) and the 16th or so copy refused to start,
> citing lack of memory.  There was sufficient memory
> at this point to run a 6M process but not a 65M one.

> So, my guess is that the Dyalog workspace allocation
> represents more of a programmer limit on the
> application's "high-water" mark - maximum memory
> use expected - rather than an initial allocation.

> I lack the experience with APL2000 to guess how
> it works in this regard.

> On another issue Tomas raises, that of multiple
> processes requiring multiple copies of a large
> array, I am currently working with memory-sharing
> between J and Dyalog.  I've gotten it to work but
> am still ironing out the details of interprocess
> communication and array conversion.

> I intend to post this once I've got a good version,
> hopefully within a month or so.

> Devon McCormick




Wed, 26 May 2004 08:08:27 GMT  
 J/APL memory use [was Re: SV: Space efficient sorting of arrays]
I am not sure what tool you are using to determine 'the amount of physically
allocated memory' as System Monitor does not do it, but I have no trouble in
starting 20 or so copies of Dyalog with 16Mb workspaces on a 128Mb machine.
Running Windows ME or 98SE. System Monitor measures Allocated Memory, which
includes all of the disk you are willing to use as swap file. Which is where
the 65Mb
will come from in the end if there is no other space(supposing your machine
is small). Indeed, Dyalog will start with a 160Mb workspace on the same
128Mb
machine. All we are really talking about here is table space for the paging.
I think that paging in APL is particularly disastrous as the processing of
arrays usually means one use of the page; but of course I keep reading that
arrays are small :).

In Windows, memory is shared via 'file mapping'. What this means is that a
disk file is 'mapped' to a portion of the virtual address space of each
process sharing the data. The disk file may be all or part of a physical
disk file, including the swap file. Each process sharing the 'file' (shared
memory) may map the file to a different part of it's virtual space. Note
that while a disk file is maintained the actual sharing is in RAM. A
coherent copy of the space is maintained on disk via the normal file
operations. It is (I think) necessary to have a DLL which 'owns' the shared
space, a persistent process may be created to own the space if no processes
are actively sharing the space, otherwise the shared space goes away with
the demise of the last active process sharing the space(as does the DLL).
Shared memory is accessed with the usual pointers for accessing private
memory. There are system calls to create the shared space and to map it into
the virtual space of each participating process. The actual mechanism
differs between NT and 98 but mainly in limitations on the 98 end(may it
rest in peace). Synchronization is in the hands of the programmer.

In the APL context the difficulty is accessing the internal data structures
used by APL, although Dyalog provides some tools to aid in this process. See
the manual chapter on writing your own AP's for a little more information.
Shared variables and OLE are instances that could provide alternatives to
writing the code yourself(in C). I am not familiar with other vendors
capabilities.


Quote:
> [snip all but most recent post]
> That does sound nice, but maybe Win98 handles the entire thing
differently,
> as i am unable to get the same results. Win98 allows you to open a number
of
> system resource tracking windows and to put an update poll interval of for
> example one second. In fact, those windows do reflect exactly the expected
> situation when starting Dyalog APL; the amount of physically allocated
> memory increases exactly with the WS size. But i could very well imagine,
> even expect, that NT might give APL a better starting point for memory
> allocation.

> Of course the natural next question would be: If APL only allocates 6 MB
> even though one said 65 MB, what happens if APL then needs the 65 MB but
> there is no longer such an amount left in Windows... Then the 65 is no
> longer guaranteed.

> I would very much like to hear once you got memory sharing between OS
> processes to work. That would really power up a combined APL/J or APL/APL
> environment in some extreme cases or in a multi-processor NT server
> solution. I remember having heard someone using TCP/IP through loopback to
> achieve such an exchange, but that would not provide the access to shared
> data.

> I even found the post i sent a year ago when arguing that shared memory
> would actually be an option to consider seriously. Extract below:
> ---
> I am trying to say that there may (and must!) be ways for different
> processes in the OS to access a shared area of memory, in order to avoid
> streaming the data out and then back in to another process in the very
same
> computer through winsock loopback.

> Memory mapped files may be a good alternative, as you say. And indeed
there
> ARE methods for sharing memory in NT too. Here's an excerpt from an
article
> about page tables at
> http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

> "For Windows NT, a better implementation than the scheme described above
was
> chosen for sharing memory. Rather than having multiple PTEs point to the
> same physical page, another layer of page tables was put in place
> exclusively for shared memory. When two or more processes share a page of
> memory, an additional structure called a prototype page-table entry is
used
> to reference the shared page. Each process's PTE points to the prototype
> PTE, which, in turn, points to the actual shared page."
> ---
> / Tomas



> > I've noticed that in Dyalog APL, even though you specify
> > a workspace size, the interpreter apparently only uses
> > this number to check that there is this much memory
> > available before starting the session - it does not
> > necessarily allocate and use this much.

> > (J, on the other hand, appears to grab increasing
> > amounts of virtual memory as it needs to and is
> > not so good about releasing it: sometimes, when
> > working with large arrays, I find I must save my
> > work and restart J in order to continue.)

> > Perhaps someone versed in Dyalog internals could
> > better explain this but I'm basing my theory on
> > observed behavior of the system.  To whit, I have
> > some background processes written in Dyalog that
> > start at system startup and stay up while I'm
> > logged in.  Even though I run with 65M workspaces,
> > the (Windows NT) Task Manager shows the dyalogrt.exe
> > for these processes consuming only 5 or 6 M.

> > However, once, during testing a new version of this
> > process, I started about 15 copies without realizing
> > it (because my usual shutdown was failing to kill
> > the process) and the 16th or so copy refused to start,
> > citing lack of memory.  There was sufficient memory
> > at this point to run a 6M process but not a 65M one.

> > So, my guess is that the Dyalog workspace allocation
> > represents more of a programmer limit on the
> > application's "high-water" mark - maximum memory
> > use expected - rather than an initial allocation.

> > I lack the experience with APL2000 to guess how
> > it works in this regard.

> > On another issue Tomas raises, that of multiple
> > processes requiring multiple copies of a large
> > array, I am currently working with memory-sharing
> > between J and Dyalog.  I've gotten it to work but
> > am still ironing out the details of interprocess
> > communication and array conversion.

> > I intend to post this once I've got a good version,
> > hopefully within a month or so.

> > Devon McCormick




Wed, 26 May 2004 13:48:03 GMT  
 J/APL memory use [was Re: SV: Space efficient sorting of arrays]


Quote:
> I am not sure what tool you are using to determine 'the amount of
physically
> allocated memory' as System Monitor does not do it

Sorry, a translation error from my Swedish Windows :). Allocated physical
memory is what i meant.

But i'm still not convinced that Dyalog APL increases allocated memory as it
needs it internally. I did a simple test of starting APL with a clear wssize
of 100 MB. The Windows system monitor indicated the change. Then i assigned
100000000{rank}'a' into a variable - and there was no indication of that in
the monitor (in fact nothing happened at all)... which would make it easy to
believe that APL just grabs all of that memory and does whatever it wants
with it - even keeps it empty and unused. Although i would like the proposed
behaviour of dynamic memory allocation much more :).

But the mapping you describe below - would it allow me to have say four APL
processes running on separate processors in NT, all working on own TCP/IP
sockets, then just e.g. doing something as simple as performing reads and
writes on a common huge array? Without any additional disk activity?
/ Tomas

, but I have no trouble in

Quote:
> starting 20 or so copies of Dyalog with 16Mb workspaces on a 128Mb
machine.
> Running Windows ME or 98SE. System Monitor measures Allocated Memory,
which
> includes all of the disk you are willing to use as swap file. Which is
where
> the 65Mb
> will come from in the end if there is no other space(supposing your
machine
> is small). Indeed, Dyalog will start with a 160Mb workspace on the same
> 128Mb
> machine. All we are really talking about here is table space for the
paging.
> I think that paging in APL is particularly disastrous as the processing of
> arrays usually means one use of the page; but of course I keep reading
that
> arrays are small :).

> In Windows, memory is shared via 'file mapping'. What this means is that a
> disk file is 'mapped' to a portion of the virtual address space of each
> process sharing the data. The disk file may be all or part of a physical
> disk file, including the swap file. Each process sharing the 'file'
(shared
> memory) may map the file to a different part of it's virtual space. Note
> that while a disk file is maintained the actual sharing is in RAM. A
> coherent copy of the space is maintained on disk via the normal file
> operations. It is (I think) necessary to have a DLL which 'owns' the
shared
> space, a persistent process may be created to own the space if no
processes
> are actively sharing the space, otherwise the shared space goes away with
> the demise of the last active process sharing the space(as does the DLL).
> Shared memory is accessed with the usual pointers for accessing private
> memory. There are system calls to create the shared space and to map it
into
> the virtual space of each participating process. The actual mechanism
> differs between NT and 98 but mainly in limitations on the 98 end(may it
> rest in peace). Synchronization is in the hands of the programmer.

> In the APL context the difficulty is accessing the internal data
structures
> used by APL, although Dyalog provides some tools to aid in this process.
See
> the manual chapter on writing your own AP's for a little more information.
> Shared variables and OLE are instances that could provide alternatives to
> writing the code yourself(in C). I am not familiar with other vendors
> capabilities.



> > [snip all but most recent post]
> > That does sound nice, but maybe Win98 handles the entire thing
> differently,
> > as i am unable to get the same results. Win98 allows you to open a
number
> of
> > system resource tracking windows and to put an update poll interval of
for
> > example one second. In fact, those windows do reflect exactly the
expected
> > situation when starting Dyalog APL; the amount of physically allocated
> > memory increases exactly with the WS size. But i could very well
imagine,
> > even expect, that NT might give APL a better starting point for memory
> > allocation.

> > Of course the natural next question would be: If APL only allocates 6 MB
> > even though one said 65 MB, what happens if APL then needs the 65 MB but
> > there is no longer such an amount left in Windows... Then the 65 is no
> > longer guaranteed.

> > I would very much like to hear once you got memory sharing between OS
> > processes to work. That would really power up a combined APL/J or
APL/APL
> > environment in some extreme cases or in a multi-processor NT server
> > solution. I remember having heard someone using TCP/IP through loopback
to
> > achieve such an exchange, but that would not provide the access to
shared
> > data.

> > I even found the post i sent a year ago when arguing that shared memory
> > would actually be an option to consider seriously. Extract below:
> > ---
> > I am trying to say that there may (and must!) be ways for different
> > processes in the OS to access a shared area of memory, in order to avoid
> > streaming the data out and then back in to another process in the very
> same
> > computer through winsock loopback.

> > Memory mapped files may be a good alternative, as you say. And indeed
> there
> > ARE methods for sharing memory in NT too. Here's an excerpt from an
> article
> > about page tables at
> > http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

> > "For Windows NT, a better implementation than the scheme described above
> was
> > chosen for sharing memory. Rather than having multiple PTEs point to the
> > same physical page, another layer of page tables was put in place
> > exclusively for shared memory. When two or more processes share a page
of
> > memory, an additional structure called a prototype page-table entry is
> used
> > to reference the shared page. Each process's PTE points to the prototype
> > PTE, which, in turn, points to the actual shared page."
> > ---
> > / Tomas



> > > I've noticed that in Dyalog APL, even though you specify
> > > a workspace size, the interpreter apparently only uses
> > > this number to check that there is this much memory
> > > available before starting the session - it does not
> > > necessarily allocate and use this much.

> > > (J, on the other hand, appears to grab increasing
> > > amounts of virtual memory as it needs to and is
> > > not so good about releasing it: sometimes, when
> > > working with large arrays, I find I must save my
> > > work and restart J in order to continue.)

> > > Perhaps someone versed in Dyalog internals could
> > > better explain this but I'm basing my theory on
> > > observed behavior of the system.  To whit, I have
> > > some background processes written in Dyalog that
> > > start at system startup and stay up while I'm
> > > logged in.  Even though I run with 65M workspaces,
> > > the (Windows NT) Task Manager shows the dyalogrt.exe
> > > for these processes consuming only 5 or 6 M.

> > > However, once, during testing a new version of this
> > > process, I started about 15 copies without realizing
> > > it (because my usual shutdown was failing to kill
> > > the process) and the 16th or so copy refused to start,
> > > citing lack of memory.  There was sufficient memory
> > > at this point to run a 6M process but not a 65M one.

> > > So, my guess is that the Dyalog workspace allocation
> > > represents more of a programmer limit on the
> > > application's "high-water" mark - maximum memory
> > > use expected - rather than an initial allocation.

> > > I lack the experience with APL2000 to guess how
> > > it works in this regard.

> > > On another issue Tomas raises, that of multiple
> > > processes requiring multiple copies of a large
> > > array, I am currently working with memory-sharing
> > > between J and Dyalog.  I've gotten it to work but
> > > am still ironing out the details of interprocess
> > > communication and array conversion.

> > > I intend to post this once I've got a good version,
> > > hopefully within a month or so.

> > > Devon McCormick




Wed, 26 May 2004 23:35:23 GMT  
 J/APL memory use [was Re: SV: Space efficient sorting of arrays]
The point here is that all memory is virtual to APL, so any amount can be
allocated by APL at any time, whether used or not(up to 2 or 4Gb, depending
upon the Windows version). The amount of physical memory used is dependent
upon how the operating system schedules tasks and the activity of the task.
By physical memory here I mean RAM. But the disk is also 'Physical' so that
it is likely that much or most of a large workspace containing large arrays
may actually be on disk in the swap file. I don't know whether Dyalog
allocates the entire workspace or not, assume it does; it is still only
virtual not actual RAM.

The only 'Physical Memory' available through System Monitor is "Remaining
Physical Memory" which is the memory (RAM) available that Windows has not
currently allocated for something else. This is usually less than 10 Mb due
to the fact that Windows allocates most of  memory not in active use by
programs to Disk cache and buffers of various kinds. All other memory
amounts in System Monitor refer to virtual memory, not physical RAM.

It is indeed possible to have a collection of APL or any other programs in
any combination sharing memory without it going through the disk in
practice, however, an updated disk copy will be maintained just as an
updated copy of any file or data array will be, depending upon the paging
requirements of the machine. There is no guarantee the actual data on disk
will reflect that in memory in this instance, any more than there is for any
other data array or disk file(until the file is closed). If the mapping is
done through the swap file there is no copy of the data in any event after
the sharing programs terminate.

All this is a gross simplification of the process, if you have the patience
you should consult the Memory section of the Microsoft System Development
Kit (SDK) Documentation available on CD-ROM from
http://developerstore.com/devstore/product.asp?productID=7647 .
 I am not sure what the cost in Sweden would be, but the Documentation is
free, shipping must be paid. At one time this could be downloaded but it is
very large.


Quote:


> > I am not sure what tool you are using to determine 'the amount of
> physically
> > allocated memory' as System Monitor does not do it

> Sorry, a translation error from my Swedish Windows :). Allocated physical
> memory is what i meant.

> But i'm still not convinced that Dyalog APL increases allocated memory as
it
> needs it internally. I did a simple test of starting APL with a clear
wssize
> of 100 MB. The Windows system monitor indicated the change. Then i
assigned
> 100000000{rank}'a' into a variable - and there was no indication of that
in
> the monitor (in fact nothing happened at all)... which would make it easy
to
> believe that APL just grabs all of that memory and does whatever it wants
> with it - even keeps it empty and unused. Although i would like the
proposed
> behaviour of dynamic memory allocation much more :).

> But the mapping you describe below - would it allow me to have say four
APL
> processes running on separate processors in NT, all working on own TCP/IP
> sockets, then just e.g. doing something as simple as performing reads and
> writes on a common huge array? Without any additional disk activity?
> / Tomas

> , but I have no trouble in
> > starting 20 or so copies of Dyalog with 16Mb workspaces on a 128Mb
> machine.
> > Running Windows ME or 98SE. System Monitor measures Allocated Memory,
> which
> > includes all of the disk you are willing to use as swap file. Which is
> where
> > the 65Mb
> > will come from in the end if there is no other space(supposing your
> machine
> > is small). Indeed, Dyalog will start with a 160Mb workspace on the same
> > 128Mb
> > machine. All we are really talking about here is table space for the
> paging.
> > I think that paging in APL is particularly disastrous as the processing
of
> > arrays usually means one use of the page; but of course I keep reading
> that
> > arrays are small :).

> > In Windows, memory is shared via 'file mapping'. What this means is that
a
> > disk file is 'mapped' to a portion of the virtual address space of each
> > process sharing the data. The disk file may be all or part of a physical
> > disk file, including the swap file. Each process sharing the 'file'
> (shared
> > memory) may map the file to a different part of it's virtual space. Note
> > that while a disk file is maintained the actual sharing is in RAM. A
> > coherent copy of the space is maintained on disk via the normal file
> > operations. It is (I think) necessary to have a DLL which 'owns' the
> shared
> > space, a persistent process may be created to own the space if no
> processes
> > are actively sharing the space, otherwise the shared space goes away
with
> > the demise of the last active process sharing the space(as does the
DLL).
> > Shared memory is accessed with the usual pointers for accessing private
> > memory. There are system calls to create the shared space and to map it
> into
> > the virtual space of each participating process. The actual mechanism
> > differs between NT and 98 but mainly in limitations on the 98 end(may it
> > rest in peace). Synchronization is in the hands of the programmer.

> > In the APL context the difficulty is accessing the internal data
> structures
> > used by APL, although Dyalog provides some tools to aid in this process.
> See
> > the manual chapter on writing your own AP's for a little more
information.
> > Shared variables and OLE are instances that could provide alternatives
to
> > writing the code yourself(in C). I am not familiar with other vendors
> > capabilities.



> > > [snip all but most recent post]
> > > That does sound nice, but maybe Win98 handles the entire thing
> > differently,
> > > as i am unable to get the same results. Win98 allows you to open a
> number
> > of
> > > system resource tracking windows and to put an update poll interval of
> for
> > > example one second. In fact, those windows do reflect exactly the
> expected
> > > situation when starting Dyalog APL; the amount of physically allocated
> > > memory increases exactly with the WS size. But i could very well
> imagine,
> > > even expect, that NT might give APL a better starting point for memory
> > > allocation.

> > > Of course the natural next question would be: If APL only allocates 6
MB
> > > even though one said 65 MB, what happens if APL then needs the 65 MB
but
> > > there is no longer such an amount left in Windows... Then the 65 is no
> > > longer guaranteed.

> > > I would very much like to hear once you got memory sharing between OS
> > > processes to work. That would really power up a combined APL/J or
> APL/APL
> > > environment in some extreme cases or in a multi-processor NT server
> > > solution. I remember having heard someone using TCP/IP through
loopback
> to
> > > achieve such an exchange, but that would not provide the access to
> shared
> > > data.

> > > I even found the post i sent a year ago when arguing that shared
memory
> > > would actually be an option to consider seriously. Extract below:
> > > ---
> > > I am trying to say that there may (and must!) be ways for different
> > > processes in the OS to access a shared area of memory, in order to
avoid
> > > streaming the data out and then back in to another process in the very
> > same
> > > computer through winsock loopback.

> > > Memory mapped files may be a good alternative, as you say. And indeed
> > there
> > > ARE methods for sharing memory in NT too. Here's an excerpt from an
> > article
> > > about page tables at
> > > http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

> > > "For Windows NT, a better implementation than the scheme described
above
> > was
> > > chosen for sharing memory. Rather than having multiple PTEs point to
the
> > > same physical page, another layer of page tables was put in place
> > > exclusively for shared memory. When two or more processes share a page
> of
> > > memory, an additional structure called a prototype page-table entry is
> > used
> > > to reference the shared page. Each process's PTE points to the
prototype
> > > PTE, which, in turn, points to the actual shared page."
> > > ---
> > > / Tomas



> > > > I've noticed that in Dyalog APL, even though you specify
> > > > a workspace size, the interpreter apparently only uses
> > > > this number to check that there is this much memory
> > > > available before starting the session - it does not
> > > > necessarily allocate and use this much.

> > > > (J, on the other hand, appears to grab increasing
> > > > amounts of virtual memory as it needs to and is
> > > > not so good about releasing it: sometimes, when
> > > > working with large arrays, I find I must save my
> > > > work and restart J in order to continue.)

> > > > Perhaps someone versed in Dyalog internals could
> > > > better explain this but I'm basing my theory on
> > > > observed behavior of the system.  To whit, I have
> > > > some background processes written in Dyalog that
> > > > start at system startup and stay up while I'm
> > > > logged in.  Even though I run with 65M workspaces,
> > > > the (Windows NT) Task Manager shows the dyalogrt.exe
> > > > for these processes consuming only 5 or 6 M.

> > > > However, once, during testing a new version of this
> > > > process, I started about 15 copies without realizing
> > > > it (because my usual shutdown was failing to kill
> > > > the process) and the 16th or so copy refused to start,
> > > > citing lack of memory.  There

...

read more »



Thu, 27 May 2004 00:56:36 GMT  
 J/APL memory use [was Re: SV: Space efficient sorting of arrays]


Quote:
> I've noticed that in Dyalog APL, even though you specify
> a workspace size, the interpreter apparently only uses
> this number to check that there is this much memory
> available before starting the session - it does not
> necessarily allocate and use this much.

I have talked with Geoff Streeter (he wrote the Dyalog memory manager) on
this matter in the past.

This is how I remember it (I may be wrong, my memory is poor<grin>, also I
cannot remember if this related to Dyalog APL/W under 16 and/or 32 bit
Windows. Sorry Geoff)

Dyalog uses/reserves sufficient memory to load the workspace (or issues a
"WS TOO LARGE" error).

Once the workspace is loaded, Dyalog will issue memory allocation calls each
time more memory is required, up to the point it reaches its MAXWS limit. It
will then run a garbage collection, releasing existing memory for re-use. If
the available memory is still insufficient, a WS FULL message is produced.
Once memory is allocated, it is NOT released until Dyalog terminates.

One effect of this is that if you at the start of your program, request your
max limit of memory, by creating a sufficiently large dummy variable. Dyalog
will request the required memory in a single call, and as a result, get a
contiguous block of memory. The dummy variable can then be deleted.

The result is, Dyalog only has to perform a single memory allocation, for
the whole of the session, rather than each time say a new variable is
created. And Dyalog runs faster as it is using continues memory rather than
fragmented chunks.

The one downside to this is, if you dont need all the memory you request,
that memory is unavailable to the rest of Windows, for the duration of your
Dyalog session.

My apologies in advance, if I have mis-led anyone here.



Fri, 28 May 2004 06:56:18 GMT  
 
 [ 6 post ] 

 Relevant Pages 

1. J/APL memory use [was Re: SV: Space efficient sorting of

2. Space efficient sorting of arrays

3. Space efficient sorting of arrays

4. re efficient sorting of arrays

5. WANTED: APL-II way to sort array so each column is in descending order

6. Sorting an array in memory

7. JS-EAI with *JS*-callback

8. js.exception 3279.js

9. NB. gray.js: a J verb that generates a grayscale postscript image from a 2d array

10. used space 0, free space 0

11. space-efficient top-N algorithm

12. Anybody know a fast efficient sorting Algorithm

 

 
Powered by phpBB® Forum Software