> David, do J and K use Windows memory by some "flexible"
> concept? In APL, you
> have to tell the interpreter in advance how much memory you
> want to allocate
> for the session, for example 100 MB. APL will then reserve
> 100 MB regardless
> of how much it in fact uses internally.

> The reason for asking is that i figured you would need twice
> that amount if
> you pass an array between J and K, i.e. *if* both J and K use
> memory on the
> same, pre-allocation basis, that amount would have to exist
> twice within the
> operating system. And i could imagine that the same happens
> just because of
> the large arrays themselves; if you have a 100 MB array in J
> and pass it on
> to K, won't there be a need of totally 200 MB of simultaneous
> storage? Then
> we have the same problem as when using APL alone; there is
> the original
> array and the temporary copy of it.

> ...unless you pass the array by looping and reduce it's size
> in the giving
> end while increasing it's size in the receiving end? Or use
> shared data
> between different OS threads, which is a *rather* tricky
> thing.

> In addition, i could imagine that the OS itself may need a
> relatively big
> memory allocation, if you pass the data through OLE.

> In fact i suspect that APLWin consumes memory *very* stupidly
> in this
> particular case, creating three instances of the array, so in
> that sense you
> are right.
> / Tomas

> > > It is indeed trivial to roll in and roll out. But - as
> far as I can
> see - it
> > > does not solve the original problem which was posed to
> the newsgroup by
> Ric
> > > Sherlock.

> > The original statement from the original Post was:

> > > Now I'm trying to sort large arrays (>100,000 rows) and
> am running into
> > > WSFULL problems.

> > And it seemed to me that if I was having trouble with that,
> all other
> > things being equal, I'd simply roll-out, sort, and roll-in,
> and I'd expect
> > it to cost me only a few seconds for the scale mentioned.

> > > Maybe I'm a bit slow today. Perhaps your point is
> actually "If you
> wanted to
> > > go there, you wouldn't be starting from here"?

> > > :-)

> > Perhaps it's me that's slow. But at least it isn't my
> computer languages!

> (J, on the other hand, appears to grab increasing
> amounts of virtual memory as it needs to and is
> not so good about releasing it: sometimes, when
> working with large arrays, I find I must save my
> work and restart J in order to continue.)

> Perhaps someone versed in Dyalog internals could
> better explain this but I'm basing my theory on
> observed behavior of the system.  To whit, I have
> some background processes written in Dyalog that
> start at system startup and stay up while I'm
> logged in.  Even though I run with 65M workspaces,
> the (Windows NT) Task Manager shows the dyalogrt.exe
> for these processes consuming only 5 or 6 M.

> However, once, during testing a new version of this
> process, I started about 15 copies without realizing
> it (because my usual shutdown was failing to kill
> the process) and the 16th or so copy refused to start,
> citing lack of memory.  There was sufficient memory
> at this point to run a 6M process but not a 65M one.

> So, my guess is that the Dyalog workspace allocation
> represents more of a programmer limit on the
> application's "high-water" mark - maximum memory
> use expected - rather than an initial allocation.

> I lack the experience with APL2000 to guess how
> it works in this regard.

> On another issue Tomas raises, that of multiple
> processes requiring multiple copies of a large
> array, I am currently working with memory-sharing
> between J and Dyalog.  I've gotten it to work but
> am still ironing out the details of interprocess
> communication and array conversion.

> I intend to post this once I've got a good version,
> hopefully within a month or so.

> Devon McCormick

> Of course the natural next question would be: If APL only allocates 6 MB
> even though one said 65 MB, what happens if APL then needs the 65 MB but
> there is no longer such an amount left in Windows... Then the 65 is no
> longer guaranteed.

> I would very much like to hear once you got memory sharing between OS
> processes to work. That would really power up a combined APL/J or APL/APL
> environment in some extreme cases or in a multi-processor NT server
> solution. I remember having heard someone using TCP/IP through loopback to
> achieve such an exchange, but that would not provide the access to shared
> data.

> I even found the post i sent a year ago when arguing that shared memory
> would actually be an option to consider seriously. Extract below:
> ---
> I am trying to say that there may (and must!) be ways for different
> processes in the OS to access a shared area of memory, in order to avoid
> streaming the data out and then back in to another process in the very
same
> computer through winsock loopback.

> Memory mapped files may be a good alternative, as you say. And indeed
there
> ARE methods for sharing memory in NT too. Here's an excerpt from an
article
> about page tables at
> http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

> "For Windows NT, a better implementation than the scheme described above
was
> chosen for sharing memory. Rather than having multiple PTEs point to the
> same physical page, another layer of page tables was put in place
> exclusively for shared memory. When two or more processes share a page of
> memory, an additional structure called a prototype page-table entry is
used
> to reference the shared page. Each process's PTE points to the prototype
> PTE, which, in turn, points to the actual shared page."
> ---
> / Tomas

> > I've noticed that in Dyalog APL, even though you specify
> > a workspace size, the interpreter apparently only uses
> > this number to check that there is this much memory
> > available before starting the session - it does not
> > necessarily allocate and use this much.

> > (J, on the other hand, appears to grab increasing
> > amounts of virtual memory as it needs to and is
> > not so good about releasing it: sometimes, when
> > working with large arrays, I find I must save my
> > work and restart J in order to continue.)

> > Perhaps someone versed in Dyalog internals could
> > better explain this but I'm basing my theory on
> > observed behavior of the system.  To whit, I have
> > some background processes written in Dyalog that
> > start at system startup and stay up while I'm
> > logged in.  Even though I run with 65M workspaces,
> > the (Windows NT) Task Manager shows the dyalogrt.exe
> > for these processes consuming only 5 or 6 M.

> > However, once, during testing a new version of this
> > process, I started about 15 copies without realizing
> > it (because my usual shutdown was failing to kill
> > the process) and the 16th or so copy refused to start,
> > citing lack of memory.  There was sufficient memory
> > at this point to run a 6M process but not a 65M one.

> > So, my guess is that the Dyalog workspace allocation
> > represents more of a programmer limit on the
> > application's "high-water" mark - maximum memory
> > use expected - rather than an initial allocation.

> > I lack the experience with APL2000 to guess how
> > it works in this regard.

> > On another issue Tomas raises, that of multiple
> > processes requiring multiple copies of a large
> > array, I am currently working with memory-sharing
> > between J and Dyalog.  I've gotten it to work but
> > am still ironing out the details of interprocess
> > communication and array conversion.

> > I intend to post this once I've got a good version,
> > hopefully within a month or so.

> > Devon McCormick

> In Windows, memory is shared via 'file mapping'. What this means is that a
> disk file is 'mapped' to a portion of the virtual address space of each
> process sharing the data. The disk file may be all or part of a physical
> disk file, including the swap file. Each process sharing the 'file'
(shared
> memory) may map the file to a different part of it's virtual space. Note
> that while a disk file is maintained the actual sharing is in RAM. A
> coherent copy of the space is maintained on disk via the normal file
> operations. It is (I think) necessary to have a DLL which 'owns' the
shared
> space, a persistent process may be created to own the space if no
processes
> are actively sharing the space, otherwise the shared space goes away with
> the demise of the last active process sharing the space(as does the DLL).
> Shared memory is accessed with the usual pointers for accessing private
> memory. There are system calls to create the shared space and to map it
into
> the virtual space of each participating process. The actual mechanism
> differs between NT and 98 but mainly in limitations on the 98 end(may it
> rest in peace). Synchronization is in the hands of the programmer.

> In the APL context the difficulty is accessing the internal data
structures
> used by APL, although Dyalog provides some tools to aid in this process.
See
> the manual chapter on writing your own AP's for a little more information.
> Shared variables and OLE are instances that could provide alternatives to
> writing the code yourself(in C). I am not familiar with other vendors
> capabilities.

> > [snip all but most recent post]
> > That does sound nice, but maybe Win98 handles the entire thing
> differently,
> > as i am unable to get the same results. Win98 allows you to open a
number
> of
> > system resource tracking windows and to put an update poll interval of
for
> > example one second. In fact, those windows do reflect exactly the
expected
> > situation when starting Dyalog APL; the amount of physically allocated
> > memory increases exactly with the WS size. But i could very well
imagine,
> > even expect, that NT might give APL a better starting point for memory
> > allocation.

> > Of course the natural next question would be: If APL only allocates 6 MB
> > even though one said 65 MB, what happens if APL then needs the 65 MB but
> > there is no longer such an amount left in Windows... Then the 65 is no
> > longer guaranteed.

> > I would very much like to hear once you got memory sharing between OS
> > processes to work. That would really power up a combined APL/J or
APL/APL
> > environment in some extreme cases or in a multi-processor NT server
> > solution. I remember having heard someone using TCP/IP through loopback
to
> > achieve such an exchange, but that would not provide the access to
shared
> > data.

> > I even found the post i sent a year ago when arguing that shared memory
> > would actually be an option to consider seriously. Extract below:
> > ---
> > I am trying to say that there may (and must!) be ways for different
> > processes in the OS to access a shared area of memory, in order to avoid
> > streaming the data out and then back in to another process in the very
> same
> > computer through winsock loopback.

> > Memory mapped files may be a good alternative, as you say. And indeed
> there
> > ARE methods for sharing memory in NT too. Here's an excerpt from an
> article
> > about page tables at
> > http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

> > "For Windows NT, a better implementation than the scheme described above
> was
> > chosen for sharing memory. Rather than having multiple PTEs point to the
> > same physical page, another layer of page tables was put in place
> > exclusively for shared memory. When two or more processes share a page
of
> > memory, an additional structure called a prototype page-table entry is
> used
> > to reference the shared page. Each process's PTE points to the prototype
> > PTE, which, in turn, points to the actual shared page."
> > ---
> > / Tomas

> > > I've noticed that in Dyalog APL, even though you specify
> > > a workspace size, the interpreter apparently only uses
> > > this number to check that there is this much memory
> > > available before starting the session - it does not
> > > necessarily allocate and use this much.

> > > (J, on the other hand, appears to grab increasing
> > > amounts of virtual memory as it needs to and is
> > > not so good about releasing it: sometimes, when
> > > working with large arrays, I find I must save my
> > > work and restart J in order to continue.)

> > > Perhaps someone versed in Dyalog internals could
> > > better explain this but I'm basing my theory on
> > > observed behavior of the system.  To whit, I have
> > > some background processes written in Dyalog that
> > > start at system startup and stay up while I'm
> > > logged in.  Even though I run with 65M workspaces,
> > > the (Windows NT) Task Manager shows the dyalogrt.exe
> > > for these processes consuming only 5 or 6 M.

> > > However, once, during testing a new version of this
> > > process, I started about 15 copies without realizing
> > > it (because my usual shutdown was failing to kill
> > > the process) and the 16th or so copy refused to start,
> > > citing lack of memory.  There was sufficient memory
> > > at this point to run a 6M process but not a 65M one.

> > > So, my guess is that the Dyalog workspace allocation
> > > represents more of a programmer limit on the
> > > application's "high-water" mark - maximum memory
> > > use expected - rather than an initial allocation.

> > > I lack the experience with APL2000 to guess how
> > > it works in this regard.

> > > On another issue Tomas raises, that of multiple
> > > processes requiring multiple copies of a large
> > > array, I am currently working with memory-sharing
> > > between J and Dyalog.  I've gotten it to work but
> > > am still ironing out the details of interprocess
> > > communication and array conversion.

> > > I intend to post this once I've got a good version,
> > > hopefully within a month or so.

> > > Devon McCormick

> Sorry, a translation error from my Swedish Windows :). Allocated physical
> memory is what i meant.

> But i'm still not convinced that Dyalog APL increases allocated memory as
it
> needs it internally. I did a simple test of starting APL with a clear
wssize
> of 100 MB. The Windows system monitor indicated the change. Then i
assigned
> 100000000{rank}'a' into a variable - and there was no indication of that
in
> the monitor (in fact nothing happened at all)... which would make it easy
to
> believe that APL just grabs all of that memory and does whatever it wants
> with it - even keeps it empty and unused. Although i would like the
proposed
> behaviour of dynamic memory allocation much more :).

> But the mapping you describe below - would it allow me to have say four
APL
> processes running on separate processors in NT, all working on own TCP/IP
> sockets, then just e.g. doing something as simple as performing reads and
> writes on a common huge array? Without any additional disk activity?
> / Tomas

> , but I have no trouble in
> > starting 20 or so copies of Dyalog with 16Mb workspaces on a 128Mb
> machine.
> > Running Windows ME or 98SE. System Monitor measures Allocated Memory,
> which
> > includes all of the disk you are willing to use as swap file. Which is
> where
> > the 65Mb
> > will come from in the end if there is no other space(supposing your
> machine
> > is small). Indeed, Dyalog will start with a 160Mb workspace on the same
> > 128Mb
> > machine. All we are really talking about here is table space for the
> paging.
> > I think that paging in APL is particularly disastrous as the processing
of
> > arrays usually means one use of the page; but of course I keep reading
> that
> > arrays are small :).

> > In Windows, memory is shared via 'file mapping'. What this means is that
a
> > disk file is 'mapped' to a portion of the virtual address space of each
> > process sharing the data. The disk file may be all or part of a physical
> > disk file, including the swap file. Each process sharing the 'file'
> (shared
> > memory) may map the file to a different part of it's virtual space. Note
> > that while a disk file is maintained the actual sharing is in RAM. A
> > coherent copy of the space is maintained on disk via the normal file
> > operations. It is (I think) necessary to have a DLL which 'owns' the
> shared
> > space, a persistent process may be created to own the space if no
> processes
> > are actively sharing the space, otherwise the shared space goes away
with
> > the demise of the last active process sharing the space(as does the
DLL).
> > Shared memory is accessed with the usual pointers for accessing private
> > memory. There are system calls to create the shared space and to map it
> into
> > the virtual space of each participating process. The actual mechanism
> > differs between NT and 98 but mainly in limitations on the 98 end(may it
> > rest in peace). Synchronization is in the hands of the programmer.

> > In the APL context the difficulty is accessing the internal data
> structures
> > used by APL, although Dyalog provides some tools to aid in this process.
> See
> > the manual chapter on writing your own AP's for a little more
information.
> > Shared variables and OLE are instances that could provide alternatives
to
> > writing the code yourself(in C). I am not familiar with other vendors
> > capabilities.

> > > [snip all but most recent post]
> > > That does sound nice, but maybe Win98 handles the entire thing
> > differently,
> > > as i am unable to get the same results. Win98 allows you to open a
> number
> > of
> > > system resource tracking windows and to put an update poll interval of
> for
> > > example one second. In fact, those windows do reflect exactly the
> expected
> > > situation when starting Dyalog APL; the amount of physically allocated
> > > memory increases exactly with the WS size. But i could very well
> imagine,
> > > even expect, that NT might give APL a better starting point for memory
> > > allocation.

> > > Of course the natural next question would be: If APL only allocates 6
MB
> > > even though one said 65 MB, what happens if APL then needs the 65 MB
but
> > > there is no longer such an amount left in Windows... Then the 65 is no
> > > longer guaranteed.

> > > I would very much like to hear once you got memory sharing between OS
> > > processes to work. That would really power up a combined APL/J or
> APL/APL
> > > environment in some extreme cases or in a multi-processor NT server
> > > solution. I remember having heard someone using TCP/IP through
loopback
> to
> > > achieve such an exchange, but that would not provide the access to
> shared
> > > data.

> > > I even found the post i sent a year ago when arguing that shared
memory
> > > would actually be an option to consider seriously. Extract below:
> > > ---
> > > I am trying to say that there may (and must!) be ways for different
> > > processes in the OS to access a shared area of memory, in order to
avoid
> > > streaming the data out and then back in to another process in the very
> > same
> > > computer through winsock loopback.

> > > Memory mapped files may be a good alternative, as you say. And indeed
> > there
> > > ARE methods for sharing memory in NT too. Here's an excerpt from an
> > article
> > > about page tables at
> > > http://msdn.microsoft.com/library/techart/msdn_ntvmm.htm:

> > > "For Windows NT, a better implementation than the scheme described
above
> > was
> > > chosen for sharing memory. Rather than having multiple PTEs point to
the
> > > same physical page, another layer of page tables was put in place
> > > exclusively for shared memory. When two or more processes share a page
> of
> > > memory, an additional structure called a prototype page-table entry is
> > used
> > > to reference the shared page. Each process's PTE points to the
prototype
> > > PTE, which, in turn, points to the actual shared page."
> > > ---
> > > / Tomas

> > > > I've noticed that in Dyalog APL, even though you specify
> > > > a workspace size, the interpreter apparently only uses
> > > > this number to check that there is this much memory
> > > > available before starting the session - it does not
> > > > necessarily allocate and use this much.

> > > > (J, on the other hand, appears to grab increasing
> > > > amounts of virtual memory as it needs to and is
> > > > not so good about releasing it: sometimes, when
> > > > working with large arrays, I find I must save my
> > > > work and restart J in order to continue.)

> > > > Perhaps someone versed in Dyalog internals could
> > > > better explain this but I'm basing my theory on
> > > > observed behavior of the system.  To whit, I have
> > > > some background processes written in Dyalog that
> > > > start at system startup and stay up while I'm
> > > > logged in.  Even though I run with 65M workspaces,
> > > > the (Windows NT) Task Manager shows the dyalogrt.exe
> > > > for these processes consuming only 5 or 6 M.

> > > > However, once, during testing a new version of this
> > > > process, I started about 15 copies without realizing
> > > > it (because my usual shutdown was failing to kill
> > > > the process) and the 16th or so copy refused to start,
> > > > citing lack of memory.  There