Need fast method to save data to disk or memory 
Author Message
 Need fast method to save data to disk or memory

The task I have is to read data from an I/O - mapped static ram chip
(128k x 16) and save it to either the hard drive or memory.
The static-ram acts like a cache which can buffer data for up to
1.25 seconds, at which point it gets over-written and thus data gets
lost.

So, in simple terms, I'd like to know if I should either attack this
problem by using machine language to access DOS low-level file functions,
or low-level disk functions (INT 13), or to save this data to extended
memory.   The average data-rate is about 200k per second.

I have zero experience with protected-mode, extended memory machine-
language programming, so if that is needed for quick-and-dirty extended
memory access, then I'm gonna ask a lot of questions.

Anyone know how to do this?
E-mail or post, thanks, Wayne.



Tue, 25 Mar 1997 09:47:05 GMT  
 Need fast method to save data to disk or memory
WS> The task I have is to read data from an I/O - mapped static ram chip
WS> (128k x 16) and save it to either the hard drive or memory.
WS> The static-ram acts like a cache which can buffer data for up to
WS> 1.25 seconds, at which point it gets over-written and thus data gets

WS> So, in simple terms, I'd like to know if I should either attack this
WS> problem by using machine language to access DOS low-level file functions,
WS> or low-level disk functions (INT 13), or to save this data to extended
WS> memory.   The average data-rate is about 200k per second.

1st, if amount of data to be stored in single run exceeds your RAM size,
the data must be stored to disk. It must be hard disk, because floppies
have data rate up to 500kb/s, while you need 200kB/s=1600kb/s.
Standard (may get faster one) hard disk data transfer rate is 5Mb/s (bit
rate during sector read/write), this makes 500kB/s at interleave 1:1.
This means you may use interleave 1:1 or 1:2 (gives 250kB/s).
Better use 1:1, the 1:2 has too small safety margin.

2nd, data transfer rate using DOS calls isn't constant: usually it is
similar to disk speed, and sometimes DOS needs reach some other place
on disk and then uses some more time. Source of real troble are disk
access faults - they may need second or two for recovery, and during
this time you would lose data if not using intermediate buffer.
The intermediate buffer should be capable to keep amount of data
transferred during at least two seconds - i.e. it should be 400kB.
If you cannot use such amount of low RAM, the buffer must be in
extended (or expanded) RAM. Since you write "read data from an I/O -
mapped static ram chip" I understand the data can be accessed by
some REP INSW. This means if you use extended memory you must either
get the data in Protected Mode directly to the required address, or
get the data to low RAM and use Block Memory Move call to put the
data there; next you must move the data again before you can write
it to disk. Expanded memory allows you to map some memory at Real
Mode address, put data there, and map again when need to write it
to disk. Thus, expanded memory needs no extra time for data moving
and should be preferred. There are programs emulating expanded
memory (it may be hardware), like QEMM 386. There are also programs
caching disk I/O (in expanded memory, too), like SMARTDRV.
Using expanded memory for keeping data don't need much more time
(adds time for mapping changes only) than using low RAM and gives
possibility to have long intermediate buffer.
I suppose you can collect data and store it in expanded memory in
some interrupt service - this way you will not lose data even if
disk write sometimes lasts many seconds.
I have file with description of LIM (Lotus-Intel-Microsoft)
Extended Memory Specification, including example programs
in C, Turbo Pascal and assembly. It is 450kB (66kB packed).

3rd, newest DOS (6.) uses disk compression, if your data can be
compressed and you have fast computer, it gives better disk write
speed, otherwise compression causes waste of time and must disable
it. Just try with and w/o compression and compare time used for
disk writing. If data is random, compression cannot help.



Sat, 29 Mar 1997 22:04:49 GMT  
 Need fast method to save data to disk or memory
WS> The task I have is to read data from an I/O - mapped static ram chip
WS> (128k x 16) and save it to either the hard drive or memory.
WS> The static-ram acts like a cache which can buffer data for up to
WS> 1.25 seconds, at which point it gets over-written and thus data gets

WS> So, in simple terms, I'd like to know if I should either attack this
WS> problem by using machine language to access DOS low-level file functions,
WS> or low-level disk functions (INT 13), or to save this data to extended
WS> memory.   The average data-rate is about 200k per second.

1st, if amount of data to be stored in single run exceeds your RAM size,
the data must be stored to disk. It must be hard disk, because floppies
have data rate up to 500kb/s, while you need 200kB/s=1600kb/s.
Standard (may get faster one) hard disk data transfer rate is 5Mb/s (bit
rate during sector read/write), this makes 500kB/s at interleave 1:1.
This means you may use interleave 1:1 or 1:2 (gives 250kB/s).
Better use 1:1, the 1:2 has too small safety margin.

2nd, data transfer rate using DOS calls isn't constant: usually it is
similar to disk speed, and sometimes DOS needs reach some other place
on disk and then uses some more time. Source of real troble are disk
access faults - they may need second or two for recovery, and during
this time you would lose data if not using intermediate buffer.
The intermediate buffer should be capable to keep amount of data
transferred during at least two seconds - i.e. it should be 400kB.
If you cannot use such amount of low RAM, the buffer must be in
extended (or expanded) RAM. Since you write "read data from an I/O -
mapped static ram chip" I understand the data can be accessed by
some REP INSW. This means if you use extended memory you must either
get the data in Protected Mode directly to the required address, or
get the data to low RAM and use Block Memory Move call to put the
data there; next you must move the data again before you can write
it to disk. Expanded memory allows you to map some memory at Real
Mode address, put data there, and map again when need to write it
to disk. Thus, expanded memory needs no extra time for data moving
and should be preferred. There are programs emulating expanded
memory (it may be hardware), like QEMM 386. There are also programs
caching disk I/O (in expanded memory, too), like SMARTDRV.
Using expanded memory for keeping data don't need much more time
(adds time for mapping changes only) than using low RAM and gives
possibility to have long intermediate buffer.
I suppose you can collect data and store it in expanded memory in
some interrupt service - this way you will not lose data even if
disk write sometimes lasts many seconds.
I have file with description of LIM (Lotus-Intel-Microsoft)
Extended Memory Specification, including example programs
in C, Turbo Pascal and assembly. It is 450kB (66kB packed).

3rd, newest DOS (6.) uses disk compression, if your data can be
compressed and you have fast computer, it gives better disk write
speed, otherwise compression causes waste of time and must disable
it. Just try with and w/o compression and compare time used for
disk writing. If data is random, compression cannot help.



Sat, 29 Mar 1997 21:47:41 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. how do i save an image in memory to disk

2. How to save binary image data in BLOB back to disk

3. saving serial data strings to disk continuously

4. How to save experimental data into hard-disk.

5. Disk usage - saving and retrieving data ?

6. Save Data to Memory about ActiveX in LabView

7. Large data array (operations) via disk disk files

8. virtual method call faster that non-virtual method?

9. Large memory and fast interrupts: need advice from x86 gurus

10. Q: Acquiring data using two E series devices connected by RTSI and streaming data to disk

11. How can i save a pict as JPEG on disk, for use with web

12. Saving Reports to Disk - tfn.clw [01/01]

 

 
Powered by phpBB® Forum Software