DOS disk space allocation for indexed files 
Author Message
 DOS disk space allocation for indexed files

In the mini/midrange world we had lots of useful utilities to help manage
the disk space used by data files, especially those highly active transaction
and history files that grow and shrink on a routine basis.  Now I'm in a client
server environment and I'm a little confused.

How DOS does it remains a mystery to me.  Assume I have a history file with
100,000 records and it occupies X bytes.

I purge a set of records from the file, say... 15,000 records, and the file
still
occupies X bytes.  So I run Micro Focus' REBUILD utility, and guess what,
the file still occupies X bytes, less a few in the .IDX where the keys have
been removed.

As an experiement I wrote a quick program to write out a new file using the old
file as input.  Now the new file occupies much less space.

Is this the only solution to recovering disk space when a file has shrunk?

I have a transaction file that may contain up to 3,500 records before the end
of the day, then gets purged, and DOS reports this sucker eats up almost 4Mb
of disk space.  I need to get this resolved, what are the rest of you in the
PeeCee
world doing?

Thanks,
JC Jones.



Sun, 11 Mar 2001 03:00:00 GMT  
 DOS disk space allocation for indexed files

Quote:

>occupies X bytes.  So I run Micro Focus' REBUILD utility, and guess what,
>the file still occupies X bytes, less a few in the .IDX where the keys have
>been removed.

Rebuild has two modes -- "rebuild filename" is the first, and basically
does a completely non-destructive rebuild. That is what you are doing.
That type of rebuild is done to re-org and balance the index tree for
speed, as well as a recovery method for power failures and glitches.
You should set up a batch file to do that on all Isam files, and put it
aside
as an emergency recovery tool.  For ninety-five percent of all user calls
re weird things happening, telling the user to submit that batch file and
call back with the results will take care of it.

(BTW-I redirect the output to the printer in the batch file. When there
are problems still there, you tell the user to fax you the printer output,
and you will be able to take care of another 4 1/2 % of the problems
on the next call)

Now, that type of rebuild always leaves your data file unaffected, and so
can be done over and over again. If you have to patch around a bad block
or something, you can null out the bad data and recover what is left, etc.
Problem is, as you discovered, it does not remove empty space in the data
file.  For that, you run the rebuild like so: Rebuild infile, outfile. That
produces
a brand new output, all nicely cleaned up, but it is a brand new file.  The
cycle
can be ["backup", "rebuild backup to newfile", "start again"] or ["rename",
"rebuild oldfile to newfile", "startagain"].  Eithor way, you are in trouble
if
it fails and the user trys again. That is, run it twice, and you
self-destruct.

That is because the only safe sequence is really "do it", if it fails, you
have
to know exactly why to avoid disaster. You may want to do an extra backup
and put it aside while you think about it.  So I seldom give such procedures
to any but the very safest users. (twice in the last fif{*filter*} years, and the
one
guy destroyed himself.)

Over the years, I have found the safest way is as follows.  You set up
maintenance
such that purges are regular and often (daily or weekly).  The idea is to
keep
your data file allocated but empty after purges.  In other words, the data
file sort of hits a constant size that is comfortable. When you do a defrag,
you end up with a block of continginous data on the disk that is "MY SPACE".
Nice and efficient.  Then on a daily basis, you run that recovery batch
file.
That keeps your index's nice and tight.

Have the "compress data" batch file ready and tested for a rainy day,
but put it aside in an area for the pro running the place (or self). I
usually
run it only when I am on site fixing a problem, or am tidying up at year
ends.

Last, but not least, rebuild will tell you how to do the above by going
into DOS and typing "rebuild help" or "rebuild/?" or something like that.
I always have to do it when I put the batch files together.



Sun, 11 Mar 2001 03:00:00 GMT  
 DOS disk space allocation for indexed files

Thanks, Donald.  

The REBUILD INFILE, OUTFILE  is just like COPY WITH REORG on some other
platforms I've worked on.  You're right, it can cause major
problems in the hands of an inexperienced user.

BTW just typing REBUILD will list the parms.  I always like to see a
running count but can never remember whether it's /v or /e.

Thanks again,
JC Jones



Mon, 12 Mar 2001 03:00:00 GMT  
 DOS disk space allocation for indexed files
Dear JC,

j> How DOS does it remains a mystery to me.  Assume I have a history file
j> with 100,000 records and it occupies X bytes.
j> I purge a set of records from the file, say... 15,000 records, and the
j> file still occupies X bytes.  So I run Micro Focus' REBUILD utility,
j> and guess what, the file still occupies X bytes, less a few in the .IDX
j> where the keys have been removed.

This behavior is not (only) related to DOS file handling. If you delete a  
record from your file, the record is marked as deleted and the space it  
consumes is entered into the free space list (FSL) in the IDX file. If you  
add a record to your file, you will see that the file does not grow as  
long as there are enough deleted records in the file.

That REBUILD had no effect is simply because you did not use it correctly.  
If you use the syntax "REBUILD file" the index (and only the index) is  
rebuild. By doing this you even delete the FSL so that unused space in the  
data file cannot be reused.

Try reorganizing the file using the syntax "REBUILD infile,outfile".

Best regards

Sven

Send a mail with subject "SEND PGPKEY" to receive my public key



Mon, 12 Mar 2001 03:00:00 GMT  
 DOS disk space allocation for indexed files
Dear Donald,

DT> your data file allocated but empty after purges.  In other words, the
DT> data file sort of hits a constant size that is comfortable. When you
DT> do a defrag, you end up with a block of continginous data on the disk
DT> that is "MY SPACE". Nice and efficient.  Then on a daily basis, you
DT> run that recovery batch file. That keeps your index's nice and tight.

Well - nice thought but it does not do what you want to achieve. By  
running your recovery batch file (rebuild file) the index is rebuild and  
it's free space list is deleted so that "MY SPACE" is never reused and the  
file keeps on growing when adding records.

Best regards

Sven

Send a mail with subject "SEND PGPKEY" to receive my public key



Mon, 12 Mar 2001 03:00:00 GMT  
 DOS disk space allocation for indexed files

Quote:
>Try reorganizing the file using the syntax "REBUILD
>infile,outfile".

Yeppers, that works just fine.  Since I'm a W2 here I don't
have to worry about a ham-fisted user renaming and deleting
the wrong files.

I only have to worry about my own ham-fisted renaming and
deleting the wrong files :-)

Thanks to all,
JC Jones.



Tue, 13 Mar 2001 03:00:00 GMT  
 DOS disk space allocation for indexed files
On Wed, 23 Sep 1998 21:13:03 -0400, "Donald Tees"

Quote:

>For that, you run the rebuild like so: Rebuild infile, outfile. That
>produces
>a brand new output, all nicely cleaned up, but it is a brand new file.  The
>cycle
>can be ["backup", "rebuild backup to newfile", "start again"] or ["rename",
>"rebuild oldfile to newfile", "startagain"].  Eithor way, you are in trouble
>if
>it fails and the user trys again. That is, run it twice, and you
>self-destruct.

>That is because the only safe sequence is really "do it", if it fails, you
>have
>to know exactly why to avoid disaster.

speaking of which, i was writing a online database defrag spec... it's
on comp.databases from a few days ago.

something like a binary tree, BUT...

the data file has a pointer back to the binary tree's pointer,
and a size, and a deleted boolean variable.

this would theoretically allow you to defrag the entire file in the
background so long as each record defraged is atomic.

but i haven't thought of how to do an index defrag... yet




Wed, 14 Mar 2001 03:00:00 GMT  
 
 [ 7 post ] 

 Relevant Pages 

1. Space Allocation on P/390

2. Data space allocation

3. Finding files on a given disk on other accessed disk

4. IBM Mainframers: VSAM Space Allocation Software

5. OT: VSAM DASD Space Allocation

6. Array space allocation with pointers

7. CSP sol. for Space Allocation

8. Large data array (operations) via disk disk files

9. Resource id allocation space exhausted problem

10. Obtaining available disk space (APL*PLUS III)

11. How to retrieve (in assembler) the disk space allocated to a data set

12. Free disk space in Carbon/OS X apps?

 

 
Powered by phpBB® Forum Software