Invalid page faults when accessing large arrays 
Author Message
 Invalid page faults when accessing large arrays

I'm sure this is probably a FAQ, in fact I think I asked it once, a long
time ago, when I used to use C regularly... but I can't find it.

typdef struct {
   long warp[6];
   char added;

Quote:
} SECTOR;

I wrote a program which uses a malloc'ed array of these, normally about
5000 of them.  If I try to access an element of the array, however, I get
two invalid page faults, one in my program, and one in Kernel32.  How to
prevent this?

I'm using my trusty (crusty) BC++ 4.

Thanks for any pointers...

Krum



Sun, 23 Nov 2003 10:37:04 GMT  
 Invalid page faults when accessing large arrays

Quote:

> I'm sure this is probably a FAQ, in fact I think I asked it once, a long
> time ago, when I used to use C regularly... but I can't find it.

> typdef struct {
>    long warp[6];
>    char added;
> } SECTOR;

> I wrote a program which uses a malloc'ed array of these, normally about
> 5000 of them.  If I try to access an element of the array, however, I get
> two invalid page faults, one in my program, and one in Kernel32.  How to
> prevent this?

> I'm using my trusty (crusty) BC++ 4.

I just learned here today that the standard guaranties that (at least
one) object(s) of size up to 64k can be allocated. If on your
implementation sizeof(long) is 8 and longs need to be aligned on quad
words then each SECTOR will use 6*8+1+3(padding)=52. For 5000 of these
you need 260,000 which blows the 64k limit of the standard. I never used
BC++ but you say it's crusty, so maybe it's using the small memory model
of intel? It may have a switch to overcome that...
my $0.02 Tobias.


Sun, 23 Nov 2003 10:58:41 GMT  
 Invalid page faults when accessing large arrays

Quote:

>> I'm sure this is probably a FAQ, in fact I think I asked it
>> once, a long time ago, when I used to use C regularly... but I
>> can't find it.

>> typdef struct {
>>    long warp[6];
>>    char added;
>> } SECTOR;

>> I wrote a program which uses a malloc'ed array of these,
>> normally about 5000 of them.  If I try to access an element of
>> the array, however, I get two invalid page faults, one in my
>> program, and one in Kernel32.  How to prevent this?

>> I'm using my trusty (crusty) BC++ 4.

> I just learned here today that the standard guaranties that (at
> least one) object(s) of size up to 64k can be allocated. If on
> your implementation sizeof(long) is 8 and longs need to be
> aligned on quad words then each SECTOR will use
> 6*8+1+3(padding)=52. For 5000 of these you need 260,000 which
> blows the 64k limit of the standard. I never used BC++ but you
> say it's crusty, so maybe it's using the small memory model of
> intel? It may have a switch to overcome that... my $0.02
> Tobias.

Tobias, what yoiu learned was probably not what you should have
learned. As the given size constraints is a requirement for
an objects, not a malloced memory block, as the OP explained.

I also doubt that BC++ 4 has longs with 8 bytes width, and would
guess that they have 32Bits with CHAR_BITS set to 8 so the size
would probably be 4. Anyhow one should not rely on that.

Than again I think your assumption concerning the small memory
model might actually be what causes the OP's problem. But I
don't know BC++4 too. :)
--

"LISP  is worth learning for  the profound enlightenment  experience
you will have when you finally get it; that experience will make you
a better programmer for the rest of your days."   -- Eric S. Raymond



Sun, 23 Nov 2003 19:50:45 GMT  
 Invalid page faults when accessing large arrays

Quote:

> I'm sure this is probably a FAQ, in fact I think I asked it once, a long
> time ago, when I used to use C regularly... but I can't find it.

> typdef struct {
>    long warp[6];
>    char added;
> } SECTOR;

> I wrote a program which uses a malloc'ed array of these, normally about
> 5000 of them.  If I try to access an element of the array, however, I get
> two invalid page faults, one in my program, and one in Kernel32.  How to
> prevent this?

> I'm using my trusty (crusty) BC++ 4.

I don't know for sure because it has been a loooong time since I've used
the Borland compiler, but are you sure that it can handle large
allocations like this without having to resort to some sort of trickery
like huge pointers?  Are you using the right memory model, one that
allows large data?

You may want to upgrade to a less crusty compiler since BC++ 4.0 is from
the days when real mode was still around and compilers that wanted to
still support real mode had to play games to access more than 64K of
data.

-- Thomas Brown



Sun, 23 Nov 2003 21:55:25 GMT  
 Invalid page faults when accessing large arrays

Quote:

> I just learned here today that the standard guaranties that (at least
> one) object(s) of size up to 64k can be allocated.

Not necessarily with malloc(), though.  And the standard has some
weasel wording around that guarantee, anyhow.

The minimum is 32767 bytes for C89 and 65535 bytes for C99.



Sun, 23 Nov 2003 21:49:02 GMT  
 Invalid page faults when accessing large arrays
I'll assume you're on Win32, in which case there should be no problem w/ the
allocation.  I would guess you did this:

SECTOR *foo = (SECTOR*)malloc(5000);

You need

SECTOR *foo = (SECTOR*)malloc(5000*sizeof(SECTOR));
or just
SECTOR *foo = new SECTOR[5000]; on C++

Otherwise just post some code.

Andy


Quote:
> I'm sure this is probably a FAQ, in fact I think I asked it once, a long
> time ago, when I used to use C regularly... but I can't find it.

> typdef struct {
>    long warp[6];
>    char added;
> } SECTOR;

> I wrote a program which uses a malloc'ed array of these, normally about
> 5000 of them.  If I try to access an element of the array, however, I get
> two invalid page faults, one in my program, and one in Kernel32.  How to
> prevent this?

> I'm using my trusty (crusty) BC++ 4.

> Thanks for any pointers...

> Krum



Sun, 23 Nov 2003 22:12:52 GMT  
 Invalid page faults when accessing large arrays


Quote:
> I'll assume you're on Win32, in which case there should be no
> problem w/ the allocation.  I would guess you did this:

> SECTOR *foo = (SECTOR*)malloc(5000);

> You need

> SECTOR *foo = (SECTOR*)malloc(5000*sizeof(SECTOR));

You really should lose that cast. What he needs is probably
SECTOR *foo = malloc(500 * sizeof *foo);

Quote:
> or just
> SECTOR *foo = new SECTOR[5000]; on C++

That is off topic in CLC.

--

"LISP  is worth learning for  the profound enlightenment  experience
you will have when you finally get it; that experience will make you
a better programmer for the rest of your days."   -- Eric S. Raymond



Sun, 23 Nov 2003 23:05:45 GMT  
 Invalid page faults when accessing large arrays


Quote:
> You really should lose that cast. What he needs is probably
> SECTOR *foo = malloc(500 * sizeof *foo);

I had

SECTOR* sector = malloc(5000 * sizeof(SECTOR));

I got it working.  See my thanks post.



Mon, 24 Nov 2003 06:02:35 GMT  
 Invalid page faults when accessing large arrays


Wed, 18 Jun 1902 08:00:00 GMT  
 Invalid page faults when accessing large arrays


Quote:
> I wrote a program which uses a malloc'ed array of these, normally about
> 5000 of them.  If I try to access an element of the array, however, I get
> two invalid page faults, one in my program, and one in Kernel32.  How to
> prevent this?

Thanks for the tips.  I had to work around the 64K limit by dynamically
allocating an array of sectors, then allocating the small array within
each sector.

Here's the result.  This will be of interest to TW2002 players.  Since
the popular helpers SUCK when it comes to probe hunting (oh, for an
update to GuruIBM!), you can run a ZT map and then use this to analyze
the exported .TWX file.  It outputs a list you can use to assist your
initial exploration.

The next version is going to be TOP SECRET.  Sorry.

-- begin code --

// twxrip.c
// Extracts sector data from a TWX-format binary, sorts sectors into
// descending order according to distance from Stardock, and writes
// the list to stdout.  Tested with BC++ 4.  Should be almost directly
// portable -- just modify includes as required.

#include <stdio.h>
#include <malloc.h>
#include <winsock.h>   // for ntohl()

typedef struct {           // Size: 256 bytes
   char id[4];             // File ID: "TWEX"
   long time_created;      // Timestamp when created (See TS)
   long ver;               // File version (See VE)
   long sectors;           // Total number of sectors
   long stardock;          // StarDock location (-1=Unknown)
   long cls0port_sol;      // Class 0 port: Sol (-1=Unknown)
   long cls0port_alpha;    // Class 0 port: Alpha Centauri (-1=Unknown)
   long cls0port_rylos;    // Class 0 port: Rylos (-1=Unknown)
   long crc32;             // Checksum (See CH)
   char reserved[220];     // Reserved for future use (Set to 0)

Quote:
} TWX_FILE_HEADER;

typedef struct {           // Size: 96 bytes
   char info;              // Sector info (See SI)
   char navhaz;            // NavHaz percentage (0-100) (-1=Unknown)
   short reserved2;        // Reserved for future use (Set to 0)
   long sector_update;     // Timestamp from last sector update
                           // (0=Never updated) (See TS and USI)
   long ftrs;              // Fighters (-1=Unknown)
   short ftr_owner;        // Fighter owner (Reserved, set to -1)
   char ftr_type;          // Fighter type (1=Toll, 2=Offensive,
                           // 3=Defensive, 0=Rogue, -1=Unknown)
   char anom;              // Anomality (0=No, 1=Yes, -1=Unknown)
   short armids;           // Armid mines (-1=Unknown)
   short armid_owner;      // Armid owner (Reserved, set to -1)
   short limpets;          // Limpet mines (-1=Unknown)
   short limpet_owner;     // Limpet owner (Reserved, set to -1)
   long port_amt[3];       // Port amount [FOE] (-1=Unknown)
   char port_per[3];       // Port percentage [FOE] (0-100) (-1=Unknown)
   char warps;             // # of warp sectors (-1=Unknown)
   long warp_sect[6];      // Warp sectors
   long port_update;       // Timestamp from last port update
                           // (0=Never updated) (See TS and USI)
   long density;           // Sector density (-1=Unknown)
   char reserved[24];      // Reserved for future use (Set to 0)

Quote:
} TWX_SECTOR;

typedef struct {
   long* warp;
   char added;

Quote:
} SECTOR;

#define NULL_SECTOR -1
#define FALSE       0
#define TRUE        1

int main(int argc, char* argv[]) {
   char* filename;
   FILE* twxfile;
   TWX_FILE_HEADER twxHeader;
   TWX_SECTOR twxSector;
   long sectors;           // num sectors
   long stardock;          // location of stardock
   SECTOR* sector;         // extracted data
   long* stack;            // sort stack
   long workIdx;           // stack work index
   long topIdx;            // stack top index
   long s, w;              // loop counters

   if(argc > 1) filename=argv[1];
   else {
      fprintf(stderr, "Usage: twxrip <filename>\n");
      return 1;
   }

   twxfile = fopen(filename, "rb");
   if(!twxfile) {
      fprintf(stderr, "Could not open %s\n", filename);
      return 1;
   }

   fread(&twxHeader, sizeof(TWX_FILE_HEADER), 1, twxfile);
   sectors = ntohl(twxHeader.sectors);
   stardock = ntohl(twxHeader.stardock);
   if(stardock == -1) {
      fprintf(stderr, "Stardock location unknown!\n");
      fclose(twxfile);
      return 1;
   }
   // convert to zero-based numbering
   --stardock;

   sector = malloc(sectors * sizeof(SECTOR));
   stack = malloc(sectors * sizeof(long));

   if(sector && stack) {
      for(s = 0; s < sectors; ++s) {
         sector[s].warp = malloc(6 * sizeof(long));
         if(!sector[s].warp) {
            // kick it in reverse...
            for(--s; s >= 0; --s) free(sector[s].warp);
            free(sector);
            free(stack);
            fclose(twxfile);
            fprintf(stderr, "Out of memory");
            return 1;
         }
      }
   } else {
      fprintf(stderr, "Out of memory");
      return 1;
   }

   for(s = 0; s < sectors; ++s) {
      fread(&twxSector, sizeof(TWX_SECTOR), 1, twxfile);
      for(w = 0; w < 6; ++w)
      // convert to zero-based numbering
      sector[s].warp[w] = ntohl(twxSector.warp_sect[w]) - 1;
   }

   // initialize
   for(s = 0; s < sectors; ++s) sector[s].added = FALSE;
   workIdx = 0;
     topIdx = 0;
   stack[topIdx] = stardock;
   sector[stardock].added = TRUE;

   // kick it
   while(workIdx <= topIdx) {
      // for each warp of sector at workIdx...
      #define WORKSECT sector[stack[workIdx]]
      for(w = 0; w < 6; ++w) {
         // if warp exists...
         if(WORKSECT.warp[w] != NULL_SECTOR) {
            // and has not already been added...
            if(! sector[WORKSECT.warp[w]].added) {
               // add it to the stack...
               ++topIdx;
               stack[topIdx] = WORKSECT.warp[w];
               sector[WORKSECT.warp[w]].added = TRUE;
            }
         }
      }
      ++workIdx;
   }

   // completed sorting, output results
   fprintf(stdout, "Unreachable sectors:\n");
   for(s = 0; s < sectors; ++s)
      // convert back to one-based numbering
      if(!sector[s].added) fprintf(stdout, "%li\n", s + 1);
   fprintf(stdout, "\nSectors by distance (greatest to least):\n");
   for( ; topIdx >= 0; --topIdx)
      // convert back to one-based numbering
      fprintf(stdout, "%li\n", stack[topIdx] + 1);

   for(s = 0; s < sectors; ++s) free(sector[s].warp);
   free(sector);
   free(stack);
   fclose(twxfile);
   return 0;

Quote:
}

// end twxrip.c


Mon, 24 Nov 2003 06:17:36 GMT  
 
 [ 10 post ] 

 Relevant Pages 

1. Invalid page fault in Windows98

2. Invalid Page Fault?

3. caused an invalid page fault in module ...

4. Invalid Page fault in module MSJT3032.DLL

5. Invalid page fault

6. Invalid page fault

7. invalid page fault

8. Invalid Page Fault in Release mode with ML compiler option

9. invalid page fault ...

10. MSDEV caused an invalid page fault in Module SSSCC.DLL After Install SP2

11. invalid page fault when I exit my program

12. Help - receiving invalid page fault in kernel32.dll

 

 
Powered by phpBB® Forum Software