Help: Need to display a large text file 
Author Message
 Help: Need to display a large text file

Hi Everybody.
Need to display a huge text file in a MFC CRichEdit (preferrably)
or CEdit or some other MFC window. The file can be 500 MB. This
is larger than Virtual Memory. Please help by giving pointer to
source code that does this or by explaning how.

My best current idea is to have buttons that allow for scrolling
or override the scrollbar so that when I reach the end of the
currently displayed portion I will read the next portion of the
file and throw the least recently used portion from memory. I
will always keep 3 portions in memory. These portions will be
1000 lines long.

Another problem is allowing the user to jump directly to a line
number.If he wants to go to a line 5000 lines ahead, I have to
read all the lines till that line and then read it. Currently I
can't see how to get past this problem unless the text file is
written in fixed size lines.

Thanks for your help,
Guy Shaham



Thu, 25 Jul 2002 03:00:00 GMT  
 Help: Need to display a large text file
If the file is read-only, which I suppose it is, I would forego prewritten
controls altogether.

If you are just supposed to display the contents on-screen, you might as
well paint it yourself. You will probably not have any normal
scrollbar-handling with that size of a file, and if you can force a
fixed-line length format, you would really have it much easier. You could
have an indexfile with the line-starts, but 10 000 000 lines or whatever is
still a bit large to handle that way if you must reindex it.

So, to conclude, draw it yourself in a normal CView, based on a 'current
line' value (just reading the relevant part of the file), go fixed length if
possible. Otherwise, investigate if it is possible to use an indexfile for
the line starts.

Johan Rosengren
Responsable Informatique
PACTA S.A.



Quote:
> Hi Everybody.
> Need to display a huge text file in a MFC CRichEdit (preferrably)
> or CEdit or some other MFC window. The file can be 500 MB. This
> is larger than Virtual Memory. Please help by giving pointer to
> source code that does this or by explaning how.

> My best current idea is to have buttons that allow for scrolling
> or override the scrollbar so that when I reach the end of the
> currently displayed portion I will read the next portion of the
> file and throw the least recently used portion from memory. I
> will always keep 3 portions in memory. These portions will be
> 1000 lines long.

> Another problem is allowing the user to jump directly to a line
> number.If he wants to go to a line 5000 lines ahead, I have to
> read all the lines till that line and then read it. Currently I
> can't see how to get past this problem unless the text file is
> written in fixed size lines.

> Thanks for your help,
> Guy Shaham



Thu, 25 Jul 2002 03:00:00 GMT  
 Help: Need to display a large text file
Well, forget the built-in controls. You don't need to use fixed-width
fonts as long as you don't want to do selection at the character level
(you can do this with variable-width fonts but it is very, very, very
painful and tedious to get right!)

You will need an index file, as suggested below, to speed up the
search. If the file is "stable" then you only need to create this
once. If it can be appended to, then you simply create an index file
and force the date-written of the index file to correspond to the
date-written of the main file, and then only worry about indexing the
new lines which have been added, and do that only if someone asks to
jump to a line outside the range. If the file can be arbirarily
modified you are in deep trouble, since you can use the technique I
just described to see if the index is correct, but if it is wrong, you
don't know how wrong it is. In a system I built some years ago, each
line in the file had a key on which it was sorted, so I could use an
obsolete index and treat it as a "hint" as to where to go; if I was
off, I revised the index for just the region I was looking at.

Working with massive data requires subtley. Reindexing a 500MB file
takes nontrivial time, so you have to come up with some heuristics for
optimizing this. Can you say more about the nature of the content of
this large file and how it is being used (read-only, appendable,
arbitrarily changeable, keyed, etc.)?
                        joe

On Sun, 6 Feb 2000 14:31:12 +0100, "Johan Rosengren"

Quote:

>If the file is read-only, which I suppose it is, I would forego prewritten
>controls altogether.

>If you are just supposed to display the contents on-screen, you might as
>well paint it yourself. You will probably not have any normal
>scrollbar-handling with that size of a file, and if you can force a
>fixed-line length format, you would really have it much easier. You could
>have an indexfile with the line-starts, but 10 000 000 lines or whatever is
>still a bit large to handle that way if you must reindex it.

>So, to conclude, draw it yourself in a normal CView, based on a 'current
>line' value (just reading the relevant part of the file), go fixed length if
>possible. Otherwise, investigate if it is possible to use an indexfile for
>the line starts.

>Johan Rosengren
>Responsable Informatique
>PACTA S.A.



>> Hi Everybody.
>> Need to display a huge text file in a MFC CRichEdit (preferrably)
>> or CEdit or some other MFC window. The file can be 500 MB. This
>> is larger than Virtual Memory. Please help by giving pointer to
>> source code that does this or by explaning how.

>> My best current idea is to have buttons that allow for scrolling
>> or override the scrollbar so that when I reach the end of the
>> currently displayed portion I will read the next portion of the
>> file and throw the least recently used portion from memory. I
>> will always keep 3 portions in memory. These portions will be
>> 1000 lines long.

>> Another problem is allowing the user to jump directly to a line
>> number.If he wants to go to a line 5000 lines ahead, I have to
>> read all the lines till that line and then read it. Currently I
>> can't see how to get past this problem unless the text file is
>> written in fixed size lines.

>> Thanks for your help,
>> Guy Shaham

Joseph M. Newcomer [MVP]

Web: http://www3.pgh.net/~newcomer
MVP Tips: http://www3.pgh.net/~newcomer/mvp_tips.htm


Thu, 25 Jul 2002 03:00:00 GMT  
 Help: Need to display a large text file


Quote:
> Well, forget the built-in controls. You don't need to use fixed-width
> fonts as long as you don't want to do selection at the character level
> (you can do this with variable-width fonts but it is very, very, very
> painful and tedious to get right!)

If it wasn't obvious from my answer (the fixed-length lines), I naturally
meant fixed number of characters in the *file*, not on screen.

Johan Rosengren
Responsable Informatique
PACTA S.A.



Fri, 26 Jul 2002 03:00:00 GMT  
 Help: Need to display a large text file
You can scan the file once when you start up the app and whenever the
text file changes and keep track of at what byte position each line or
each nth line begins.  If you keep track of every 100th line, when a
user wants to jump to line 5145, you can go directly to the position
for line 5100 and do a sequential read from there to find the line you
really want.

If the text file doesn't change very often, you may want to save the
index file rather than recreating whenever the app starts.



Quote:
>Hi Everybody.
>Need to display a huge text file in a MFC CRichEdit (preferrably)
>or CEdit or some other MFC window. The file can be 500 MB. This
>is larger than Virtual Memory. Please help by giving pointer to
>source code that does this or by explaning how.

>My best current idea is to have buttons that allow for scrolling
>or override the scrollbar so that when I reach the end of the
>currently displayed portion I will read the next portion of the
>file and throw the least recently used portion from memory. I
>will always keep 3 portions in memory. These portions will be
>1000 lines long.

>Another problem is allowing the user to jump directly to a line
>number.If he wants to go to a line 5000 lines ahead, I have to
>read all the lines till that line and then read it. Currently I
>can't see how to get past this problem unless the text file is
>written in fixed size lines.

>Thanks for your help,
>Guy Shaham

Don Grasberger

(remove --- from address to e-mail)



Fri, 26 Jul 2002 03:00:00 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. Need to display a large amount of text in a view

2. Need help displaying and scrolling LARGE bitmap

3. CEdit::CharFromPos failing for large text, need help

4. Help - Insertion in large text files

5. Need help on save texts format by Textout to a text file

6. HELP: Binary Large Objects (need to store binary file w/in database)

7. HELP: Binary Large Objects (need to store binary file w/in database)

8. Seeking comments -- problems with large text display

9. Opening a text file/Displaying text line by line

10. Need help displaying .BMP file

11. Need HELP with Displaying Bitmap files

12. Need help displaying BMP files

 

 
Powered by phpBB® Forum Software