File Reading in 'J' 
Author Message
 File Reading in 'J'


Quote:
>  I'm trying to design a simple program where J would
> read a file line by line (delimiting with LF).
> In my particular case the file can be huge so
> I don't want to load it into memory.

> What I am essentially trying to mimic is the PERL construct:
> while (<>)

        > {
... > do something interesting with the line
        > }
Quote:

> in a way that makes reasonable use of the speed and power of J.
>  In my initial attempts my J code took a lot longer than my PERL
> code, so I *know* I am doing things in an abysmal way from 'J's
> viewpoint. Can anyone suggest an approach?

I am not going to solve this for you but I send you some things
you can test on your own.

require'convert misc'

while. T do. B end.

you can have T to test for LF and/or EOF

whilst. T do. B end.

whilst does the same but executes the code once first before the evaluation.

for B pick some combination from below

PATH=:'c:\'
FILE=:'autoexec.bat'
FILE_POS=:0
REC_LENGD=:80
FILE_SIZE=: fsize PATH,FILE
REC_LENGD < FILE_SIZE

REC=: fread (PATH,FILE);FILE_POS,REC_LENGD

FIND_LF=:(>{.(REC=LF)#i. #REC)-1

REC_USE=:FIND_LF {. REC

av REC_USE

c=:fread PATH,FILE
av c

EOF=: 26 { a.

FIND_EOF=:(>{.(c=EOF)#i. #c)-1

d=: chop c

Combine this into verbs and read in various lengths of recs and
do timings. You will notice that if you start by reading one char at a time
(the way C does) then J will not give you much speed.

Increase the size gradually until you think you can not comfortably
handle more at once in your locale at once (for whatever reasons you
may have for making that decision. I read in many megs at one time
and handle it without problems.)

Then try split up your records with chop and operate on them and then read some
more. You will notice an enormous increase in speed. Speeds you will
never reach with your C program (if it is not trivial).

/Gosi



Tue, 28 Mar 2000 03:00:00 GMT  
 File Reading in 'J'

Thank you for trying to help.

I'm not sure I understand your answer completely yet, but it does appear
to me that what you suggest involves calling `fread' once per line.
If so, that seems to take minutes, and since Perl solves the problem
in a few seconds I can't believe that fine tuning will make up the difference.

Do I misunderstand (i.e. `fread' isn't called once per line)?

Thanks again...

Quote:


> >  I'm trying to design a simple program where J would
> > read a file line by line (delimiting with LF).
> > In my particular case the file can be huge so
> > I don't want to load it into memory.

> > What I am essentially trying to mimic is the PERL construct:
> > while (<>)
>         > {
> ... > do something interesting with the line
>         > }

> > in a way that makes reasonable use of the speed and power of J.
> >  In my initial attempts my J code took a lot longer than my PERL
> > code, so I *know* I am doing things in an abysmal way from 'J's
> > viewpoint. Can anyone suggest an approach?
> I am not going to solve this for you but I send you some things
> you can test on your own.

> require'convert misc'

> while. T do. B end.

> you can have T to test for LF and/or EOF

> whilst. T do. B end.

> whilst does the same but executes the code once first before the evaluation.

> for B pick some combination from below

> PATH=:'c:\'
> FILE=:'autoexec.bat'
> FILE_POS=:0
> REC_LENGD=:80
> FILE_SIZE=: fsize PATH,FILE
> REC_LENGD < FILE_SIZE

> REC=: fread (PATH,FILE);FILE_POS,REC_LENGD

> FIND_LF=:(>{.(REC=LF)#i. #REC)-1

> REC_USE=:FIND_LF {. REC

> av REC_USE

> c=:fread PATH,FILE
> av c

> EOF=: 26 { a.

> FIND_EOF=:(>{.(c=EOF)#i. #c)-1

> d=: chop c

> Combine this into verbs and read in various lengths of recs and
> do timings. You will notice that if you start by reading one char at a time
> (the way C does) then J will not give you much speed.

> Increase the size gradually until you think you can not comfortably
> handle more at once in your locale at once (for whatever reasons you
> may have for making that decision. I read in many megs at one time
> and handle it without problems.)

> Then try split up your records with chop and operate on them and then read some
> more. You will notice an enormous increase in speed. Speeds you will
> never reach with your C program (if it is not trivial).

> /Gosi



Wed, 29 Mar 2000 03:00:00 GMT  
 File Reading in 'J'


Quote:
> Thank you for trying to help.
> I'm not sure I understand your answer completely yet,
> but it does appear
> to me that what you suggest involves calling 'fread' once per line.
> If so, that seems to take minutes, and since Perl solves the
> problem in a few seconds I can't believe that fine tuning
> will make up the difference.
> Do I misunderstand (i.e. 'fread' isn't called once per line)?

I am not really sure what you are looking for and because
there are various ways of reading files in J then I was
merely pointing at a means for you to test for yourself
how you can read files in a few ways.

0) You can read the whole file at once
1) You can read one char at a time
2) You can read several chars at a time

1) and 2) are basically a variation of the same theme
but have the advantage of possibly beeing similar to what you
are familiar with.

0) is the easiest and most efficient way.

What is important is what you do with the data
after you have read it.

If all you want to do is read the data and not doing
anything with it then try to read as much as you can
at one read, the whole file if possible is obviously best.
Then you do not need to implement any controls or
anything and it is very fast.
/Gosi



Sat, 01 Apr 2000 03:00:00 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. 'wav' files reading

2. File reading and line continuation with '\'

3. Opening a file with ACTION='READ'

4. File Object -- function 'read'

5. Reading unformatted files 'bit a bit'

6. Reading fortran 'unformatted' binary files

7. reading 'stream_lz' data files

8. stream('file','c','seek ='x) problem

9. 'file mkdir' doesn't fail

10. File Reading in `J'

11. .DAT file can't be read under Netware

12. reading Ini files - what's the trick?

 

 
Powered by phpBB® Forum Software