FYI - Re: Speed of searching file using Queues 
Author Message
 FYI - Re: Speed of searching file using Queues

Hi,

Last week I explained my delima of a search form I created that had to do a
string search on a TPS file containing over 200,000 records. It was getting
slow.... About 20-25 seconds for a string search. I was using Instring() to
search for partial strings so I could find things like "F9YC" in the string
"454F9YC5". I got alot of help from everyone, so here is what I did and the
results for FYI purposes.

Created a Queue that was exact replica of the file on disk. Basically looped
through each record added it to the queue. Time: about 2.5 seconds, but I
only do this at program start up. This application is started and terminated
alot so keeping the queue in sync with the data on disk is no problem. I
also do a quick compare before the search to count the records in the queue
and the disk file, if there different I rebuild the queue, this only detects
adds or deletes which is good enough, if there the same I skip the rebuild.

I then search the queue using the same instring -- Time: about 3 seconds
max.

So using a queue in this case reduced the search time from 25 seconds to  a
maximum of  5 (usually 3) - Very Nice!

Thanks for the input.

- Joe



Sat, 01 Jan 2005 07:24:54 GMT  
 FYI - Re: Speed of searching file using Queues
Joe:
    Very interesting. But I can't understand why is this faster. You are
making the same operations than before,(and some new)
You Iterated the entire file, and you applied the instring function to each
record.
Now you are doing the same, but usin a queue as an intermedita storage.
The only explanation is that runing instring over a file field is slower
that doing it over the queue.
Are you shure there is no other change in the program that can reise
performance?

Just curious.

--
Sebastin Streiger
Departamento Sistemas
Red Megatone
Tel: 0342-4502745
Int: 745



Quote:
> Hi,

> Last week I explained my delima of a search form I created that had to do
a
> string search on a TPS file containing over 200,000 records. It was
getting
> slow.... About 20-25 seconds for a string search. I was using Instring()
to
> search for partial strings so I could find things like "F9YC" in the
string
> "454F9YC5". I got alot of help from everyone, so here is what I did and
the
> results for FYI purposes.

> Created a Queue that was exact replica of the file on disk. Basically
looped
> through each record added it to the queue. Time: about 2.5 seconds, but I
> only do this at program start up. This application is started and
terminated
> alot so keeping the queue in sync with the data on disk is no problem. I
> also do a quick compare before the search to count the records in the
queue
> and the disk file, if there different I rebuild the queue, this only
detects
> adds or deletes which is good enough, if there the same I skip the
rebuild.

> I then search the queue using the same instring -- Time: about 3 seconds
> max.

> So using a queue in this case reduced the search time from 25 seconds to
a
> maximum of  5 (usually 3) - Very Nice!

> Thanks for the input.

> - Joe



Fri, 21 Jan 2005 23:25:09 GMT  
 FYI - Re: Speed of searching file using Queues
Hi Sebastian,

On Sat, 20 Jul 2002 18:37:30 -0300, "Sebastian Streiger"

Quote:

>The only explanation is that runing instring over a file field is slower
>that doing it over the queue.
>Are you shure there is no other change in the program that can reise
>performance?

As I understand it he loads the entire file into the queue when the
program starts.  That means no file access during the queue search,
just memory access (of course that can mean file access if the OS
swaps virtual memory to disk) and that means faster access.  If the
file record structure and the queue structure are exactly the same,
the assignment could by done by using:

MyQueue = MYF:Record

Which would probably be very fast.  

Loading 200,000 records into a queue in 2.5 seconds is pretty good
performance and I'm interested in what kind of CPU/hardware this is
running on?

Best regards,

Arnr Baldvinsson
Icetips Software        
San Antonio, Texas, USA
www.icetips.com

ICQ:  113314380

Subscribe to information from Icetips.com:
http://www.icetips.com/getnotificationinfo.htm



Fri, 21 Jan 2005 23:25:24 GMT  
 FYI - Re: Speed of searching file using Queues
Arnor:

    What I mean is that if you can load a queue in 2.5 seconds then
traversing the file should take less than 2.5.
    Iterating the queue and looking for matches using instring 3 second, so
the instring search (without queue activity) should take less than 3
seconds.
    Now, iterating the file and "instringing" each record shouldn't take
longer than 2.5 + 3 = 5.5 sec.
    How can  the first program take 20 to 30 segonds to execute.
    Could there be a better performance of instring function over queue
fields than over file fields?
    There may be are other context variables that have changed between both
performance test, maybe a background program running, like an antivirus.
    I'm very intrested in what could have cause such a performance change.

--
Sebastin Streiger
Departamento Sistemas
Red Megatone
Tel: 0342-4502745
Int: 745



Quote:
> Hi Sebastian,

> On Sat, 20 Jul 2002 18:37:30 -0300, "Sebastian Streiger"

> >The only explanation is that runing instring over a file field is slower
> >that doing it over the queue.
> >Are you shure there is no other change in the program that can reise
> >performance?

> As I understand it he loads the entire file into the queue when the
> program starts.  That means no file access during the queue search,
> just memory access (of course that can mean file access if the OS
> swaps virtual memory to disk) and that means faster access.  If the
> file record structure and the queue structure are exactly the same,
> the assignment could by done by using:

> MyQueue = MYF:Record

> Which would probably be very fast.

> Loading 200,000 records into a queue in 2.5 seconds is pretty good
> performance and I'm interested in what kind of CPU/hardware this is
> running on?

> Best regards,

> Arnr Baldvinsson
> Icetips Software
> San Antonio, Texas, USA
> www.icetips.com

> ICQ:  113314380

> Subscribe to information from Icetips.com:
> http://www.icetips.com/getnotificationinfo.htm



Fri, 21 Jan 2005 23:33:25 GMT  
 FYI - Re: Speed of searching file using Queues
Hi Sebastian,

On Wed, 24 Jul 2002 19:44:15 -0300, "Sebastian Streiger"

Quote:

>    Could there be a better performance of instring function over queue
>fields than over file fields?

Don't think so.  The interesting part here is why reading the file
into the queue is so quick.

Best regards,

Arnr Baldvinsson
Icetips Software        
San Antonio, Texas, USA
www.icetips.com

ICQ:  113314380

Subscribe to information from Icetips.com:
http://www.icetips.com/getnotificationinfo.htm



Fri, 21 Jan 2005 23:33:36 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. searching one file using the contents of another file

2. FYI: (Un)Binding queue string fields

3. Clarification of Queue Operation / Behavior when using multiple Queues

4. FYI - Strange speeds with latest IBM SysUtil API's

5. Using the Top Speed file driver in Clarion for DOS 3.1 -Reply

6. Using the Top Speed file driver in Clarion for DOS 3.1

7. CLASSes and QUEUEs (was: Re: QUEUE in QUEUE)

8. in-file search/replace question without using mv or cp

9. File search using TT's CatSearch Plugin

10. CW2001.01 Search for number key using GET(file,key) PROBLEM

11. Using Debug to search a very large file?

12. Dropdown directory / file search using TCL 7.4 / TK 4.0

 

 
Powered by phpBB® Forum Software