Processing flat file that is always appending 
Author Message
 Processing flat file that is always appending

I need to be able to read from an ever increasing regular
text file, such that I need to process each line looking
for text etc. In the world I know I'd use this shell
tail -f filename | program.
Right now my wsh script is looping, making a copy,
sleeping, doing a diff and then string matching the new
lines. This worked fine until the file got to 20 MEG. I
need to do this more efficiently.
Any tips would be appreciated.
Thanks


Mon, 01 Aug 2005 07:20:31 GMT  
 Processing flat file that is always appending
I recently offered a solution to a similar problem in this newsgroup with
the subject "line match in file".

I've modified that solution for you below.

This supports either "ADO" or "FSO" based on "cTYP".

Modify "cFIL" to point to your file.

Modify "cTXT" to have your search string..

It should work as is.

(Watch for word wrap especially in the comment header)

    Option Explicit
   '====================================
=======================================
   '  "Qgrepper.vbs"    Date Created: 07-Feb-2003    Date Updated:
12-Feb-2003
   '
   '  This VBS (Visual Basic Script) program does the following:
   '  1)  Specify "cFIL" = a text file to be examined.
   '  2)  Specify "cTXT" = text to find within a line.
   '  3)  Open the file and examine the records using "ADODB.Stream".
   '  4)  Report only the first instance found.
   '
   '  To test, either double-click on the filename in Windows Explorer or
   '  from the Command prompt, type "cscript.exe //nologo Qgreping.vbs".
   '
   '  Changes:

'  ---------------------------------------- --------------------------------
-
   '  12-Feb-2003.1  Changed.
   '  07-Feb-2003.0  Created.
   '
   '==========================
=================================================
    Const cVBS = "Qgrepper.vbs"
    Const cFIL = "Qgrepper.vbs"
    Const cTXT = "Explicit"
'    Const cTYP = "ADO"
    Const cTYP = "FSO"
   '*
   '*  Start Message
   '*
    MsgBox "'" & cVBS & "' started."
   '*
   '*  Processing
   '*
    Call Processing()
   '*
   '*  Finish Message
   '*
    MsgBox "'" & cVBS & "' finished."

Sub Processing()
   '*
   '*
   '*
    WScript.Echo "'" & cFIL & "' grep via '" & cTYP & "'"
    Dim strDAT
    If cTYP = "ADO" Then
       '*
       '*  Declare Objects, LoadFromFile, Cleanup Objects
       '*
        Dim objADO
        Set objADO = CreateObject("ADODB.Stream")
            objADO.Open
            objADO.Type = 2 '= adTypeText
            objADO.Charset = "ascii"
            objADO.LoadFromFile(cFIL)
        WScript.Echo "'" & cFIL & "' = " & objADO.Size & " bytes."
            strDAT = objADO.ReadText
            objADO.Close
        Set objADO = Nothing
    Else
       '*
       '*  Declare Objects, ReadAll, Cleanup Objects
       '*
        Dim objFSO
        Set objFSO = CreateObject("Scripting.FileSystemObject")
        Dim objGFI
        Set objGFI = objFSO.GetFile(cFIL)
        WScript.Echo "'" & cFIL & "' = " & objGFI.Size & " bytes."
        Set objGFI = Nothing
        Dim objOTF
        Set objOTF = objFSO.OpenTextFile(cFIL,1)
            strDAT = objOTF.ReadAll()
            objOTF.Close
        Set objOTF = Nothing
        Set objFSO = Nothing
    End If
   '*
   '*
   '*
    Dim arrDAT
        arrDAT = Split(strDAT,vbCrLf)
    WScript.Echo "'" & cFIL & "' = " & UBound(arrDAT) & " lines."
   '*
   '*  Declare Variables
   '*
    Dim intTXT
    Dim strTXT
        strTXT = "'" & cFIL & "' does not contain '" & cTXT & "'"
   '*
   '*  Examine File
   '*
    For intTXT = 0 To UBound(arrDAT)
        If InStr(arrDAT(intTXT),cTXT) > 0 Then
            strTXT = "'" & cFIL & "' contains '" & cTXT & "' on line " &
intTXT+1 & ":"
            strTXT = strTXT & vbCrLf & vbCrLf & arrDAT(intTXT)
            Exit For
        End If
    Next
    WScript.Echo strTXT
End Sub


Quote:
> I need to be able to read from an ever increasing regular
> text file, such that I need to process each line looking
> for text etc. In the world I know I'd use this shell
> tail -f filename | program.
> Right now my wsh script is looping, making a copy,
> sleeping, doing a diff and then string matching the new
> lines. This worked fine until the file got to 20 MEG. I
> need to do this more efficiently.
> Any tips would be appreciated.
> Thanks



Mon, 01 Aug 2005 07:43:21 GMT  
 Processing flat file that is always appending
Your script does indeed work, but it goes through the file
finds the first match and then terminates. What I need is
the ability to is find all the matches in the file upon
start up of the script and then the script waits for new
lines to added to the source file and parses each new line
as it comes in looking for the match. Is that part
difficult to to add to this excellent core task ?
Thanks

Quote:
>-----Original Message-----
>I recently offered a solution to a similar problem in
this newsgroup with
>the subject "line match in file".

>I've modified that solution for you below.
>>> I need to be able to read from an ever increasing
regular
>> text file, such that I need to process each line looking
>> for text etc. In the world I know I'd use this shell
>> tail -f filename | program.
>> Right now my wsh script is looping, making a copy,
>> sleeping, doing a diff and then string matching the new
>> lines. This worked fine until the file got to 20 MEG. I
>> need to do this more efficiently.
>> Any tips would be appreciated.
>> Thanks

>.



Mon, 01 Aug 2005 08:53:36 GMT  
 Processing flat file that is always appending
Not at all.

I merely provided a very good starting point.

Do you need it solved for you or can you modify it?


Quote:
> Your script does indeed work, but it goes through the file
> finds the first match and then terminates. What I need is
> the ability to is find all the matches in the file upon
> start up of the script and then the script waits for new
> lines to added to the source file and parses each new line
> as it comes in looking for the match. Is that part
> difficult to to add to this excellent core task ?
> Thanks

> >-----Original Message-----
> >I recently offered a solution to a similar problem in
> this newsgroup with
> >the subject "line match in file".

> >I've modified that solution for you below.
> >>> I need to be able to read from an ever increasing
> regular
> >> text file, such that I need to process each line looking
> >> for text etc. In the world I know I'd use this shell
> >> tail -f filename | program.
> >> Right now my wsh script is looping, making a copy,
> >> sleeping, doing a diff and then string matching the new
> >> lines. This worked fine until the file got to 20 MEG. I
> >> need to do this more efficiently.
> >> Any tips would be appreciated.
> >> Thanks

> >.



Mon, 01 Aug 2005 09:45:25 GMT  
 Processing flat file that is always appending
Okay... here it is.  Again, watch for word wrap.

    Option Explicit
   '=====================================
======================================
   '  "Qgrepper.vbs"    Date Created: 12-Feb-2003    Date Updated:
12-Feb-2003
   '
   '  This VBS (Visual Basic Script) program does the following:
   '  1)  Specify "cFIL" = a text file to be examined.
   '  2)  Specify "cTXT" = text to find within a line.
   '  3)  Open the file and examine the records using.
   '  4)  Report all instances found (by line number).
   '
   '  To test, either double-click on the filename in Windows Explorer or
   '  from the Command prompt, type "cscript.exe //nologo Qgreping.vbs".
   '
   '  Changes:

'  ----------------------------------- -------------------------------------
-
   '  12-Feb-2003.1  Changed re "find all instances".
   '  12-Feb-2003.0  Created.
   '
   '=======================================
====================================
    Const cVBS = "Qgrepper.vbs"
    Const cFIL = "Qgrepper.vbs"
    Const cTXT = "Explicit"
   'Const cTYP = "ADO"
    Const cTYP = "FSO"
   '*
   '*  Start Message
   '*
    MsgBox "'" & cVBS & "' started."
   '*
   '*  Processing
   '*
    Call Processing()
   '*
   '*  Finish Message
   '*
    MsgBox "'" & cVBS & "' finished."

Sub Processing()
   '*
   '*
   '*
   'WScript.Echo "'" & cFIL & "' grep via '" & cTYP & "'"
    Dim strDAT
    If cTYP = "ADO" Then
       '*
       '*  Declare Objects, LoadFromFile, Cleanup Objects
       '*
        Dim objADO
        Set objADO = CreateObject("ADODB.Stream")
            objADO.Open
            objADO.Type = 2 '= adTypeText
            objADO.Charset = "ascii"
            objADO.LoadFromFile(cFIL)
       'WScript.Echo "'" & cFIL & "' = " & objADO.Size & " bytes."
            strDAT = objADO.ReadText
            objADO.Close
        Set objADO = Nothing
    Else
       '*
       '*  Declare Objects, ReadAll, Cleanup Objects
       '*
        Dim objFSO
        Set objFSO = CreateObject("Scripting.FileSystemObject")
        Dim objGFI
        Set objGFI = objFSO.GetFile(cFIL)
       'WScript.Echo "'" & cFIL & "' = " & objGFI.Size & " bytes."
        Set objGFI = Nothing
        Dim objOTF
        Set objOTF = objFSO.OpenTextFile(cFIL,1)
            strDAT = objOTF.ReadAll()
            objOTF.Close
        Set objOTF = Nothing
        Set objFSO = Nothing
    End If
   '*
   '*
   '*
    Dim arrDAT
        arrDAT = Split(strDAT,vbCrLf)
   'WScript.Echo "'" & cFIL & "' = " & UBound(arrDAT) & " lines."
   '*
   '*  Declare Variables
   '*
    Dim intTXT
    Dim strTXT
        strTXT = ""
   '*
   '*  Examine File
   '*
    For intTXT = 0 To UBound(arrDAT)
        If InStr(arrDAT(intTXT),cTXT) > 0 Then
            strTXT = strTXT & intTXT+1 & ": " & arrDAT(intTXT) & vbCrLf
        End If
    Next
   '*
   '*  Display Results
   '*
    If strTXT = "" Then
         strTXT = "'" & cFIL & "' does not contain '" & cTXT & "'"
    Else
        strTXT = "'" & cFIL & "' contains " & vbCrLf & "'" & cTXT & "' on
lines: " & vbCrLf & vbCrLf & strTXT
    End If
    WScript.Echo strTXT
End Sub

Quote:


> > Your script does indeed work, but it goes through the file
> > finds the first match and then terminates. What I need is
> > the ability to is find all the matches in the file upon
> > start up of the script and then the script waits for new
> > lines to added to the source file and parses each new line
> > as it comes in looking for the match. Is that part
> > difficult to to add to this excellent core task ?
> > Thanks

> > >-----Original Message-----
> > >I recently offered a solution to a similar problem in
> > this newsgroup with
> > >the subject "line match in file".

> > >I've modified that solution for you below.
> > >>> I need to be able to read from an ever increasing
> > regular
> > >> text file, such that I need to process each line looking
> > >> for text etc. In the world I know I'd use this shell
> > >> tail -f filename | program.
> > >> Right now my wsh script is looping, making a copy,
> > >> sleeping, doing a diff and then string matching the new
> > >> lines. This worked fine until the file got to 20 MEG. I
> > >> need to do this more efficiently.
> > >> Any tips would be appreciated.
> > >> Thanks

> > >.



Mon, 01 Aug 2005 11:49:28 GMT  
 Processing flat file that is always appending
My script only examines a file that is available to be opened.

It does nothing about your other requirement:

"and then the script waits for new lines to added to the source file and
parses each new line as it comes in looking for the match.".


Quote:
> Your script does indeed work, but it goes through the file
> finds the first match and then terminates. What I need is
> the ability to is find all the matches in the file upon
> start up of the script and then the script waits for new
> lines to added to the source file and parses each new line
> as it comes in looking for the match. Is that part
> difficult to to add to this excellent core task ?
> Thanks



Mon, 01 Aug 2005 11:53:36 GMT  
 Processing flat file that is always appending

Quote:

> I need to be able to read from an ever increasing regular
> text file, such that I need to process each line looking
> for text etc. In the world I know I'd use this shell
> tail -f filename | program.
> Right now my wsh script is looping, making a copy,
> sleeping, doing a diff and then string matching the new
> lines. This worked fine until the file got to 20 MEG. I
> need to do this more efficiently.
> Any tips would be appreciated.
> Thanks

Hi, Erfuert
I some questions about this.
Right now, some system adds lines to this 20 megabyte file.
1) Will this go on forever - the file just gets bigger until it the hard drive won't hold any more?
2) If your script were to rename the file, would the system crash because the file is not there to append to?

-Paul Randall



Mon, 01 Aug 2005 12:47:58 GMT  
 Processing flat file that is always appending
It is simply a syslog file that routers are writing to on
a win2000 box. I can control the stop of the syslog
service, init a new file and restart when I need, probably
daily as the file gets to about 20 meg/day. During that
one day, I'd like to be able to read each newline and
parse it for matches. I just tried a move of the file to a
new name while the syslog service was running and it was
unfazed, it recreated the old file no problem. I think
that is my solution, thanks for making me check that.

Quote:
>Hi, Erfuert
>I some questions about this.
>Right now, some system adds lines to this 20 megabyte
file.
>1) Will this go on forever - the file just gets bigger

until it the hard drive won't hold any more?
Quote:
>2) If your script were to rename the file, would the

system crash because the file is not there to append to?
Quote:

>-Paul Randall

>.



Mon, 01 Aug 2005 14:40:41 GMT  
 
 [ 8 post ] 

 Relevant Pages 

1. Combine 2 files (append one to the other) (textstream / append)

2. Help with SQL and flat file data

3. import outlook address book to flat file

4. import flat file

5. writing form data to a flat file.

6. Write to flat file in .asp

7. Export (flat file through textstream or print#) need to break the fields at column 73

8. Help with SQL and flat file

9. print a flat-file image in PS

10. Using FSO to append multiple .csv excel files into one file

11. PS file from MAC always smaller than from PC

12. FreeDist - A freeware distiller that can also append pdf files to 1 pdf

 

 
Powered by phpBB® Forum Software