Problem parsing a log file into a db (a little bit long) 
Author Message
 Problem parsing a log file into a db (a little bit long)

Hello,

I'm trying to put all my mail server logs into a MySQL database for full
text searching. I'm not sure if it's viable for what I really want, but
the problem is I cannot get the logs into the db in the first place.

The logs look like this (munged):
2006-06-22 00:01:12 plain_auth authenticator failed for
host10-10-10-10.foo.bar.com (abcd) [86.130.246.181]: 535 Incorrect
authentication data (set_id=abcd)

H=10.11.12.foo.bar.baz (fred) [80.189.39.100] P=smtp S=11630

and so on.

I've tracked down the error to when I am parsing the date field, so I
simplified the logs to just:
2006-06-22 00:01:12
2006-06-22 00:01:12
2006-06-22 00:01:13

... and I still get the error.

The database looks like this:
+-------+------------------+------+-----+---------+----------------+
| Field | Type             | Null | Key | Default | Extra          |
+-------+------------------+------+-----+---------+----------------+
| id    | int(10) unsigned | NO   | PRI |         | auto_increment |
| date  | char(19)         | YES  |     |         |                |
| ml    | text             | YES  | MUL |         |                |
+-------+------------------+------+-----+---------+----------------+

I've simplified the date field to a char in case MySQL doesn't like the
format of the date.

The Tcl script I am using is this:

#!/usr/local/bin/tclsh8.4

package require sql

set file ./mainlog.01
set f [open $file r]

if {[catch {set conn [sql connect localhost fred]} err]} {
   puts $err
   exit 1

Quote:
}

sql selectdb $conn fred

while {[gets $f line] >= 0} {
   set date [string range $line 0 18]
   #set ml [string range $line 20 end]
   sql query $conn "insert into logs (date) VALUES ('$date')"

Quote:
}

I'm using string range as the log part of the entry (anything after the
date) has numerous different formats depending on what is logged.

One row gets inserted (even if I insert with the date and the ml entry,
but then the script halts with:

fneeded struct::matrix    2.0   [list source [file join $dir matrix.tcl]]
package ifneeded struct::pool      1.2.1 [list source [file join $dir
pool.tcl]]
package ifneeded struct::record    1.2.1 [list source [file join $dir
record.tcl]]
package ifneeded struct::list      1.4   [list source [file join $dir
list.tcl]]
package ifneeded struct::set       2.1   [list source [file join $dir
sets.tcl]]
package ifneeded struct::prioqueue 1.3   [list source [file join $dir
prioqueue.tcl]]
package ifneeded struct::skiplist  1.3   [list source [file join $dir
skiplist.tcl]]

package ifneeded struct::graph     1.2.1 [list source [file join $dir
graph1.tcl]]
package ifneeded struct::tree      1.2.1 [list source [file join $dir
tree1.tcl]]
package ifneeded struct::matrix    1.2.1 [list source [file join $dir
matrix1.tcl]]

     while executing
"sql query $conn "insert into logs (date) VALUES ('$date')""
     ("while" body line 5)
     invoked from within
"while {[gets $f line] >= 0} {
   set date [string range $line 0 18]
   #set ml [string range $line 20 end]

   sql query $conn "insert into logs (date) V..."
     (file "./test2.tcl" line 16)

The log is exactly as above, including the missing "i" in "fneeded".

Can someone help? I've tried all sorts of simplifications but am not
really getting anywhere.

Regards,
Frem.



Tue, 09 Dec 2008 19:02:34 GMT  
 Problem parsing a log file into a db (a little bit long)
Fremlins schrieb:
Quote:
> Hello,

> I'm trying to put all my mail server logs into a MySQL database for full
> text searching. I'm not sure if it's viable for what I really want, but
> the problem is I cannot get the logs into the db in the first place.

> The logs look like this (munged):
> 2006-06-22 00:01:12 plain_auth authenticator failed for
> host10-10-10-10.foo.bar.com (abcd) [86.130.246.181]: 535 Incorrect
> authentication data (set_id=abcd)

> H=10.11.12.foo.bar.baz (fred) [80.189.39.100] P=smtp S=11630

> and so on.

If your mainly after fulltext searching instead of sql functionality
also take a look at xapian.
http://www.xapian.org

For example code using it with Tcl see:
http://wiki.tcl.tk/15992
and the examples directory in the xapian bindings distribution.

If you just want to have LIKE available, use MySQL or some other sql
database. (sqlite, mysql, postgresql)

Michael



Tue, 09 Dec 2008 20:20:25 GMT  
 Problem parsing a log file into a db (a little bit long)
Fremlins schrieb:

Quote:
> Hello,

> I'm trying to put all my mail server logs into a MySQL database for full
> text searching. I'm not sure if it's viable for what I really want, but
> the problem is I cannot get the logs into the db in the first place.

> The logs look like this (munged):
> 2006-06-22 00:01:12 plain_auth authenticator failed for
> host10-10-10-10.foo.bar.com (abcd) [86.130.246.181]: 535 Incorrect
> authentication data (set_id=abcd)

> H=10.11.12.foo.bar.baz (fred) [80.189.39.100] P=smtp S=11630

> and so on.

> I've tracked down the error to when I am parsing the date field, so I
> simplified the logs to just:
> 2006-06-22 00:01:12
> 2006-06-22 00:01:12
> 2006-06-22 00:01:13

> ... and I still get the error.

> The database looks like this:
> +-------+------------------+------+-----+---------+----------------+
> | Field | Type             | Null | Key | Default | Extra          |
> +-------+------------------+------+-----+---------+----------------+
> | id    | int(10) unsigned | NO   | PRI |         | auto_increment |
> | date  | char(19)         | YES  |     |         |                |
> | ml    | text             | YES  | MUL |         |                |
> +-------+------------------+------+-----+---------+----------------+

> I've simplified the date field to a char in case MySQL doesn't like the
> format of the date.

> The Tcl script I am using is this:

> #!/usr/local/bin/tclsh8.4

> package require sql

> set file ./mainlog.01
> set f [open $file r]

> if {[catch {set conn [sql connect localhost fred]} err]} {
>   puts $err
>   exit 1
> }

> sql selectdb $conn fred

> while {[gets $f line] >= 0} {
>   set date [string range $line 0 18]
>   #set ml [string range $line 20 end]
>   sql query $conn "insert into logs (date) VALUES ('$date')"
> }

> I'm using string range as the log part of the entry (anything after the
> date) has numerous different formats depending on what is logged.

> One row gets inserted (even if I insert with the date and the ml entry,
> but then the script halts with:

> fneeded struct::matrix    2.0   [list source [file join $dir matrix.tcl]]
> package ifneeded struct::pool      1.2.1 [list source [file join $dir
> pool.tcl]]
> package ifneeded struct::record    1.2.1 [list source [file join $dir
> record.tcl]]
> package ifneeded struct::list      1.4   [list source [file join $dir
> list.tcl]]
> package ifneeded struct::set       2.1   [list source [file join $dir
> sets.tcl]]
> package ifneeded struct::prioqueue 1.3   [list source [file join $dir
> prioqueue.tcl]]
> package ifneeded struct::skiplist  1.3   [list source [file join $dir
> skiplist.tcl]]

> package ifneeded struct::graph     1.2.1 [list source [file join $dir
> graph1.tcl]]
> package ifneeded struct::tree      1.2.1 [list source [file join $dir
> tree1.tcl]]
> package ifneeded struct::matrix    1.2.1 [list source [file join $dir
> matrix1.tcl]]

>     while executing
> "sql query $conn "insert into logs (date) VALUES ('$date')""
>     ("while" body line 5)
>     invoked from within
> "while {[gets $f line] >= 0} {
>   set date [string range $line 0 18]
>   #set ml [string range $line 20 end]

>   sql query $conn "insert into logs (date) V..."
>     (file "./test2.tcl" line 16)

> The log is exactly as above, including the missing "i" in "fneeded".

> Can someone help? I've tried all sorts of simplifications but am not
> really getting anywhere.

The error sounds like the sql package tries to package require the
Tcllib struct package and your tcllib seems to have a broken
pkgIndex.tcl file for struct.

Try reinstalling your tcllib or use a more recent one from an ActivTcl
install, which tracks CVS AFAIK.

Michael



Tue, 09 Dec 2008 20:22:50 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. Parsing Logs...Please advise (long post)

2. Some questions and annotations to FP (a little bit longer)

3. dll fortran for visual basic .net (little bit long data output)

4. newbie needs a little help parsing a comma delimited file

5. Elegant Scheme solution to little parsing problem wanted

6. parsing a long text file for specific text

7. 3D array of 26-bit-values - performance problems (long)

8. DB: Gadfly 's killing my DB files

9. log_file in expect :: problem in writing to the log file

10. Problems with creating a log file.

11. Expect and interact -o file logging problems

12. Expect and interact -o problems with logging to file

 

 
Powered by phpBB® Forum Software