log file processing - awk or sed 
Author Message
 log file processing - awk or sed

Hi,
I have a little problem with logfile from my application.

My log looks like that :

02/09/17
12:00
12
34
45
67
02/09/17
12:10
11
32
62
69
....

I need a log in that format :
02/09/17|12:00|11|32|62
02/09/17|12:10|14|38|66
I could be pipe or any other delimiter.
So, how can I do this using sed ? or awk ?

regards,
Bart.



Sun, 06 Mar 2005 18:27:24 GMT  
 log file processing - awk or sed
Submitted by "Bartek Brzezi?ski" to comp.lang.awk:

Quote:
> Hi,
> I have a little problem with logfile from my application.

> My log looks like that :

> 02/09/17
> 12:00
> 12
> 34
> 45
> 67
> 02/09/17
> 12:10
> 11
> 32
> 62
> 69
> ....

> I need a log in that format :
> 02/09/17|12:00|11|32|62
> 02/09/17|12:10|14|38|66
> I could be pipe or any other delimiter.
> So, how can I do this using sed ? or awk ?

> regards,
> Bart.

Sorry, but no good AWK solution to this.

paste -s -d '|||||\n' logfile

--
Andreas K?h?ri
--------------------------------------------------------------
Stable, secure, portable, free:     www.netbsd.org



Sun, 06 Mar 2005 19:09:49 GMT  
 log file processing - awk or sed
Hello,

Quote:

> My log looks like that :

> 02/09/17
> 12:00
> 12
> 34
> 45
> 67
> 02/09/17
> 12:10
> 11
> 32
> 62
> 69
> ....

> I need a log in that format :
> 02/09/17|12:00|11|32|62
> 02/09/17|12:10|14|38|66
> I could be pipe or any other delimiter.
> So, how can I do this using sed ? or awk ?

The following simple program shoud work:

awk 'BEGIN{getline x};/[/]/{print x; x=$0; next};{x=x"|"$0};END{print x}'

The regex /[/]/ serves as an identification of start of a new log entry.

(Of course, this can be done with sed too but I'm lazy now.)

HTH,
        Stepan



Sun, 06 Mar 2005 19:18:47 GMT  
 log file processing - awk or sed

Quote:

> Submitted by "Bartek Brzezi?ski" to comp.lang.awk:
>> Hi,
>> I have a little problem with logfile from my application.

>> My log looks like that :

>> 02/09/17
>> 12:00
>> 12
>> 34
>> 45
>> 67
>> 02/09/17
>> 12:10
>> 11
>> 32
>> 62
>> 69
>> ....

>> I need a log in that format :
>> 02/09/17|12:00|11|32|62
>> 02/09/17|12:10|14|38|66
>> I could be pipe or any other delimiter.
>> So, how can I do this using sed ? or awk ?

> Sorry, but no good AWK solution to this.

     ;) You forgot to tell Stepan that! ;)

Quote:
> paste -s -d '|||||\n' logfile

    OT for this group, but what the heck! (Follow-up set)

    (Untested, but will handle any number of lines between dates.)

       sep='|'
       newline=
       while IFS= read line
       do
           case $line in
             [0-9][0-9]/[01][0-9]/[0-3][0-9])
                          [ "$newline" ] && echo "$newline"
                           newline= ;;
             *) newline=${newline:+$newline$sep}$line ;;
           esac
       done < logfile

--
    Chris F.A. Johnson                        http://cfaj.freeshell.org
    ===================================================================
    My code (if any) in this post is copyright 2002, Chris F.A. Johnson
    and may be copied under the terms of the GNU General Public License



Sun, 06 Mar 2005 21:24:12 GMT  
 log file processing - awk or sed

Quote:

>Hi,
>I have a little problem with logfile from my application.

>My log looks like that :

>02/09/17
>12:00
>12
>34
>45
>67
>02/09/17
>12:10
>11
>32
>62
>69
>....

>I need a log in that format :
>02/09/17|12:00|11|32|62
>02/09/17|12:10|14|38|66
>I could be pipe or any other delimiter.
>So, how can I do this using sed ? or awk ?

perhaps something like this (untested) code:

awk '{printf(%s,%s),$0,NR%6==0 ? "\n" : "|" }' logfile

Chuck Demas

--
  Eat Healthy    |   _ _   | Nothing would be done at all,

  Die Anyway     |    v    | That no one could find fault with it.



Mon, 07 Mar 2005 02:34:52 GMT  
 log file processing - awk or sed

Quote:



>>Hi,
>>I have a little problem with logfile from my application.

>>My log looks like that :

>>02/09/17
>>12:00
>>12
>>34
>>45
>>67
>>02/09/17
>>12:10
>>11
>>32
>>62
>>69
>>....

>>I need a log in that format :
>>02/09/17|12:00|11|32|62
>>02/09/17|12:10|14|38|66
>>I could be pipe or any other delimiter.
>>So, how can I do this using sed ? or awk ?

>perhaps something like this (untested) code:

>awk '{printf(%s,%s),$0,NR%6==0 ? "\n" : "|" }' logfile

Or as suggested in another thread:

gawk 'RS=(NR%6==0 ? "\n" : "|")' logfile

Chuck Demas

--
  Eat Healthy    |   _ _   | Nothing would be done at all,

  Die Anyway     |    v    | That no one could find fault with it.



Mon, 07 Mar 2005 02:39:07 GMT  
 log file processing - awk or sed

Quote:





>>>Hi,
>>>I have a little problem with logfile from my application.

>>>My log looks like that :

>>>02/09/17
>>>12:00
>>>12
>>>34
>>>45
>>>67
>>>02/09/17
>>>12:10
>>>11
>>>32
>>>62
>>>69
>>>....

>>>I need a log in that format :
>>>02/09/17|12:00|11|32|62
>>>02/09/17|12:10|14|38|66
>>>I could be pipe or any other delimiter.
>>>So, how can I do this using sed ? or awk ?

>>perhaps something like this (untested) code:

>>awk '{printf(%s,%s),$0,NR%6==0 ? "\n" : "|" }' logfile

>Or as suggested in another thread:

>gawk 'RS=(NR%6==0 ? "\n" : "|")' logfile

Actually, the suggestion is:

    gawk 'RS=NR%6?"|":"\n"' logfile

Every byte costs money.  Think of the poor starving children in Uganda.



Mon, 07 Mar 2005 04:20:20 GMT  
 log file processing - awk or sed

Quote:







>>>>Hi,
>>>>I have a little problem with logfile from my application.

>>>>My log looks like that :

>>>>02/09/17
>>>>12:00
>>>>12
>>>>34
>>>>45
>>>>67
>>>>02/09/17
>>>>12:10
>>>>11
>>>>32
>>>>62
>>>>69
>>>>....

>>>>I need a log in that format :
>>>>02/09/17|12:00|11|32|62
>>>>02/09/17|12:10|14|38|66
>>>>I could be pipe or any other delimiter.
>>>>So, how can I do this using sed ? or awk ?

>>>perhaps something like this (untested) code:

>>>awk '{printf(%s,%s),$0,NR%6==0 ? "\n" : "|" }' logfile

>>Or as suggested in another thread:

>>gawk 'RS=(NR%6==0 ? "\n" : "|")' logfile

>Actually, the suggestion is:

>    gawk 'RS=NR%6?"|":"\n"' logfile

>Every byte costs money.  Think of the poor starving children in Uganda.

But in the above you've got the order reversed, IIRC.

But it's a great method you proposed!  I just couldn't remember your
name to give you the credit.

Chuck Demas

--
  Eat Healthy    |   _ _   | Nothing would be done at all,

  Die Anyway     |    v    | That no one could find fault with it.



Mon, 07 Mar 2005 06:38:31 GMT  
 log file processing - awk or sed


Quote:








> >>>>Hi,
> >>>>I have a little problem with logfile from my application.

> >>>>My log looks like that :

> >>>>02/09/17
> >>>>12:00
> >>>>12
> >>>>34
> >>>>45
> >>>>67
> >>>>02/09/17
> >>>>12:10
> >>>>11
> >>>>32
> >>>>62
> >>>>69
> >>>>....

> >>>>I need a log in that format :
> >>>>02/09/17|12:00|11|32|62
> >>>>02/09/17|12:10|14|38|66
> >>>>I could be pipe or any other delimiter.
> >>>>So, how can I do this using sed ? or awk ?

> >>>perhaps something like this (untested) code:

> >>>awk '{printf(%s,%s),$0,NR%6==0 ? "\n" : "|" }' logfile

> >>Or as suggested in another thread:

> >>gawk 'RS=(NR%6==0 ? "\n" : "|")' logfile

> >Actually, the suggestion is:

> >    gawk 'RS=NR%6?"|":"\n"' logfile

> >Every byte costs money.  Think of the poor starving children in Uganda.

> But in the above you've got the order reversed, IIRC.

> But it's a great method you proposed!  I just couldn't remember your
> name to give you the credit.

> Chuck Demas

> --
>   Eat Healthy    |   _ _   | Nothing would be done at all,

>   Die Anyway     |    v    | That no one could find fault with it.


Hi, guys -

Great concept there.

I actually ran your code (since Kenny got it down so small).

Chuck, Kenny has the correct order around the ":" for NR%6.

Kenny, you lost the "O" in "ORS" when you quoted your own post.

I found that this program yeilds the desired results:

    ORS=NR%6?"|":"\n"

- Dan



Mon, 07 Mar 2005 15:45:29 GMT  
 log file processing - awk or sed

Quote:







>> >>Or as suggested in another thread:

>> >>gawk 'RS=(NR%6==0 ? "\n" : "|")' logfile

>> >Actually, the suggestion is:

>> >    gawk 'RS=NR%6?"|":"\n"' logfile

>> >Every byte costs money.  Think of the poor starving children in Uganda.

>> But in the above you've got the order reversed, IIRC.

>> But it's a great method you proposed!  I just couldn't remember your
>> name to give you the credit.

>Hi, guys -

>Great concept there.

>I actually ran your code (since Kenny got it down so small).

>Chuck, Kenny has the correct order around the ":" for NR%6.

>Kenny, you lost the "O" in "ORS" when you quoted your own post.

>I found that this program yeilds the desired results:

>    ORS=NR%6?"|":"\n"

Yeah, I forgot NR%6 is yes for non-multiples of 6.  :-)

Chuck Demas

--
  Eat Healthy    |   _ _   | Nothing would be done at all,

  Die Anyway     |    v    | That no one could find fault with it.



Tue, 08 Mar 2005 06:15:28 GMT  
 log file processing - awk or sed

Quote:
> Hi,
> I have a little problem with logfile from my application.

> My log looks like that :

> 02/09/17
> 12:00
> 12
> 34
> 45
> 67
> 02/09/17
> 12:10
> 11
> 32
> 62
> 69
> ....

> I need a log in that format :
> 02/09/17|12:00|11|32|62
> 02/09/17|12:10|14|38|66
> I could be pipe or any other delimiter.
> So, how can I do this using sed ? or awk ?

awk 'BEGIN{d="|"}
{ if ( $0 ~ /\//) { printf "\n"$0}else{printf d$0}}
END{print}' fil

Plain and simple.
You can of course add some checking, but this way you can at least have
an infinite number of records between each date.
You can use the NR variable to remove the first newline.

--
Stein Arne



Tue, 08 Mar 2005 19:22:02 GMT  
 log file processing - awk or sed


Quote:

> > Hi,
> > I have a little problem with logfile from my application.

> > My log looks like that :

> > 02/09/17
> > 12:00
> > 12
> > 34
> > 45
> > 67
> > 02/09/17
> > 12:10
> > 11
> > 32
> > 62
> > 69
> > ....

> > I need a log in that format :
> > 02/09/17|12:00|11|32|62
> > 02/09/17|12:10|14|38|66
> > I could be pipe or any other delimiter.
> > So, how can I do this using sed ? or awk ?

> awk 'BEGIN{d="|"}
> { if ( $0 ~ /\//) { printf "\n"$0}else{printf d$0}}
> END{print}' fil

> Plain and simple.
> You can of course add some checking, but this way you can at least have
> an infinite number of records between each date.
> You can use the NR variable to remove the first newline.

> --
> Stein Arne

Some reductionist notes...

Any 'true' action that consists of {if(c)s} should be reduced to c{s}.  Any
'true' action that consists of {if(c)t; else f} can be reduced to c{t;
next}{f}, but this may be less efficient.

I would consider using AWK's built-in OFS and ORS, rather than my own
variable ('d') and constant ("\n").  In one-liners, variable initialization
can be moved to the command line, too.

Any match-operator comparisons against a regular expresion, $0~//, can be
reduced to // (because that's just how stand-alone RE's in expressions
behave).

Of course, {s} reduces to s.

A warning...

Using printf to print data from the format string is dangerous (e.g. any
percent signs (%) in the data), so 'printf "%s", str' is generally
preferred.

Some errors...

'print', even in and END action, does 'print $0', using the last $0 read.
This is why your last field was "6969" in your testing (You did test it,
didn't you?).

Your output will always begin with either "\n" (a blank line) or "|".

Using these facts, I get:

    awk -v OFS="|"
    '{ if (/\//) printf ORS"%s",$0; else printf OFS"%s",$0 }
    END { print "" }' fil

From here, I would note the repeated printf construct, and use a format
string variable instead:

    { if (/\//) fmt = ORS"%s"; else fmt = OFS"%s"; printf fmt,$0 }

Then I would apply the '?:' operator and the %s similarity.

    { fmt = (/\// ? ORS : OFS) "%s"; printf fmt,$0 }

Finally, I would reduce out the variable fmt.

    { printf (/\// ? ORS : OFS) "%s",$0 }

So I end up with:
    awk -v OFS="|"
    '{ printf (/\// ? ORS : OFS) "%s",$0 } END { print "" }'
    fil

It being me, since I don't get anything out of using ORS and OFS, I would
just use constants.

    awk '{ printf (/\// ? "\n" : "|") "%s",$0 } END { print "" }' fil

I think this is simpler in structure, and easier to understand and read than
an if() statement with multiple 'print's.

It also clearly exposes the fact that if anything is printed, it will start
with a blank line or a pipe, based on whether the fil is empty, or whether
the first line has a slash or not.

You might ant to pipe this input through
    awk "NR>1"
to kill the leasing blank line.

    - Dan



Wed, 09 Mar 2005 00:43:03 GMT  
 log file processing - awk or sed

Quote:


>> awk 'BEGIN{d="|"}
>> { if ( $0 ~ /\//) { printf "\n"$0}else{printf d$0}}
>> END{print}' fil

> Some reductionist notes...

<snip>
> Some errors...

> 'print', even in and END action, does 'print $0', using the last $0 read.
> This is why your last field was "6969" in your testing (You did test it,
> didn't you?).

Yes, I did test it, and it did not print anything but a newline on HP-UX
11i (ksh88) with /usr/bin/awk.
I see now that you are right, at least in gawk 3.1.0.

<snip>

Quote:
> It being me, since I don't get anything out of using ORS and OFS, I would
> just use constants.

>     awk '{ printf (/\// ? "\n" : "|") "%s",$0 } END { print "" }' fil

This was fun. You dissected my small program and produced a much better
result (I'm also a reductionist).

To my defence: I feel I'm just starting with awk and I have to free my
self from thinking regular shell scripting.

I really learnt a lot from this, especially since you gave a step-by-step
reduction/transformation.

--
Stein Arne



Sat, 12 Mar 2005 22:48:36 GMT  
 log file processing - awk or sed



Quote:


> >> awk 'BEGIN{d="|"}
> >> { if ( $0 ~ /\//) { printf "\n"$0}else{printf d$0}}
> >> END{print}' fil

> > Some reductionist notes...

> <snip>
> > Some errors...

> > 'print', even in and END action, does 'print $0', using the last $0
read.
> > This is why your last field was "6969" in your testing (You did test it,
> > didn't you?).

> Yes, I did test it, and it did not print anything but a newline on HP-UX
> 11i (ksh88) with /usr/bin/awk.
> I see now that you are right, at least in gawk 3.1.0.

> <snip>
> > It being me, since I don't get anything out of using ORS and OFS, I
would
> > just use constants.

> >     awk '{ printf (/\// ? "\n" : "|") "%s",$0 } END { print "" }' fil

> This was fun. You dissected my small program and produced a much better
> result (I'm also a reductionist).

> To my defence: I feel I'm just starting with awk and I have to free my
> self from thinking regular shell scripting.

> I really learnt a lot from this, especially since you gave a step-by-step
> reduction/transformation.

> --
> Stein Arne

Hi there -

Quote:
> > 'print', even in and END action, does 'print $0', using the last $0
read.
> > This is why your last field was "6969" in your testing (You did test it,
> > didn't you?).

> Yes, I did test it...

 Hooray!  I got so tired of dealing with code that wasn't even syntactically
correct (in other NGs) that I always test my code before I post.  (Of
course, that makes the rare off-the-cuff answer horribly embarrasing if it's
wrong.)

Quote:
>..., and it did not print anything but a newline on HP-UX
> 11i (ksh88) with /usr/bin/awk.
> I see now that you are right, at least in gawk 3.1.0.

A trailing blank line in the data could do this.  If there is none, then I
have been, up until now, unfamiliar with any AWK implementation that either
(1) did not accept "print" as "print $0", or (2) did set $0 to blank in the
END action.  But now I am.

Quote:
> To my defence: I feel I'm just starting with awk...

There are a lot of dark corners, and a lot of shifting about that can be
done.  Most of it past the basic syntax I've picked up directly here, from
these great people, some of whom write the implementations, themselves--I
have not had too many of my own original "a ha!"s from reading the man pages
or books.  The trick is trying to remember them all at once!  And it is like
Mah Jong: if you have two ways to pick up your tiles, you may pick the pair
that will hide a tile needed later, forever.

Quote:
> and I have to free myself from thinking regular shell scripting.

But eventually it will become second nature.  I have to free myself from the
horrors of C++, LotusScript, C#, Java, Visual Basic, ECMA-Script, Pascal
dialects, VB Scripting Edition, T-SQL....every day, it seems like.  When I
sit down at the keyboard, I'n not even sure how to quote a string anymore :)

I'm glad it helped.

    - Dan



Sun, 13 Mar 2005 11:44:48 GMT  
 
 [ 14 post ] 

 Relevant Pages 

1. e-mail processing awk/sed

2. Using awk And/Or sed To Edit Fourth Field Of File

3. How to get info from a file with sed/awk/perl

4. deleting records in a file using AWK, SED, or kshell

5. Getting awk (or sed or anything else) to put single quotes into file

6. sed/awk script to format a file??

7. Processing a log file and Duplicate Lines

8. Expect: How to log different process to different files

9. AWK File processing

10. Processing Configuration Files: pget.awk

11. awk output into file during processing

12. awk process in awk ??

 

 
Powered by phpBB® Forum Software