Shell vars into Awk executable script not so easy 
Author Message
 Shell vars into Awk executable script not so easy

I've been stumbling over attempts to pass variables into an awk
executable script all day.

Passing variables into command line awk.. no problem:
   awk '/^something/{print $0,var1}' var1=somethingelse FILE
   output: something somethingelse

Into an awk script, again, no problem:
   awk -f scriptfile var1=somethingelse FILE

But none of this works on an executable awk script.  Or so it seems.
Maybe a quoting technique or something else? That makes it work.

An executable awk script named one.awk:
#!/bin/awk -f
/^something/{print $0,var1}

How to establish `var1' from the command line



Tue, 14 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy

Quote:

> Into an awk script, again, no problem:
>    awk -f scriptfile var1=somethingelse FILE

> But none of this works on an executable awk script.  Or so it seems.
> Maybe a quoting technique or something else? That makes it work.

> An executable awk script named one.awk:
> #!/bin/awk -f
> /^something/{print $0,var1}

It should (and does here) work exactly like you probably tried it:

./one.awk var1=somethingelse FILE
^ be sure not to use another "one.awk" along your PATH

The "var1=somethingelse" idiom sets var1 only after the BEGIN block,
before the FILE. You don't have a BEGIN block in your example, so
nothing can be wrong (but maybe you posted an abridged version of the
script you actually tried).

Are you sure your /bin/awk is the same as the awk in your PATH (try
"which awk" or "type awk" to find out)?

I use "#! /usr/local/bin/gawk -f" anywhere, so I'm sure I don't get the
oldest awk that the vendor could find somewhere in their cellar.

Regards...
                Michael



Tue, 14 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy
Try this:  Prior to executine one.awk

export var="what you want"
one.awk

then change the something line to
/^something/{print $0,ENVIRON["var1"]}

Just a thought



Quote:
> But none of this works on an executable awk script.  Or so it seems.
> Maybe a quoting technique or something else? That makes it work.

> An executable awk script named one.awk:
> #!/bin/awk -f
> /^something/{print $0,var1}

> How to establish `var1' from the command line

Sent via Deja.com http://www.deja.com/
Before you buy.


Tue, 14 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy

Quote:


> > Into an awk script, again, no problem:
> >    awk -f scriptfile var1=somethingelse FILE

> > But none of this works on an executable awk script.  Or so it seems.
> > Maybe a quoting technique or something else? That makes it work.

> > An executable awk script named one.awk:
> > #!/bin/awk -f
> > /^something/{print $0,var1}

> It should (and does here) work exactly like you probably tried it:

> ./one.awk var1=somethingelse FILE
> ^ be sure not to use another "one.awk" along your PATH

Now that I've complained about it, it works here also.  Oh well.

Quote:

> The "var1=somethingelse" idiom sets var1 only after the BEGIN block,
> before the FILE. You don't have a BEGIN block in your example, so
> nothing can be wrong (but maybe you posted an abridged version of the
> script you actually tried).

I did try all sorts of experiments and may have used and example with a
BEGIN block somewhere in the mix.  Maybe where I got off on the wrong foot.

Quote:

> Are you sure your /bin/awk is the same as the awk in your PATH (try
> "which awk" or "type awk" to find out)?

The only awk like program on my machine is gawk-3.0.4  which is
symlinked to /bin/awk.  It must have been something sloppy in my
experiments or else the BEGIN block factor.

Working now so thanks for the reinforcement.



Tue, 14 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy

Quote:

> Try this:  Prior to executine one.awk

> export var="what you want"
> one.awk

> then change the something line to
> /^something/{print $0,ENVIRON["var1"]}

> Just a thought

Doesn't work for me but seems like it should

cat FILE:
something
not something

cat one.awk:
#!/bin/awk -f
/^something/{print $0, ENVIRON[VAR1]}

call .. $ VAR1=somethingelse ;export VAR1

$ env|grep VAR1:
VAR1=somethingelse

$ ./one.awk FILE
something



Tue, 14 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy


Quote:

>> Try this:  Prior to executine one.awk

>> export var="what you want"
>> one.awk

>> then change the something line to
>> /^something/{print $0,ENVIRON["var1"]}

>> Just a thought

>Doesn't work for me but seems like it should

>cat FILE:
>something
>not something

>cat one.awk:
>#!/bin/awk -f
>/^something/{print $0, ENVIRON[VAR1]}

You didn't follow directions, try:

/^something/{print $0, ENVIRON["VAR1"]}

That uses the string "VAR1" and not the unassigned awk variable VAR1

Chuck Demas
Needham, Mass.

Quote:

>call .. $ VAR1=somethingelse ;export VAR1

>$ env|grep VAR1:
>VAR1=somethingelse

>$ ./one.awk FILE
>something

--
  Eat Healthy    |   _ _   | Nothing would be done at all,

  Die Anyway     |    v    | That no one could find fault with it.



Wed, 15 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy

[...]

Quote:
> >Doesn't work for me but seems like it should

> >cat FILE:
> >something
> >not something

> >cat one.awk:
> >#!/bin/awk -f
> >/^something/{print $0, ENVIRON[VAR1]}

> You didn't follow directions, try:

Always my down fall...

Quote:

> /^something/{print $0, ENVIRON["VAR1"]}

> That uses the string "VAR1" and not the unassigned awk variable VAR1

Yup..

But this all reminds me of one reason why I posted about this variable
passing business.

This seems like a little harder nut.

Can we pass variables from the command line to an executable awk
script, that are just positions on the command line.  $1 $2 etc, in
the same way you can a shell script?

I have a program (search.awk) that people here helped me put together.
It is  a search program designed  to search mail or  usenet posts.  It
requires  3 changeable  regular expressions  and  a list  of files  to
search in order to be maximum usefull.

The regular expressions and file list may change with each usage.
The first two RE are found in the headers, the third in the body.

I'm using it thus far by setting the RE and the list of files on the
command line like this.

search.awk "RE1=^Subject:.*dired" "RE2=^Message-ID: " "RE3=dired-x" \
`find Mail/emacs -name '[0-9]*'`

Where all that is on one line.

Can that same stuff be passed to awk by making awk take note of the
positions?

Like:
search.awk "^Subject:.*dired" "Message-ID: " "diredx"  
and inside the program you would use something like:

/"$1"/  or RE1 = "$1"   or similar.

My experiments on this have failed so far.



Wed, 15 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy

Quote:



> > search.awk "RE1=^Subject:.*dired" "RE2=^Message-ID: " "RE3=dired-x" \
> > `find Mail/emacs -name '[0-9]*'`

> > Where all that is on one line.

> > Can that same stuff be passed to awk by making awk take note of the
> > positions?

> > Like:
> > search.awk "^Subject:.*dired" "Message-ID: " "diredx"
> > and inside the program you would use something like:

> > /"$1"/  or RE1 = "$1"   or similar.

> > My experiments on this have failed so far.

> Same concept as before

> use ENVIRON["$1"] (I really haven't tried it, but that is what I would
> try.)

> You might have to put this in a BEGIN statement to capture the value,
> ie: BEGIN{RE1 = ENVIRON["$1"] }

I can't get anything to go using that approach.  I tried with and
without a BEGIN block:

cat test.awk
#!/bin/awk -f
BEGIN{
     RE1 = ENVIRON["$1"]
         }
/WORD/{print $0,RE1}

cat FILE
WORD

./test.awk "something"  FILE

./test.awk:3: fatal: cannot open file `something' for reading (No such
file or directory)

I guess the best way is to use the tried and true way of putting the
awk script inside a shell script.  Then the positional argument
operators $1 $2 etc can be used in the normal shell script way.

Another way would be to have the awk script be interactive to get the
RE setup in place, but I'm a long way from knowing how to do that.

Quote:
> Also, use single quotes when using a wildcard (*).  It may save you some
> problem, and affect where the substitution occurs

Good clue ... thanks


Wed, 15 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy


Quote:

> search.awk "RE1=^Subject:.*dired" "RE2=^Message-ID: " "RE3=dired-x" \
> `find Mail/emacs -name '[0-9]*'`

> Where all that is on one line.

> Can that same stuff be passed to awk by making awk take note of the
> positions?

> Like:
> search.awk "^Subject:.*dired" "Message-ID: " "diredx"
> and inside the program you would use something like:

> /"$1"/  or RE1 = "$1"   or similar.

> My experiments on this have failed so far.

Same concept as before

use ENVIRON["$1"] (I really haven't tried it, but that is what I would
try.)

You might have to put this in a BEGIN statement to capture the value,
ie: BEGIN{RE1 = ENVIRON["$1"] }

Also, use single quotes when using a wildcard (*).  It may save you some
problem, and affect where the substitution occurs

Sent via Deja.com http://www.deja.com/
Before you buy.



Thu, 16 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy


Quote:

>[...]

>> >Doesn't work for me but seems like it should

>> >cat FILE:
>> >something
>> >not something

>> >cat one.awk:
>> >#!/bin/awk -f
>> >/^something/{print $0, ENVIRON[VAR1]}

>> You didn't follow directions, try:

>Always my down fall...

>> /^something/{print $0, ENVIRON["VAR1"]}

>> That uses the string "VAR1" and not the unassigned awk variable VAR1

>Yup..

>But this all reminds me of one reason why I posted about this variable
>passing business.

>This seems like a little harder nut.

>Can we pass variables from the command line to an executable awk
>script, that are just positions on the command line.  $1 $2 etc, in
>the same way you can a shell script?

>I have a program (search.awk) that people here helped me put together.
>It is  a search program designed  to search mail or  usenet posts.  It
>requires  3 changeable  regular expressions  and  a list  of files  to
>search in order to be maximum usefull.

>The regular expressions and file list may change with each usage.
>The first two RE are found in the headers, the third in the body.

>I'm using it thus far by setting the RE and the list of files on the
>command line like this.

>search.awk "RE1=^Subject:.*dired" "RE2=^Message-ID: " "RE3=dired-x" \
>`find Mail/emacs -name '[0-9]*'`

>Where all that is on one line.

>Can that same stuff be passed to awk by making awk take note of the
>positions?

>Like:
>search.awk "^Subject:.*dired" "Message-ID: " "diredx"  
>and inside the program you would use something like:

>/"$1"/  or RE1 = "$1"   or similar.

>My experiments on this have failed so far.

You can take positional values from the argument list which is contained
in ARGV. Here's a demonstration program.

bash-2.03$ cat atest.awk
BEGIN {

  #copy the first three arguments to RE1, RE2, RE3
  RE1=ARGV[1]
  RE2=ARGV[2]
  RE3=ARGV[3]
  print "RE1=",RE1
  print "RE2=",RE2
  print "RE3=",RE3

  #now remove the first 3 arguments
  #so that they will not be treated as input files
  for (i=1; i<ARGC; i++) ARGV[i]=ARGV[i+3]
  ARGC-=3

Quote:
}

NR==1 {print}
bash-2.03$

I have a data file like this :-

bash-2.03$ cat FILE
first line in FILE
second line in FILE
bash-2.03$

I define the three arguments like this :-

bash-2.03$ RE1='^Subject:.*dired'
bash-2.03$ RE2='^Message-ID: '
bash-2.03$ RE3='dired-x'
bash-2.03$ echo $RE1,$RE2,$RE3
^Subject:.*dired,^Message-ID: ,dired-x
bash-2.03$

When I run the script I get this :-

bash-2.03$ gawk -f atest.awk $RE1 $RE2 $RE3 FILE
RE1= ^Subject:.*dired
RE2= ^Message-ID:
RE3= dired-x
first line in FILE
bash-2.03$

The results are exactly the same if I use Brian Kernighan's awk95
instead of gawk.

Note that the shell variables don't have to have the same names as the
awk variables.

You don't have to take the intermediate step of defining shell variables
RE1, RE2, RE3. You could put use literal values in the argument list
instead but I think this is neater. Either way it is important to use
single quotes to prevent expansion of wildcards by the shell.

Here's another alternative.

gawk -v RE1=$RE1 -v RE2=$RE2 -v RE3=$RE3 -f your_program list_of_files

Hope this helps
--
Alan Linton



Thu, 16 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy


[snip]

Quote:

>You can take positional values from the argument list which is contained
>in ARGV. Here's a demonstration program.

>bash-2.03$ cat atest.awk
>BEGIN {

>  #copy the first three arguments to RE1, RE2, RE3
>  RE1=ARGV[1]
>  RE2=ARGV[2]
>  RE3=ARGV[3]
>  print "RE1=",RE1
>  print "RE2=",RE2
>  print "RE3=",RE3

>  #now remove the first 3 arguments
>  #so that they will not be treated as input files
>  for (i=1; i<ARGC; i++) ARGV[i]=ARGV[i+3]
>  ARGC-=3

>}

>NR==1 {print}
>bash-2.03$

>I have a data file like this :-

>bash-2.03$ cat FILE
>first line in FILE
>second line in FILE
>bash-2.03$

>I define the three arguments like this :-

>bash-2.03$ RE1='^Subject:.*dired'
>bash-2.03$ RE2='^Message-ID: '
>bash-2.03$ RE3='dired-x'
>bash-2.03$ echo $RE1,$RE2,$RE3
>^Subject:.*dired,^Message-ID: ,dired-x
>bash-2.03$

>When I run the script I get this :-

>bash-2.03$ gawk -f atest.awk $RE1 $RE2 $RE3 FILE
>RE1= ^Subject:.*dired
>RE2= ^Message-ID:
>RE3= dired-x
>first line in FILE
>bash-2.03$

>The results are exactly the same if I use Brian Kernighan's awk95
>instead of gawk.

>Note that the shell variables don't have to have the same names as the
>awk variables.

>You don't have to take the intermediate step of defining shell variables
>RE1, RE2, RE3. You could put use literal values in the argument list
>instead but I think this is neater. Either way it is important to use
>single quotes to prevent expansion of wildcards by the shell.

>Here's another alternative.

>gawk -v RE1=$RE1 -v RE2=$RE2 -v RE3=$RE3 -f your_program list_of_files

>Hope this helps

It occurred to me to check if the trailing space in RE2 is preserved. I
found that there can indeed be a problem.

I changed the program to this:-

bash-2.03$ cat atest.awk
BEGIN {

  #copy the first three arguments to RE1, RE2, RE3
  RE1=ARGV[1]
  RE2=ARGV[2]
  RE3=ARGV[3]
  print "RE1=#"RE1"#"
  print "RE2=#"RE2"#"
  print "RE3=#"RE3"#"

  #now remove the first 3 arguments
  #so that they will not be treated as input files
  for (i=1; i<ARGC; i++) ARGV[i]=ARGV[i+3]
  ARGC-=3

Quote:
}

NR==1 {print}
bash-2.03$

Here's a test run :-

bash-2.03$ awk95 -f atest.awk $RE1 $RE2 $RE3 FILE
RE1=#^Subject:.*dired#
RE2=#^Message-ID:#
RE3=#dired-x#
first line in FILE
bash-2.03$

Notice that the trailing space in RE2 is gone.

But if I do this:-

bash-2.03$ awk95 -f atest.awk "$RE1" "$RE2" "$RE3" FILE
RE1=#^Subject:.*dired#
RE2=#^Message-ID: #
RE3=#dired-x#
first line in FILE
bash-2.03$

then the trailing space is preserved

So it is important to put double quotes around argument variables to
protect trailing spaces. The same applies to the alternative method at
the end of my previous post.

Use single quotes for literal arguments.

I think I have sowed enough confusion for today, sorry.
--
Alan Linton



Thu, 16 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy
[how does one get at a command-line argument from an executable awk script?]

% Same concept as before
%
% use ENVIRON["$1"] (I really haven't tried it, but that is what I would
% try.)

This won't work. In general ENVIRON will only report environment variables,
and the shell's $1 is not an environment variable. The Bourne-style shells
distinguish between environment variables and shell variables. Environment
variables are stored in a data structure which is known to the operating
system and gets passed to child processes. Shell variables get put in
a data structure which is known only to the shell process -- they're
analagous to ordinary awk variables.

 b=howdy
sets a shell variable called b, but it doesn't export it to the shell's
environment.
 export b
does this export, making b part of tne ENVIRON array.

You can't export the positional parameters $1 et al, so these can never be
read through ENVIRON.

In this case, we're talking about an executable awk script, so there
is no shell involved in processing these command-line parameters, anyway.

% You might have to put this in a BEGIN statement to capture the value,
% ie: BEGIN{RE1 = ENVIRON["$1"] }

It doesn't really matter where you use ENVIRON. It's probably faster to
save the value once in a scalar variable, but the same value should be
returned no matter where you put it.
--

Patrick TJ McPhee
East York  Canada



Thu, 16 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy


Quote:

> You can take positional values from the argument list which is
contained
> in ARGV. Here's a demonstration program.

Thanks Alan - I forgot all about ARGV and ARGC when I (all too quickly)
replied.

Sent via Deja.com http://www.deja.com/
Before you buy.



Fri, 17 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy
 (...)
 >  #copy the first three arguments to RE1, RE2, RE3
 >  RE1=ARGV[1]
 >  RE2=ARGV[2]
 >  RE3=ARGV[3]

AFAIK for using ARGV and ARGC with awk you need either gawk or mawk.
They both support executable scripts beginning with:

   #!/path/to/[mg]awk -We

then you can pass command-line arguments to those scripts and having
them set ARGV/ARGC the way you would expect. If you do not have one of
those AWK implementations you have better to use the environment, as
suggested in a previous post.

bye     --carlo
--
Against software patents:
http://www.freepatents.org - http://no-patents.prosa.it



Fri, 24 Jan 2003 03:00:00 GMT  
 Shell vars into Awk executable script not so easy


%  (...)
%  >  #copy the first three arguments to RE1, RE2, RE3
%  >  RE1=ARGV[1]
%  >  RE2=ARGV[2]
%  >  RE3=ARGV[3]
%
% AFAIK for using ARGV and ARGC with awk you need either gawk or mawk.

That's certainly not true. Any awk implementation written since 1989
supports ARGC and ARGV.
--

Patrick TJ McPhee
East York  Canada



Fri, 24 Jan 2003 03:00:00 GMT  
 
 [ 19 post ]  Go to page: [1] [2]

 Relevant Pages 

1. executable awk scripts and shell var $(basename $0)

2. #!/bin/awk -f style and shell vars

3. passing vars to shell script

4. How to use shell vars in a script?

5. passing vars to awk script

6. Pretty easy (I think) shell script question

7. Calling executable from awk script

8. AWK-Script to set a shell variable (Win32) ?

9. Passing command line variables to AWK in shell script

10. using awk in a shell script...

11. trying to call awk from within a shell script

 

 
Powered by phpBB® Forum Software