Thursday, April 28, 2011

Why doesn't my awk one-liner work when I call it from Perl?

Dear all,

I have no problem using the following command of AWK as a stand alone command, without any error:

$ awk '$9 != "NTM" && $9 != ""' myfile.txt  | less -Sn

But when I apply them inside Perl's script for qsub (i.e. running job in linux cluster) command, like this:

use strict;
use Data::Dumper;
use Carp;
use File::Basename;

my $path = "/mypath/"; 
my @files = glob($path."*.txt");

foreach my $file ( @files ) {
print "$file\n";
    wy $base = basename($file,".fc-gn_".$type);
    my $nn = $path.$base.".out";

print "$file $nn\n";

    open PIPE, "| qsub" or die $!; 
    print PIPE <<EOF;
#!/bin/sh 
#PBS -N CLEAN
#PBS -l nodes=1:ppn=2 
#PBS -r n 

awk '$9 != "NTM" && $9 !=""' $file > $nn


EOF

}

It gave the following error

awk: cmd. line:1:  != "NTM" &&  !=""
awk: cmd. line:1:  ^ syntax error

What's the right way to do it?

From stackoverflow
  • Putting a "\" character before all of your "$9" variables will fix it. They're being parsed by Perl itself.

    The following script:

    print <<EOF;
    awk '$9 != "NTM" && $9 !=""' $file > $nn
    EOF
    

    outputs:

    awk ' != "NTM" &&  !=""'  >
    

    but:

    print <<EOF;
    awk '\$9 != "NTM" && \$9 !=""' \$file > \$nn
    EOF
    

    outputs:

    awk '$9 != "NTM" && $9 !=""' $file > $nn
    

    However, since you want $file and $nn to be interpreted by Perl, the actual line should be:

    awk '\$9 != "NTM" && \$9 !=""' $file > $nn
    
  • you can use the backslash solution, or make it more visually appealing by using:

    print <<'EOF'

    (and no backslash), also if you like Perl, you might want to give a shot at gearman, it's really cool.

    paxdiablo : Removing the semicolon doesn't have any effect on the behavior in my perl version (5.10) - it still puts in blanks for the $9 variables. What version are you running?
    paxdiablo : Sorry, strike that, I forgot the single quotes around the EOF. But now you have a different problem, the $file and $nn are *not* being interpreted by perl as they should be.
    Yann : ah right, I didn't look carefully enough, you have to backslash some of it then if you want to keep it in one block

0 comments:

Post a Comment