<2016-01-14>

Anthony did this for the MathWONC project (see here).

Here is the code from cleanup_awk_linux.sh:

# cleanup_awk.sh
# bash script to run awk commands
# For removing lines from spm table .csv files
# Removes lines with "Unidentified" or any other simple criterion
#
# Use: $ bash cleanup_awk.sh spm_table_file_name.csv
#
# 2016.01.14 by adc
 
# Remove unwanted columns
#
# $1 is the variable that holds the first input argument (the file name in this case)
# Save the output to a temp file that will be deleted at the end of this script
awk -F, 'BEGIN{OFS=","} {print $5,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18;}' $1 > temp.csv;
 
# Middle of awk script includes pattern matching criteria for including and excluding certain lines
# Output to a new file name, which is like input file name but ending in "_CLEANED.csv" instead of just ".csv"
awk -F, 'BEGIN{OFS=","}  $1 > 5 && $10 !~ /Unidentified/ && $8 !~/NA/ {print}' temp.csv > ${1/.csv/_CLEANED.csv};
 
# Remove the temporary file
rm temp.csv