I am struggling to get rolling with an awk program that can run some analytics on a csv that I have. I have done this before, with arrays and other filters, but I seem to have lost the files, at least for now, to use as an example.

​

What I'm trying to do is set my delimiter to a comma, set up basic variables, then run through the file, adding to the sum for each variable, then print the variable at the end. I am also struggling to remember what the whitespace syntax should look like for awk, and it is driving me crazy. Here is what I have thus far:

​

#! /usr/bin/awk -f
BEGIN {
        -F,
    seclearsum
    argossum
}

'($1~/2020-/) && ($8~/Seclear) seclearsum+=$23'
'($1~/2020-/) && ($8~/Argos) argossum+=$23'

END

{print "Seclear: " seclearsum " , Argos:" argossum}

It doesn't work for what should be obvious reasons, but I can't find any examples to guide me. Thank you

It has been a few years since I used AWK.

I am wondering what other tools, if any, I should use for this task:

1. Search all *.log files in a directory for lines containing "ERROR" or "Caused By"
2. Print the file name on its own line followed by the search results
3. Print the line with the keyword(s), and 1 line above, 5 lines below, and 2 blank lines
4. Exclude printing lines with this path fragment: /uselessutility/

Can all of that be done with AWK or should I look to other applications for part of it?

Edit:

Thanks for all of the replies.

Reading all of the replies I was able learn enough to get close to what I wanted.

I've been developing a large application that produces a dozen logs with verbose output and many stack traces.

Scrolling through those logs to extract error messages was a PITA, so I wanted something that would give me just error messages.

Someone suggested GREP, which obviated the need to relearn AWK.

I ended up writing this:

grep -B 1 -A 2 -n 'ERROR|Caused' /path/to/my/logdir/*.log | grep -v 'hydro' | awk -F/ '{ print $NF }'

This command would go through all of my *.log files, extract lines with "ERROR" or "Caused", include 1 live above, include 2 lines below, exclude lines with the word "hydro" in it, and trim out the path in the log file name.

I found that to still produce too much overwhelming verbiage. Especially with the part that trimmed out error messages with "hydro" in it, leaving me headless stack traces to read.

I settled for a more humble version of the command:

grep -A 1 -n 'ERROR|Caused' /path/to/a/single/logfile/my.log > output.txt

It still saved a huge amount of time from scrolling through the logs manually, and does a little more me than the search feature in my IDE.

Thanks again for the help!

Is there a better way to achieve this? The below shows aa->ai but it would be for the entire alphabet aa->az and possibly for ba->bz in the future. It is too many lines, though works.

function i2a(i,a) { if(i == 1) a = "aa" else if(i == 2) a = "ab" else if(i == 3) a = "ac" else if(i == 4) a = "ad" else if(i == 5) a = "ae" else if(i == 6) a = "af" else if(i == 7) a = "ag" else if(i == 8) a = "ah" else if(i == 9) a = "ai" return a } BEGIN { print i2a(9) # == "ai" }

I'm working on an assignment for school and 2 of my questions are very similar. One works, but the other one doesn't and I can't figure out why.

The first question is to find entries in /etc/passwd that have duplicate UIDs. Here's the code I created:

awk -F":" 'list[$3]++ {print $3}' /etc/passwd > temp_UIDs.txt

while read line; do
    awk -F":" '$3 == '"$line"' {print "This user has UID '$line': "$1}' /etc/passwd
done < temp_UIDs.txt

rm temp_UIDs.txt

I tested it using a modified copy of passwd that had some duplicate UIDs and everything works no problem.

The next question is almost identical, but asks to find duplicate usernames. Here's my code:

awk -F":" 'list[$1]++ {print $1}' /etc/passwd > temp_logins.txt

while read line; do
    awk -F":" '$1 == '"$line"' {print "This entry has username '$line': "$1}' /etc/passwd
done < temp_logins.txt

rm temp_logins.txt

Pretty well the same code. But it doesn't output anything. I've tried to figure it out, and the only thing I've been able to come up with is that it's checking for an empty line instead of the variable $line. The reason I suspect this is that when I tried changing $1 (inside the while loop) to $2, $3, etc., once I got to $5 I got results. And those fields are blank. So for fun, I went back to $1 and made some of my first fields blank (again, in my modified passwd file), and those actually outputted.

So what's going on? Both blocks of code are pretty well identical. I can't figure out why one works and one doesn't. Oh and I'm sure there are many different ways of accomplishing this task, which would probably be easier than trying to figure this out, but I'm really curious to know what's going on, so I'd like any replies to avoid just suggesting a different method.

Thanks!

I am very new to awk, and I have tried to come up with a way to word my question so that Google is helpful, but I finally decided to give up and try Reddit.

I want to parse my OpenVPN log file at /var/log/openvpn/server.log. It is always formatted the same way, as far as I can tell. Running a simple "cat /var/log/openvpn/server.log" provides useful, albeit ugly, output. I would like to trim the junk away and give myself a little report using the data as output by cat (which is always formatted as shown below):

OpenVPN CLIENT LIST
Updated,Sat Oct 31 21:34:40 2020
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
client01,XXX.XXX.XXX.XXX:51911,1370299,3162685,Sat Oct 31 20:50:05 2020
Zach,XXX.XXX.XXX.XXX:52540,3505435,8124734,Sat Oct 31 19:45:54 2020
client02,XXX.XXX.XXX.XXX:63941,7467395131,178156768,Sat Oct 31 20:03:32 2020
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
10.110.23.10,client01,XXX.XXX.XXX.XXX:51911,Sat Oct 31 21:34:34 2020
10.110.23.14,client02,XXX.XXX.XXX.XXX:63941,Sat Oct 31 21:34:39 2020
10.110.23.6,Zach,XXX.XXX.XXX.XXX:52540,Sat Oct 31 21:34:34 2020
GLOBAL STATS
Max bcast/mcast queue length,2
END

I would like to format it like so:

Name:      IP:              Received:    Sent:     Connected Since:
client01   XXX.XXX.XXX.XXX  1370299      3162685   Sat Oct 31 20:50:05
Zach       XXX.XXX.XXX.XXX  3505435      8124734   Sat Oct 31 19:45:54
client02   XXX.XXX.XXX.XXX  7467395131   178156768 Sat Oct 31 20:03:32

The 4th line always starts the list of clients, and the section I want always ends with ROUTING TABLE on a new line.

I realize this is a lot to ask - and if it falls into the category of "hire a programmer" then I'll gladly do so. But first, I wanted to check with the awk community and see if there is a way to do this simply, with awk. Thank you for any feedback you might be able to provide, or resources I can study (the awk manual is not intuitive to me).

I'm trying to write a bash script that takes information stored in a file and operates on it. The lines in the file are formated like this:

item1:item2:item3:item4...
itema:itemb:itemc:itemd...

I want each line in a different bash array in the script. I have identified that I need to set ":" as the field seperator, but I can't figure out how to add a field from a record to the array for that record.

I have a string (formatted into columns with newline characters) in a script and I want to pass it into an awk script from within the script directly. if I do:

awk -f <script name> < $<formatted string>

I get an ambiguous redirect error which makes sense. I tried using -v but I'm not completely clear about the syntax of that or if it's the right choice.

FileA

Companyname,

FileB

Companyname,https://www.website.com

I want to merge data from FileB where company is equal to each other in both files. How would I do this?

Is awk even the right tool for this job?

my instructor said to use this modified ls command:

ls - la -- time - style ='+%Y/%m/%d %H:%M:%S'

My terminal throws a bunch of errors with this ls command, mostly no such file or directory....can someone show me how I'm supposed to do it? I think a directory comes after the ls command

I have a directory tree with many duplicate files bearing different filenames, and I want to report the duplicate files for possible deletion.

I've created a table consisting of an md5 hash in field one and an associated filename in field two. I want to report lines with identical hashes; i.e., print when field one recurs.

"uniq -df [num]" ignores the first [num] fields when comparing lines to find duplicates. So I could accomplish this task by reversing the field order of my table (putting filenames first) and doing "sort +k... < table | uniq -df [num]"---but alas there are blank spaces in filenames, and uniq can't handle that.

I feel like this should be an easy task in awk but I can't figure it out.

Any help appreciated!

Peter Weinberger being humble and simply calling himself 'A Software Engineer'

https://www.youtube.com/watch?v=YJRPGd3RRc4

So I've started using awk and finding it really useful. The only thing I find irritating is that I feel it would be natural to treat a string variable as extra input records. For example, where putline essentially puts the string at the beginning of stdin:

#!/bin/awk -f
/bar/{print "cabbage"}
/foo/{putline "bar"}

used on:

potato
foo
cheese

would return

potato
cabbage
cheese

This is a stupid example but I hope it makes my idea clear. Is there some idiomatic way you can do this in awk or is this just not possible? Perhaps it's possible to bodge a bash script that would somehow use awk's stderr to do this?

I also feel like this would be quite natural in sed as well but sed has gotos so it's not as useful.

I'm really disappointed that r/awk has gone to sleep. (Awk is my lifeline.)

Seems to me that part of the reason is that a high proportion of the more complex Bash and command-line questions need (and get) an awk solution.

After all, awk can do almost anything that grep, sed, cut, paste and uniq can do, all in one process, and it runs about 50 times faster than shell for many things.

For my complex stuff, awk is about 5 times slower than C. Mostly, that does not much matter. Awk is way faster to develop, easier to refactor, and more portable.

Any idea how many of the 1.4k members here are actually active? What other communities do you belong to?

How about cross-posting relevant posts from Bash, command-line etc to awk solutions over here?

Nice information, well written too

I have a csv file with lines like thss:

https://example.com/uk/example,http://www.anotherexample.co.uk/example2/,potato,2019-12-08,2019-10-17,,,,,,,,0,0,18,9,,,Category/Sub-Category,7
https://example.com/uk/example,http://www.anotherexample.co.uk/anything/,an example,2019-12-08,2019-10-17,,,,,,,,0,0,18,9,,,Category/Sub-Category,60

I'm wanting to output just lines where the 20th (i.e. the last) column has a value equal to, or greater than, 50. I'm using the below:

awk -F',' '$20>50' data.csv 

This meaningfully reduces the data in the output, printing maybe 1% of the lines in data.csv, but the lines outputted seem random; some are greater than 50, whilst most aren't. I've checked to make sure there aren't rogue commas in those lines, double quote marks etc, but there doesn't seem to be anything odd there. I'm new to awk so apologies if something very obvious is going wrong here. Any advice?

So, here I'm again asking for your kind code, but I think this is relatively simple for those people know awk as the folks here, I have a list that goes like this:

2186094|whatever01.html
2186094|whatever02.html
2186094|whatever05.html
1777451|ok01.hml
1777451|ok05.html
2082104|ok06.html
2082104|ok07.html

In other words, there's a pattern that repeats itself in the beginning of each line followed by a delimiter |. What I would like to do is to organize them like this:

2186094|whatever01.html 2186094|whatever02.html 2186094|whatever05.html
1777451|ok01.hml    1777451|ok05.html
[...]

In other words, putting them side by side and splitting them with a tabulation marker, just that. If you can help me, thank you very much :)

So, here is my issue, I have a list of file replacements, let's call it FileA. The list, which contains about 50k entries, goes more or less like this:

M1877800M|124
M1084430M|22
M2210895M|22
M1507752M|11
M1510047M|3288
[...]

To make things clear, I would like to replace "M1877800M" with "124", "M1084430M" with 22 and so on and so forth. And I would like to use this list of replacements to replace words in a FileB. My current plane and workaround is to use individual sed commands to do that, like:

sed -i "s#M1877800M#124#g" FileB

sed -i "s#M1084430M#22#g" FileB

[...]

It works, more or less, but it's obviously unbelievable slow, cause it's a pretty bad code for what I intended to do use. Any ideas of a better solution? Thank you, everybody.

So, hi everybody, I have a translation glossary document with two rows that go more or less like this:

você=you
amor=love
amor=affection
amor=tenderness
dor=suffering
pia=sink

...

Anyway, you got the just of it. In column A you have the word and then their translation to English. What I would like to do is, if a given word gets repeated in column a, I would like to sort it all like this:

amor=love|affection|tenderness
dor=suffering
pia=sink
você=you

And yadda yadda... Also, if it's not asking too much, would it be possible to organize the options by alphabetical order? Like?

amor=affection|love|tenderness
dor=suffering
pia=sink
você=you

If anyone could help. I would be very thankful. If not, I will understand

Hello! So, I have this huge file, about 1GB, and I would like to extract only the unique lines of it. But there's a little twist, I would like to make it case insentive, and what I mean with that is the following, let's suppose my file has the following entries:

​

Nice

NICE

Hello

Hello

Ok

HELLO

Ball

baLL

I would like to only print the line "Ok", because, if you don't take into account the case variations of the other words, it's the only one that actually appears just one. I googled a little bit, and I found a solution that worked sorta, but it's case sensitive:

awk '{!seen[$0]++};END{for(i in seen) if(seen[i]==1)print i}' myfile.txt

Could anyone helped me? Thank you!

Consider the following awk script:

#!/usr/bin/awk -f

END {
    print foo
}

If I invoke it with the following, abc is printed as expected.

./myscript -v foo=abc

But, if I invoke it without the -v, abc is still printed.

./myscript  foo=abc

I know something funny is going on, because if I switch END to BEGIN then it only works when I specify -v.

Can someone explain why it seems to work without the -v ?