How to print unique errors(with count) from Apache error logs

Tag Archives: one liner

How to print unique errors(with count) from Apache error logs

Published May 31, 2014 | In Unix/Server Stuff

After I moved to WordPress Multisite recently, I had to closely watch the Apache error log file for a couple of days to make sure that there weren’t any configuration mismatch.

Initially I used run the tail command to view the errors, but then I quickly realized that if there is a frequent error, then it becomes extremely difficult to find out other non frequent errors.

Command

After some fiddling with the awk and sort commands, I came up with the following one liner that prints the unique errors with their count.

Explanation

awk script is executed for each line in the error log file.

Field Separator

The -F directive is used to specify field separator. If you look at a single line in the apache error log file, you will notice that each field is enclosed by [ and ].

awk allows us to specify multiple field separators by enclosing them between []. That’s what we are specifying in -F[\[\]]

Filtering out lines that contain error

After splitting the fields, we need to filter out the lines that contain the term error in the fourth field. That’s what happens in the next part of the command. Basically the part $4 == "error"{print $7} prints the seventh field, which has the real error message if the fourth field is equal to the string “error”.

Sorting the error messages

The next step is to sort all the error messages. This is done by piping the output to the sort command.

Finding unique error messages and counting them

The next step is to find the unique error messages and then count them. This is done by piping the output to the uniq command. The -c flag does the counting.

Sorting the messages by frequency

The last step is to sort the messages again by frequency and print them. This is done by the last sort command. The -n flag sorts by numbers and the flag -r does the sorting by descending order.

Posted in Unix/Server Stuff | Tagged Apache, awk, command line, one liner, sort, uniq | Leave a comment

Rollback a commit in git, while maintaining history

Published Jul 8, 2013 | In Unix/Server Stuff

I use git for pretty much all my projects these days. When ever I do a commit, I make sure that it is atomic and has a proper commit message. One of the main reason why I keep the commits atomic is that, I can rollback the commit if needed.

Recently, I faced a situation, where in one of my projects, the feature I added a couple of commits back wasn’t working properly and I decided to completely get rid of it. Since I follow semantic versioning, I wanted to make sure that I maintain the different release of my project properly.

After a bit of digging up, I found the exact command in git to do that. Since the command was not that intuitive I thought of documenting it here, so that I know where to lookup when I face the situation again 😉

Problem

Let me explain problem properly. Let’s assume that I have the following list of commits in my history.

A -> B -> C -> D -> E

The head is at E now. The feature that I was talking about was introduced in commit C. Now I want to rollback the changes that I did in commit C, after E, but still maintain the history. In short I want the commits to look like

A -> B -> C -> D -> E -> C'

where C' is the opposite of C

Solution

I could have manually removed that changes I did in commit C. But being a fan of fancy one-liners, I was looking for a solution and found that there is a command in git which can do this for you.

All you need to do is to execute the following command.

git revert 'SHA_of_C'

It’s really fascinating to see, how the developers of git have thought of these different cases 🙂

Posted in Unix/Server Stuff | Tagged git, one liner | Leave a comment

Sum up all values of a column in a text file

Published Jul 4, 2013 | In Unix/Server Stuff

Recently, I had to sum up all integers of a column in a text file (similar to how you do in excel). After some digging up, I came up with a awk one liner to do it.

Following my tradition of documenting one liners, I am going to document this one as well 🙂

Input and output data

Here is the super simplified version of input data that I was using.

I wanted to find the sum of all values present in the 3rd column. So in the case, the output that I was expecting was 145

Command

Here is the awk one liner, which does this.

Explanation

awk script is executed for each line and the first part of the command creates a variable s that stores the sum of all values in the 3rd column.

When the end of file is reached, the second part of the command is executed, which just prints the value of the variable.

Field separator

If the columns are separated by a comma or by any other non whitespace character, then you have to just specify it by adding FS=',' to the above command.

The more I dig deeper into awk, the more I like it and it is really fascinating to see how much you can do with this tool.

I learned a lot about awk and hopefully this teaches something to you as well 😉

Posted in Unix/Server Stuff | Tagged awk, command line, one liner | 7 Comments

Count the number of empty lines in a file using grep

Published Mar 28, 2013 | In Unix/Server Stuff

Recently I had a need to count the number of empty lines in a text file. After some digging into the man pages of grep I was able to come up with a one liner which was able to do it.

Following my tradition of documenting one liners, I am going to document this one as well 🙂

Assumption

By empty line, I mean any line which either has no characters or has only whitespace (space, tab) characters.

Command

For the impatient in you, here is the actual command.

Explanation

-P '\S' – This selects all lines that have a non whitespace character
-c – Print the count of matching lines
-v – Select only the non-matching lines

So, we are first matching all lines that have a non whitespace character and then use -v option to ignore them and then -c option to print the count instead of the actual line.

If we wanted the count of all non-empty lines, then we just have to remove the -v option from the above command.

Hope this is helpful. Happy Grep’ing 😉

Posted in Unix/Server Stuff | Tagged command line, grep, one liner | 7 Comments

Remove duplicate lines based on a field

Published Mar 20, 2013 | In Unix/Server Stuff

Recently while working on formatting some data files for further processing, I had to remove duplicate lines from the file based on a particular field. After trying out cut and grep commands, I was finally able to solve it with a very concise awk command/script.

The command was so concise but still was packed with so much information and it helped me to learn more about the awk scripting language. I thought of writing about it here so that it is useful for others and also I know where to search for it, when I needed it 🙂

Feel free to use it in whatever way you want, if it solves your problem as well.

Input and output data

Let me first explain the input data I had and the output that I was expecting.

Consider a file which has the following lines. Each line has four fields.

Tom	CTO	32
Harry	Manager	45        -> Manager field is duplicate
Krish	CEO	50
Bob	Manager	49        -> Manager field is duplicate
Patrick	CFO	20

Now assume that we want to remove duplicate lines by comparing only the second field. We want the output to look like this.

Command

Get ready for the surprise. The actual command is just this.

Explanation

awk script execution and printing

awk script is executed for each line and if the result is true then the line is printed. If the result is false then the line is not printed.

Associative arrays

The awk language supports associate arrays, similar to the ones found in PHP. The script x[$2]++ fills up an associate array. The key used here is $2 which refers to the second field and x is the variable name. You can use any name for it.

The array is populated for every line. This is how the array would look like after each line.

# Before starting 
x = {}

# After line 1
x = {
	CTO => 1
    }

# After line 2
x = {
	CTO => 1
	Manager => 1
    }

# After line 3
x = {
	CTO => 1
	Manager => 1
	CEO => 1
    }

# After line 4
x = {
	CTO => 1
	Manager => 2
	CEO => 1
    }

# After line 5
x = {
	CTO => 1
	Manager => 2
	CEO => 1
	CFO => 1
    }

Conditional evaluation

The ! operator results in a boolean evaluation which determines whether a particular line should be passed on to the output (printed) or not.

When the field is not present in the array, then it results in a zero value which is false. The ! (not) operator evaluates it to non-zero, which results in a true value and the line is passed on to the output (printed). When a duplicate is found, the array returns a non-zero count, which is true, but the ! converts it to false and that line is not passed on to the output.

The expanded version of the above command would be

But what is the fun in using the expanded version 😉

Field separator

In the input file that I had, the fields were separated by whitespace, so I didn’t have to specify the field separators. But if you are using a non-whitespace field separators, then you can specify it by adding FS="," to the above command.

This one-liner actually thought me that awk supports a full programming language that can be used to create scripts and also increased my understanding of the way awk command works. Hopefully this teaches something for you as well 🙂

I know that this is already a concise version, but if you think that this can be improved, then do let me know.

Posted in Unix/Server Stuff | Tagged awk, command line, one liner | 5 Comments

Night Dreaming (by Sudar)

Tag Archives: one liner