Sometime back I wrote about a simple awk script that I wrote which allowed me to find unique errors from Apache error log files.
After I wrote that script I found myself executing that script every day in the morning to figure out if there were any errors in my sites. After a couple of days I wrote another script to automatically parse the error log file and email me if there were any errors. As usual I thought of writing about it here so that it would be useful for someone else as well 🙂
Posted in Python, Unix/Server Stuff Tagged Apache, command line, Python 5 Comments
After I moved to WordPress Multisite recently, I had to closely watch the Apache error log file for a couple of days to make sure that there weren’t any configuration mismatch.
Initially I used run the tail command to view the errors, but then I quickly realized that if there is a frequent error, then it becomes extremely difficult to find out other non frequent errors.
After some fiddling with the awk and sort commands, I came up with the following one liner that prints the unique errors with their count.
awk script is executed for each line in the error log file.
The -F directive is used to specify field separator. If you look at a single line in the apache error log file, you will notice that each field is enclosed by [ and ].
awk allows us to specify multiple field separators by enclosing them between []. That’s what we are specifying in -F[\[\]]
After splitting the fields, we need to filter out the lines that contain the term error in the fourth field. That’s what happens in the next part of the command. Basically the part $4 == "error"{print $7} prints the seventh field, which has the real error message if the fourth field is equal to the string “error”.
The next step is to sort all the error messages. This is done by piping the output to the sort command.
The next step is to find the unique error messages and then count them. This is done by piping the output to the uniq command. The -c flag does the counting.
The last step is to sort the messages again by frequency and print them. This is done by the last sort command. The -n flag sorts by numbers and the flag -r does the sorting by descending order.
Posted in Unix/Server Stuff Tagged Apache, awk, command line, one liner, sort, uniq Leave a comment
Recently, I was checking the apache error logs of a project I was working on and found that it has lot of 404 errors for images. I used my apache error parsing script and found that most of the errors were because the production db was still having some references to images present in the stage machine.
I had to do a quick find and replace in the MySQL db and after a quick search came up with the following query, using the replace function.
UPDATE table SET field = REPLACE(field, 'stage-url', 'prod-url');
Since I was about to run this query on a prod db directly, I wanted to add another condition to replace only rows that contained the stage url as an additional precaution (of course after taking backup of the db 😉 ).
UPDATE table SET field = REPLACE(field, 'stage-url', 'prod-url') WHERE INSTR(field, 'stage-url') > 0;
Hope this query is useful for someone who also had a shock like me when they looked at the apache error logs 😉
Posted in Database Programming Tagged Apache, MySQL, Query, replace 3 Comments
I must confess that I am a stats freak. If you are a long time reader of my blog, then you would have known that by now yourself. 😉 This explains the reason why I want to preserve my Apache log files in spite of using a variety of stat services like Google Analytics, WordPress stats, statscounter, performancing metrics (before it was closed).
The default Apache configuration preserves the log files only for the last 10 days, but I wanted to permanently archive this files. After some searches in Google I came across an excellent program called Cronolog. Cronolog is a simple filter program which writes each log entry to a separate log file named after the filename format specified. You can use a variety of parameters like current date, time etc to define the filename template.
First we have to install cronolog, either by using aptitude or by downloading it from its download page. Then you have to change the log file name path in the virtual host file. (In Ubuntu Gusty, the virtual host files are situated in the path /etc/apache2/sites-enabled). I am using the following file format for this blog
# Custom log file locations
LogLevel warn
ErrorLog "|/usr/sbin/cronolog /path/to/logs/%Y/%m/%Y-%m-%d-sudarmuthu.com-error.log"
CustomLog "|/usr/sbin/cronolog /path/to/logs/%Y/%m/%Y-%m-%d-sudarmuthu.com-access.log" combined
which will store my log files in separate folders for each year and for each month, like the below hierarchy
/2007/12/2007-11-01-sudarmuthu.com-access.log
/2007/12/2007-11-02-sudarmuthu.com-access.log
......
/2008/01/2008-01-01-sudarmuthu.com-access.log
/2008/01/2008-01-02-sudarmuthu.com-access.log
......
You can use a variety of modifiers for the filename and I have documented some of them in the below table. You can get more information from its documentation.
| Specifier | Description |
|---|---|
| Time fields | |
| %H | hour (00..23) |
| %I | hour (01..12) |
| %p | the locale’s AM or PM indicator |
| %M | minute (00..59) |
| %S | second (00..61, which allows for leap seconds) |
| %X | the locale’s time representation (e.g.: “15:12:47”) |
| %Z | time zone (e.g. GMT), or nothing if the time zone cannot be determined |
| Date fields | |
| %a | the locale’s abbreviated weekday name (e.g.: Sun..Sat) |
| %A | the locale’s full weekday name (e.g.: Sunday .. Saturday) |
| %b | the locale’s abbreviated month name (e.g.: Jan .. Dec) |
| %B | the locale’s full month name, (e.g.: January .. December) |
| %c | the locale’s date and time (e.g.: "Sun Dec 15 14:12:47 GMT 1996") |
| %d | day of month (01 .. 31) |
| %j | day of year (001 .. 366) |
| %m | month (01 .. 12) |
| %U | week of the year with Sunday as first day of week (00..53, where week 1 is the week containing the first Sunday of the year) |
| %W | week of the year with Monday as first day of week (00..53, where week 1 is the week containing the first Monday of the year) |
| %w | day of week (0 .. 6, where 0 corresponds to Sunday) |
| %x | locale’s date representation (e.g. today in Britain: “15/12/96”) |
| %y | year without the century (00 .. 99) |
| %Y | year with the century (1970 .. 2038) |
Posted in Unix/Server Stuff Tagged Apache, Cronolog, Linux/Unix, Logs, stats 1 Comment