Tag Archives: Unix/Server Stuff

Parse Apache error log and list down all missing images

Published Feb 18, 2013 | In Unix/Server Stuff

Recently I had to parse Apache error log and find out all images that are missing.

After referring to a couple of man pages, I came up with this one liner. I am sure I will need it again, so thought of noting it here so that I know where to look when I need it again 😉

Feel free to use it in whatever way you want, if it solves your problem as well.

Assumption

This assumes that each line in your apache error log looks like this.

Explanation

Filter all 404 lines from the log file

The first step is to filter all lines that that contain the “File does not exist” text. This is done by using sed.

  • By default, sed prints out all lines. This is prevented by the -n.
  • The second option is the regular expression followed by the p flag. This option prints out all lines which match the text.
  • The third option is the name of the error log file.

Extract the last column of matching lines

The next step is to retrieve the file name from the matching lines. This is done by using awk.

  • By default awk uses space as the delimiter and splits the lines into different columns. If you look at each line, we want the last column.
  • NF is a special variable which points to the last column.
  • print $NF prints the last column

Filter only images

The next step is to filter out only the images. This is done again by using sed.

  • I use -n again to prevent sed from printing all lines.
  • The -r is added, so that we can use extended regular expression
  • The regular expression (jpg|jpeg|png|gif)$ filters out all images and p at the end prints out only lines that match

Sort and find uniques

The sort and uniq commands sort the list and find the unique lines.

Write to a file

The final output is written to a file by using the redirection > operator. If you want to append to a file then we may have to use >> operator.

More to come

It is really amazing like how you can combine these tools to do amazing things. I am planning to document other one liners which I end up creating to solve my problems. So stay tuned 🙂

Also if you think this can be improved, then do let me know as well.

Posted in Unix/Server Stuff | Tagged , , , | 5 Comments

Handling FTP usernames with @ in them

Published Aug 4, 2012 | In Unix/Server Stuff

During my recent FTP adventures, I also found that some shared hosting sites give you an FTP username with the ‘@’ symbol in them. It is fine as long as you are going to use a GUI client to connect to FTP. But if you try using the commandline or Finder in Mac, you will have issues since the ‘@’ symbol is also used to separate the username from the host.

After some research I found that the ‘@’ symbol in the username can be replaced with ‘+’ while specifying it in the command line. I tested it with both wput and the Finder in Mac and it worked perfectly in both.

So remember, the next time you try to connect to FTP server from command line and you have a ‘@’ symbol in the username, then replace it with the ‘+’ symbol. Happy FTP’ing 😉

Posted in Unix/Server Stuff | Tagged , , | 4 Comments

Excluding .svn folders while transferring entire folder by FTP

Published Aug 3, 2012 | In Unix/Server Stuff

Recently I had to transfer an entire folder, with lot of sub-folders to an FTP server. I know that there are lot of FTP GUI tools available that can do it, but I wanted to do it in command line so that I can script it.

I searched for the solution and came across an excellent tool called wput, which does exactly that very easily. It is very similar to wget, but instead of downloading the content, it allows you to upload it.

I installed it using apt-get and was trying to upload the entire directory. It was at this point I realized that I want to exclude all the .svn folders.

I again started searching for an answer. I even posted about it in stackoverflow, but couldn’t find a solution. I then went over the man page of wput and hidden inside was this gem, which allowed you decide on which files to include/exclude from the directory.

I thought of posting it here, so that it is useful for others and also I know where to find it when I need it next time.

So all you need is just one line. If you have not installed wput before, the install it using one of the following commands based on your operating system.

Posted in Unix/Server Stuff | Tagged , , | 3 Comments