You are here: Home » Blog

Why you should open source your next pet project

Published May 20, 2013 | In Arduino, Random/Personal

After looking into my github account, lot of people used to ask me why I open source my pet projects, especially after looking at my streak at github (my current streak is at 52 days 😉 ) the frequency of the question has increased.

I used to come up with an answer by combining these words randomly (passion, show off, free, rebel, linux, anti-Microsoft, license). But today, I realized that there is another (more important) reason why you should open source your (pet) projects.

Your project might have a longer shelf life than your interest in the project and in some cases longer than your life itself.

You might lose interest in the project, but if you have open sourced it, then there might be someone else who will continue to maintain it. But instead if you had just kept it in your hard drive, then it will just become part of the cosmic dust.

Abandoned projects

I have read about many examples where projects are continued to be maintained by other people, when the original creator has lost interest. _why is one famous example. When he suddenly went offline in Aug 2009 people immediately started creating mirrors of his projects and even though he is no more (in the online world) his projects are still being maintained actively.

None of this would have happened if he had not open sourced his code.

Arduino Makefile

Let’s talk about the project that thought me this.

Sometime back, I came across a post which explained how you can program Arduino from command line by using a makefile. I immediately got hooked and started using it when even I was working on Arduino sketches.

Soon I found out a couple of issues and submitted pull requests. Some of my pull requests were accepted and others weren’t. While browsing through the issues and pull requests of the project in github, I found that there were lot of other people who have also contributed various patches to the project. It also seemed that the original author has stopped doing any development or answering pull requests.

I started to consolidate all pull requests and patches from various forks into a common fork and even updated the readme file expressing my desire that one day all these changes would get into the main repo 🙂

Fast forward a couple of months.

Last week, I was surprised to receive an email from the original author, saying that he has lost interest in the project and whether I would be interested in maintaining the project going forward. I happily accepted and he transferred the repo to my account, after I merged all my changes back into the main repo.

None of this would have been possible if he didn’t choose to open source the project initially and also let go the project when he lost interest.

Thanks to Github

Ohh, by the way, I should also mention that some of it was possible only because of github. If github wasn’t there, then it would had been difficult for me to find and consolidate other forks and he also wouldn’t have noticed my fork or the effort that I was spending on the project.

Also, the recent addition of redirects by github made the repo transfer really seamless.

So remember to open source your pet projects by at least uploading them to github. Because you may lose interest in the project some day, but the project might still continue to live long after that.

Happy Open sourcing 😉

Posted in Arduino, Random/Personal | Tagged , , , | 7 Comments

Find and replace strings in MySQL

Published May 15, 2013 | In Database Programming

Recently, I was checking the apache error logs of a project I was working on and found that it has lot of 404 errors for images. I used my apache error parsing script and found that most of the errors were because the production db was still having some references to images present in the stage machine.

I had to do a quick find and replace in the MySQL db and after a quick search came up with the following query, using the replace function.

UPDATE table SET field = REPLACE(field, 'stage-url', 'prod-url');

Since I was about to run this query on a prod db directly, I wanted to add another condition to replace only rows that contained the stage url as an additional precaution (of course after taking backup of the db 😉 ).

UPDATE table SET field = REPLACE(field, 'stage-url', 'prod-url') WHERE INSTR(field, 'stage-url') > 0;

Hope this query is useful for someone who also had a shock like me when they looked at the apache error logs 😉

Posted in Database Programming | Tagged , , , | 3 Comments

Recovering Android KeyStore password

Published May 8, 2013 | In Android/Java

Sometime back, I helped a friend of mine to launch an app in the Android market. As most of the Android developers know, one of the requirements to upload your APK file into the android market is that you have to sign it using a KeyStore.

I created a new KeyStore for him and I signed the app and he uploaded it to Android market. All this happened a couple of years back and now when he wanted to update the app again, he had to sign the apk file again with the same KeyStore.

I had remembered to backup the keystore file, but somehow couldn’t remember the exact password I used while creating the KeyStore.

After a quick search, I realized that I am not the only one who has done that before and also realized that there is no other way to upload the app again, without recovering the password 🙁

I started my search to find ways to recover the password and luckily found a tool, which took a smart list of words and then tried to guess the password by brute force. I gave it a couple of probable words that I would have used for the password and luckily I was able to recover the password with-in a couple of hours. 🙂

Some major take aways for me from this entire episode.

  • Always, always and always make sure you have more than one copy of backup
  • Don’t ever forget the password for your KeyStore. If you do then you might have to abandon your current app in the market.
  • As @pareshmayani pointed out below, make sure you store the keystore file as well in your app’s repo. God be merciful on you, if you don’t version your code
  • Sometimes keeping a simple password helps 😉

Posted in Android/Java | Tagged , , , , | 7 Comments

Setting up a home network with two routers

Published Apr 27, 2013 | In Gadgets

Recently, I solved network connectivity problem at home by setting up a slightly complex network using an old BSNL wa3002g4 router. Earlier I wrote about how I used this BSNL wa3002g4 router as a WiFi switch and lot of people said that the post was very useful. So I decided to document this setup as well, so that it will be useful for someone who wants to do the same thing 🙂

Problem

First, let me explain the problem that I was facing. I have half a dozen devices running multiple OS (two desktops, a Mac, an iPhone, couple of Android phones, couple of Raspberry Pis.. you get the idea right 😉 ) that I wanted to be in the same network and share my single internet connection. Also, I live in two floors and my WiFi signal was too week to penetrate the concrete across the floors.

In my old setup, the cable from my broadband provider was connected to a Netgear router, that acted as a network switch as well as a WiFi router. This was in my top floor where the desktops were present. But when I am in the bottom floor, the WiFi signal was too weak to be used in my laptop or in my phones.

Solution

My first thought was to buy a WiFi expander and use it to boost the WiFi signal in the bottom floor. When I was about to buy the signal booster, I remembered that I had an old BSNL wa3002g4 router, which I was using earlier as a WiFi switch. I know that it is a multi-functional device, so I was trying to see if I can use it to expand my WiFi range.

Unfortunately, it can’t act as a WiFi expander. But then I realized, I can still use it as a switch. I started to experiment a bit with it and after some trial and errors, I made it to work in the way I wanted 🙂

My current setup

The cable from my broadband provider (currently ACT broadband, who gives me 15 Mbps 😉 ) is connected to the input of my Netgear router which is placed in the top floor. It gives out both wired and wireless connections to the devices in the top floor. One of those output port from Netgear is connected to my old BSNL wa3002g4 router which is placed in the bottom floor. This second router gives wired and wireless to the devices in the bottom floor.

After I had this setup, I realized that both the routers were creating their own sub networks and the devices are not in the same network. Also I had to use two different WiFi ssid’s which kind of defeated the whole purpose.

DHCP relay

It was at this point, that I discovered that my BSNL router has a feature called DHCP relay, which allowed me to configure the connections so that they can be in the same network.

DHCP relay, allows you to specify a different DHCP server from where the device should get dynamic ip address instead of generating ip address on its own. I specified the ip address of my Netgear router as the DHCP server in the DHCP relay config of my BSNL router.

This allowed the BSNL router to relay all DHCP requests to the Netgear router instead of generating the ip address on its own. Now all my devices were connected in the same network.

Configuring WiFi

The next problem that I had to solve was setting up the WiFi ssid’s. Again by trial and error I found that if I configured both routers with the same WiFi ssid and same authentication parameters, then the devices can connect to both the routers seamlessly. I changed the WiFi config in both the routers to be the same and then my devices started connecting to the network without any issues even when I moved across floors. The devices always pick up the router which has the best signal.

And now I am finally happy that all my devices are in the same network and that I can connect to my network from any part of my home, without worrying about the signal range 🙂

Posted in Gadgets | Tagged , , , | 15 Comments

Syntax highlighting code in slides

Published Apr 6, 2013 | In Events/Conferences

If you look at most of my slides that I have created for my talks, you will find that whenever I have code, I always try to syntax highlight it.

It is very easy to comprehend code that is syntax highlighted and I always make it a point to do that, so that it is easy for my audience.

Recently a couple of people asked me how I do it and I thought of writing about it, so that you can also do it in your slides 🙂

No screenshots, please

I find that most people take the easy route and just put a screenshot of the code from an IDE. Please don’t do it. It defeats the entire purpose of posting code in the slide, since people can’t copy it. And also it becomes extremely difficult for you update it at a latter point.

So friends, don’t let friends do it 🙂

Rich Text Format (RTF)

Now that we have decided not to post screenshot, the next thing we need to do is to convert the code to RTF format.

Most IDE’s support copying the code in RTF format, so first check whether your favorite IDE supports it. If not there are many tools that are available that can do it.

If your editor is Vim

If you are like me and your favorite editor is Vim and you are using Mac, then you can use an addon called copy as rtf. This addon gives you a simple command :CopyRTF which will convert the entire buffer (or highlighted lines) to RTF, including the background and foreground color and copy it to buffer.

So, next time you show code in your slides, make sure you convert them to RTF, instead of placing a screenshot.

Posted in Events/Conferences | Tagged , , | 2 Comments

Creating new pages in WordPress through code

Published Apr 4, 2013 | In WordPress

Recently, I had to create a new page automatically in WordPress, when a Plugin was created.

I thought there will be a simple core function like wp_insert_post which I can use. It turned out that create a new page from code is not that easy in WordPress. I ended up creating a new function myself wrapping the call to wp_insert_post.

I thought of posting it here so that I know where to look when I need it again 🙂

So here is the function.

It takes 5 parameters, out of which only one is mandatory.

  • $title – The title of the post (mandatory)
  • $content – The content of the post
  • $parent – The post id of the parent page. If this is the top level page, then pass 0
  • $page_template – The page template to use for this page
  • $menu_order – The menu order field.

While you are at it, you might also want to find out whether the page with that title already exists, before creating a new one.

Posted in WordPress | Tagged , | 1 Comment

Count the number of empty lines in a file using grep

Published Mar 28, 2013 | In Unix/Server Stuff

Recently I had a need to count the number of empty lines in a text file. After some digging into the man pages of grep I was able to come up with a one liner which was able to do it.

Following my tradition of documenting one liners, I am going to document this one as well 🙂

Assumption

By empty line, I mean any line which either has no characters or has only whitespace (space, tab) characters.

Command

For the impatient in you, here is the actual command.

Explanation

  • -P '\S' – This selects all lines that have a non whitespace character
  • -c – Print the count of matching lines
  • -v – Select only the non-matching lines

So, we are first matching all lines that have a non whitespace character and then use -v option to ignore them and then -c option to print the count instead of the actual line.

If we wanted the count of all non-empty lines, then we just have to remove the -v option from the above command.

Hope this is helpful. Happy Grep’ing 😉

Posted in Unix/Server Stuff | Tagged , , | 7 Comments

Passing command-line arguments to Pig scripts

Published Mar 27, 2013 | In Hadoop/Pig

In pretty much every Pig script that you will be writing, you will have to specify at least two locations – the input and the output locations. If you are going to use multiple inputs or have to register multiple jars for UDF, then this is bound to increase.

I run most of my Pig scripts through a shell script and I was looking for a way to pass in these locations at runtime instead of hard coding them in the Pig script. After a bit of research, I found that Pig has the ability to accept command-line parameters and there are in fact multiple options to pass them. I thought of documenting them here so that I know where to look when I need to 🙂

Parameter Placeholder

First, we need to create a place holder for the parameter that needs to be replaced inside the Pig script. Let’s say you have the following line in your Pig script where you are loading an input file.

INPUT = LOAD '/data/input/20130326'

In the above statement, if you want to replace date part dynamically, then have to create a placeholder for it.

INPUT = LOAD '/data/input/$date'

Individual Parameters

To pass individual parameters to the Pig script we can use the -param option while invoking the Pig script. So the syntax would be

pig -param date=20130326 -f myfile.pig

If you want to pass two parameters then you can add one more -param option.

pig -param date=20130326 -param date2=20130426 -f myfile.pig

Param File

If there are lot of parameters that needs to be passed, or if we needed a more flexible way to do it, then we can place all of them in a single file and pass the file name using the -param_file option.

The param file uses the simple ini file format where every line contains the param name and the value. We can specify comments using the # character.

date=20130326
date2=20130426

We can pass the param file using the following syntax

pig -param_file=myfile.ini -f myfile.pig

Default Statement

We can also assign a default value to a parameter inside the Pig script using the default statement like below

%default date '20130326'

Processing Order

One good thing about parameter substitution in Pig is that you can pass in value for the same parameter using multiple options simultaneously. Pig will pick them up in the following order.

  • The default statement takes the lowest precedence.
  • The values passed using -param_file takes the next precedence.
    • If there are multiple entries for the same param is present in a file, then the one which comes later takes more precedence.
    • If there are multiple param files, then the files that are specified later will take more precedence.
  • The values that are passed using the -param option takes the next precedence.
    • If multiple values are specified for the same param, then the ones which are specified later takes more precedence.

Debugging

Sometimes, the precedence might be little confusing, especially if you have multiple files and multiple params. Pig also provides a -debug option to debug this kind of scenario’s. If you invoke Pig with this option, then it will generate a file with extension .substitued in the current directory with the place holders replaced with the correct values.

What I use?

I follow this convention while passing params in Pig and it has worked nicely for me so far.

I specify a default value using the default statement and then pass actual values using the -param_file option. If I am in a hurry and just want to test something locally, then I use -param option, but generally I try to put them in a separate ini file so that I can check-in the options as well.

Posted in Hadoop/Pig | Tagged , , | 1 Comment

Remove duplicate lines based on a field

Published Mar 20, 2013 | In Unix/Server Stuff

Recently while working on formatting some data files for further processing, I had to remove duplicate lines from the file based on a particular field. After trying out cut and grep commands, I was finally able to solve it with a very concise awk command/script.

The command was so concise but still was packed with so much information and it helped me to learn more about the awk scripting language. I thought of writing about it here so that it is useful for others and also I know where to search for it, when I needed it 🙂

Feel free to use it in whatever way you want, if it solves your problem as well.

Input and output data

Let me first explain the input data I had and the output that I was expecting.

Consider a file which has the following lines. Each line has four fields.

Now assume that we want to remove duplicate lines by comparing only the second field. We want the output to look like this.

Command

Get ready for the surprise. The actual command is just this.

Explanation

awk script execution and printing

awk script is executed for each line and if the result is true then the line is printed. If the result is false then the line is not printed.

Associative arrays

The awk language supports associate arrays, similar to the ones found in PHP. The script x[$2]++ fills up an associate array. The key used here is $2 which refers to the second field and x is the variable name. You can use any name for it.

The array is populated for every line. This is how the array would look like after each line.

Conditional evaluation

The ! operator results in a boolean evaluation which determines whether a particular line should be passed on to the output (printed) or not.

When the field is not present in the array, then it results in a zero value which is false. The ! (not) operator evaluates it to non-zero, which results in a true value and the line is passed on to the output (printed). When a duplicate is found, the array returns a non-zero count, which is true, but the ! converts it to false and that line is not passed on to the output.

The expanded version of the above command would be

But what is the fun in using the expanded version 😉

Field separator

In the input file that I had, the fields were separated by whitespace, so I didn’t have to specify the field separators. But if you are using a non-whitespace field separators, then you can specify it by adding FS="," to the above command.

This one-liner actually thought me that awk supports a full programming language that can be used to create scripts and also increased my understanding of the way awk command works. Hopefully this teaches something for you as well 🙂

I know that this is already a concise version, but if you think that this can be improved, then do let me know.

Posted in Unix/Server Stuff | Tagged , , | 5 Comments

Will be in US for the next two weeks

Published Mar 1, 2013 | In Travel/Trek

Well guys, a quick update.

I am travelling to Sunnyvale, CA tomorrow morning and will be back in Bangalore in two weeks.

It is a business trip and the first week is packed with lot of meetings. So if you have sent me an email, expect a delay in the reply 😉

Unlike the last few times, I have only one full weekend this time and I haven’t planned anything yet. Might atleast go on a city tour around SFO I guess.

The worst part of travelling to US from India, is that you have to sit in a flight for 20 hours 🙁 . I have loaded my Kindle with a couple of books and I guess it would last for the entire trip. Meanwhile if you have any recommendations that do let me know, will load it up in my Kindle.

Posted in Travel/Trek | Tagged , | 2 Comments