Looking back at 2013

Another year just ended and it’s time for publishing my year end review like the ones I did for the previous few years.

Personal Highlights

2013 was a milestone year for me in the personal front. I spawned a new process ;) (Translation for those who didn’t get it: I became a dad).

Like all parents know, this is the most rewarding as well as the most demanding experience. Luckily, my little one has kind of gotten into a proper sleeping schedule these days and so my (and my wife) zombie days are in the past.

This would explain why I was kind of out of action (less blog posts, not much public speaking etc) in the last few months. Hopefully, I should be back to action in 2014.

Open source contribution

Back in the early days of 2013, I decided to do at least one contribution per day to open source projects. I was able to maintain this streak for more than 4 months (124 days to be exact). A look at my github profile, should explain everything :)

I also took over the Arduino Makefile project and worked hard to steadily increase its popularity. Right now it has around 250 stars and close to 100 forks in Github.

I also moved most of my WordPress plugins to github. This also increase the popularity of my WordPress plugins and also user contributions.

Reading

Towards the end of the year, I decided to take a reading break and this helped me to complete around 10 novels from Isaac Asimov’s famous Foundation Series.

Blog Stats

I used my Year End stats WordPress plugin to generate the following stats.

2013-no-posts2013-no-comments2013-total-post-length 2013-avg-post-length

The total number of posts that I have written in 2013 has increase a little bit and I am happy about it.

I am planning to write a little more in 2014 as well. Hopefully my little one allows me to do so ;)

What’s in for 2014?

I am definitely planning to spend quality time with my family, especially the little one :) As most parents know, the first few years of a kid are the most rewarding ones and I don’t want to miss out on it.

Assuming that I will have time after that, I want to do the following as well.

Books

I definitely want to complete the Foundation Series that I started. I still have another 5 books to complete.

After that I am planning to read favorite person Chris Hadfield’s Autobiography: An Astronaut’s Guide to Life on Earth. I already have the book, I am just waiting to finish Foundation Series.

Also, if you have recommendation, please let me know. I am planning to read at least a dozen books this year.

Learn R

I am also planning to concentrate a little bit more on Statistic and Machine Learning and as part of it, I am planning to learn R. I am also planning to see if I compete on some competitions in Kaggle. I might also start writing about more about Machine Learning this year. So stay tuned :)

Hardwarefun

Regarding hardware stuff, I am planning expand HardwareFun.com and write more posts there and also continue working on Arduino and start playing around with Raspberry Pi as well. So it is going to be pretty exciting.

From the looks of it, 2014 seems to be an exciting as well as a busy year and I am really looking forward to it.

So how was your 2013?

Posted in Random/Personal | Tagged , , , | Leave a comment

Get the SQL query generated by WordPress

One of the good things about WordPress is that it provides a nice abstraction to DB and you can query pretty much anything from the DB, by using the appropriate functions instead of writing the query yourself.

All though it is good, there are times when you want to see what was the actual SQL query that got generated. I faced this senarious myself, when I was debugging an issue in my Bulk Delete WordPress plugin.

It turns out that it is quite easy to do it.

The WP_Query object has a method called request that returns the SQL query that was generated.

So if you want to get the query generated by the global $wpdb object then you can use the following code

<?php echo $GLOBALS['wp_query']->request; ?>

On the other hand, if you are using a custom WP_Query object then you can call the request method on your object.

<?php
$my_query = new WP_Query();
echo $my_Query->request;
?>

Posted in WordPress | Tagged , , | 1 Comment

Error while starting Jython in Mac

After I figured out how I can use Python to create Pig UDF functions, I got interested in Jython and wanted to play around with it. So I installed it in my Mac through homebrew by executing the following command.

brew install jython

Everything got installed properly, I was able to run Jython after setting
up the following lines in my [bashrc](https://github.com/sudar/dotfiles)

export JYTHON_HOME=$(brew --prefix jython)/libexec
export PATH=$PATH:$JYTHON_HOME/bin

But there was one small annoyance. Every time I was staring Jython, I was getting the following strange warning.

expr: syntax error

After a couple of web searches, I stumbled upon an email thread in the Jython-users mailing list. Basically it was due to a bug in the bash script that is used to start Jython. I opened /usr/local/Cellar/jython/2.5.3/libexec/bin/jython file and changed the line, which fixed the error.

if expr "$link" : '/' > /dev/null; then to if expr "$link" : '[/]' > /dev/null; then

I am exploring Jython more and will keep you guys updated, if I find something interesting.

Posted in Python | Tagged , , , | 1 Comment

Spawned a new process ;)

Well, I wanted to have the geekiest synonym for “We just had a baby” as the title for this post (similar to my wedding card). Since I am currently typing this from a hospital, sleep-deprived, while constantly sifting my attention between my recovering wife and my new born child, this is the best I came up with ;)

In the last 24 hours, I have experienced extreme cases of all the 8 emotions specified in Robert Plutchik’s theory, but in spite of it this had been the best 24 hours in my entire life so far :)

So guys, join me in welcoming our son to this beautiful world. By the way, all credit for the creation of this new process goes to my wife who did all the real work ;)

More details and pics coming soon, after (if) I catch up with some sleep :)

Posted in Random/Personal | Tagged , | 11 Comments

Isaac Asimov Foundation Series reading order

Last week, I asked for recommendations for some SciFi/Fantasy books and the response was enormous. Thanks to everyone who gave me recommendations. After some careful analysis, I finally decided to read Foundation Series by Isaac Asimov.

I was reading the wikipedia article of Foundation Series and then realized that the books were not published in chronological order, which bought the question – In which order should I read the books? In chronological order or published order.

After some research I came to know that Isaac Asimov himself suggested (In the Author’s note at the beginning of Prelude to Foundation) to read the books in chronological order. So I decided to follow his advice :)

I am currently reading them in the following order. The year in the bracket is the year of publication.

  • I, Robot (1950)
  • The Caves of Steel (1954)
  • The Naked Sun (1957)
  • The Robots of Dawn (1983)
  • Robots and Empire (1985)
  • The Currents of Space (1952)
  • The Stars, Like Dust (1951)
  • Pebble in the Sky (1950)
  • Prelude to Foundation (1988)
  • Forward the Foundation (1992)
  • Foundation (1951)
  • Foundation and Empire (1952)
  • Second Foundation (1953)
  • Foundation’s Edge (1982)
  • Foundation and Earth (1986)

I have already finished the first two and I am currently reading “The Naked Sun”. So far it’s been really excellent and once again thanks everyone for the recommendation.

Posted in Books | Tagged , | 1 Comment

Seeking recommendation for some Fantasy/Sci-Fi/Fiction books

It’s been quite sometime since I read anything substantial and so I am planning to take a “reading break” and want to dedicate my free time in the next couple of months for reading.

Since I have a couple of months for this, I don’t want to read short stories or a single novel. Instead, I want to read a series so that once the characters are introduced, they will continue to be present for most of the series. This will be an important factor for me, since it motivates me to complete all the books in the series. This is one of the reason, why I was able to complete the entire Sherlock Holmes series in less than a week :)

I am not very particular about the genre very much, but I would prefer either Fantasy or Sci-Fi. The series that immediately came to my mind were Harry Potter and The Lord of the Rings. I posted about it today morning in Twitter and asked for recommendations.

Within a few minutes, I got the following good suggestions and now I have the tough job of picking it up from here ;)

I have not made up my mind yet, but I am leaning towards Foundation series by Isaac Asimov.

Meanwhile, if you guys have any more suggestions, kindly do let me know.

Update: I have decided to take up Foundation Series by Isaac Asimov first. Thanks everyone for the recommendations.

Posted in Books | Tagged , | 6 Comments

Including external Pig files into Pig Latin scripts

In one of my projects, we had huge number of Pig scripts which dealt with data from a single source. The schema for this common data source is quite complex and changes every few months. Since this schema was present in all Pig files, when ever it changes, it was a real pain to update all Pig scripts.

I was looking for a way to separate out the schema into a separate Pig file and then include it in all other Pig scripts, like how you import a class in Java, instead of copy pasting it into all Pig files.

After some quick web searches, I found that from Pig 0.9 and above this feature is indeed available in Pig itself. It’s called macros. All you need to do is to just include the following line in your Pig script where you need it to be included.

import 'other-file.pig'

You can either give relative path in the above line or set the search path as well from where Pig should include the scripts. If you want to include the search path, then you can do something like this.

set pig.import.search.path '/usr/local/pig,/grid/pig';
import 'external-file.pig';

Now my Pig scripts are organized properly. Hope this helps you as well :)

Posted in Hadoop/Pig | Tagged , , | Leave a comment

Parsing JSON in Pig UDF written in Python

When I wrote about using Python to write UDF functions for Pig, I mentioned that Pig would internally be using Jython to parse the code, but 99% of time this shouldn’t be an issue. But I hit the other 1% recently :)

I had a small piece of Python code that used the built-in json module to parse JSON data. I converted that into a UDF function and when I tried to call it from Pig, I was getting “module not found” exception. After some quick checks, I found that the latest stable version of Jython is 2.5.x and json module was added from 2.6

After some web searches, I came across jyson through a blog post about using JSON in Jython. jyson is an Java implementation of JSON codec for Jython 2.5 which can be used as a drop-in replacement for Python’s built-in json module.

I downloaded jyson jar and then added it to Pig’s Dpig.additional.jars property. In the Python code, I changed the import statement to import com.xhaus.jyson.JysonCodec as json. After that everything started to work again :)

Posted in Hadoop/Pig, Python | Tagged , , , , | 1 Comment

Install OpenCV in Mac using Homebrew

Recently, I switched to a new Mac Retina (one of the numerous perks of working at Yahoo ;) ) and wanted to install OpenCV in it using Homebrew.

I ran the usual brew install opencv command and got No available formula for opencv error message. I have personally ran this command before to install OpenCV and was wondering what went wrong.

After a couple of web searches and ‘hair tearing’, I found that the opencv formula has moved to homebrew-science and I had to do the following.

brew tap homebrew/science

brew install opencv

Finally everything worked. So even if you are facing the same problem, then you know what to do now (and possibly save some hair as well) :)

Oh, by the way, if you own a Mac and don’t know what Homebrew is, then stop everything and install it right away. You can thank me later ;)

Posted in iPod/Apple | Tagged , , , | Leave a comment

Writing Pig UDF functions using Python

Recently I was working with Pig (the apache one, not the animal ;) ) and needed to implement a complex logic. Instead of struggling to write it in Pig, I decided to write a UDF (User defined Function). Also, I was too lazy to copy paste lot of boilerplate code to write the UDF in Java and decided to write it in Python. Long time readers might know that ever since I learned Python (around 7 years ago), I have been a huge fan.

In the end, I found that it was too easy to write UDF’s using Python, when compared with writing them in Java. I thought of writing about it here so that it would be helpful and will act as a starting point for people who also want to write their own UDF using Python.

Python vs Jython

Well, before we start, one thing that we have to keep in mind is that, even though we would be writing our code in Python, Pig will internally execute the code using Jython. 99% of time there will not be any difference, but it is good to keep that in mind.

Python code

First in the python side all we need to do to expose a Python function as a UDF, is to just specify a decorator to it.

Let’s say we have the following Python function that returns the length of the argument that is passed to it.

All we need to expose this function as a UDF is to add the @outputSchema decorator. So the code becomes

When data is passed from Pig to Python, it is passed as bytearray. Most of the time, this shouldn’t be a problem. But there are times when this could be a problem. In those cases, we can just convert it into proper string before we consume it. So the final code would look like this

Pig code

In the Pig side, we should do two things.

  • Register the UDF
  • Call the UDF ;)

Register the UDF

As I said in the beginning, Pig internally will use Jython to parse Python code. So we first need to register our Python file using the REGISTER statement. We can just say REGISTER 'udf.py' USING jython as pyudf

Call UDF

Once we register the UDF using the REGISTER statement, we can then call the UDF function using the alias that we created.

Here is the complete code in the Pig side.

And believe me, that’s all you need to do to write Pig UDF functions using Python. No more unneeded Java classes, boilerplate code or Jar creation process :)

Posted in Hadoop/Pig, Python | Tagged , , | 2 Comments