Read data from Google Sheet into a Python Pandas DataFrame

Recently I have done lot of data analysis in Python (more details about this in another post) and have started to like Pandas a lot. The other day I had to process some data from a Google Sheet and was wondering whether I could read the data as a Pandas DataFrame and after a quick search found the gspread package and within a few lines of code I was able to read data from Google Sheet into a Pandas DataFrame.

I thought of sharing the code here so that it would be helpful for others and I would also know whether to search for it when I need it the next time :)

So here we go, this is all we need. Of course you need to install the gspread and pandas packages.

Posted in Python | Tagged , , | Leave a comment

WP Github Gist plugin now supports the new Gist API

Like my Twitter Avatar plugin, even my WP Github Gist plugin was broken recently due to changes in the underlying API that my plugin was using. This time it was because of the changes in the Gist Embed API.

Adding user to the gist embed url

In the earlier version of the gist embed API, the url to the embed script had only the id of the gist. But in the new version the user name is also added.

So I have added a new attribute called user to the gist shortcode where you can specify the user name.

Handling old Gist shortcodes

Unfortunately the above change breaks old gist shortcodes that you might already have in your posts where the user name is not specified. I have introduced a filter called wp-github-gist-user using which you can specify the user name for old gists. This will be helpful if you have embedded gist from the same user in all your old posts. If you have embeded gists from multiple users then I am sorry, you may have to go and manually add the user to all of them.

You can checkout the sample plugin that I have created that explains how you can override the user name using this filter. Essentially you have to just add the following code to your functions.php file or create a separate plugin like I did.

Download

This is a mandatory update. You can download the latest version of the Plugin from the Plugin’s homepage or install it using the Plugin installer.

Feedback

Please keep the feedback coming and if you want me to add any new features or find a bug, please leave a comment or post about it in twitter.

Posted in WordPress | Tagged , , , , | Leave a comment

Twitter Avatar plugin now supports the new Twitter API

Because of the recent changes in Twitter API, my Twitter Avatar Reloaded WordPress plugin stopped working. I rewrote the way my plugin communicates with Twitter and updated the code to use the new API and now my plugin works again.

New Twitter Backend

Instead of rewriting the entire Twitter backend again, I am not using the excellent wp-twitter-api, provided by @timwhitlock which allows you to easily communicate with Twitter API from your WordPress plugin.

If you are a WordPress plugin developer and want to communicate to the Twitter API from your WordPress plugin, then I would highly recommend you to use wp-twitter-api, instead of creating your own.

Setting up Twitter App

The new version of the Twitter API needs you to make authenticated calls even to get some basic information about a twitter screen name. So after you installed the plugin you need to follow these steps so that the plugin could interact with the Twitter API.

  • Register a Twitter application at https://dev.twitter.com/apps
  • Note the Consumer key and Consumer secret under OAuth settings
  • Log into WordPress admin and go to Settings > Twitter API
  • Enter the consumer key and secret and click ‘Save settings’
  • Click the ‘Connect to Twitter’ button and follow the prompts.

Download

This is a mandatory update. You can download the latest version of the Plugin from the Plugin’s homepage or install it using the Plugin installer.

Feedback

Please keep the feedback coming and if you want me to add any new features or find a bug, please leave a comment or post about it in twitter.

Posted in WordPress | Tagged , , , | Leave a comment

How to print unique errors(with count) from Apache error logs

After I moved to WordPress Multisite recently, I had to closely watch the Apache error log file for a couple of days to make sure that there weren’t any configuration mismatch.

Initially I used run the tail command to view the errors, but then I quickly realized that if there is a frequent error, then it becomes extremely difficult to find out other non frequent errors.

Command

After some fiddling with the awk and sort commands, I came up with the following one liner that prints the unique errors with their count.

Explanation

awk script is executed for each line in the error log file.

Field Separator

The -F directive is used to specify field separator. If you look at a single line in the apache error log file, you will notice that each field is enclosed by [ and ].

awk allows us to specify multiple field separators by enclosing them between []. That’s what we are specifying in -F[\[\]]

Filtering out lines that contain error

After splitting the fields, we need to filter out the lines that contain the term error in the fourth field. That’s what happens in the next part of the command. Basically the part $4 == "error"{print $7} prints the seventh field, which has the real error message if the fourth field is equal to the string “error”.

Sorting the error messages

The next step is to sort all the error messages. This is done by piping the output to the sort command.

Finding unique error messages and counting them

The next step is to find the unique error messages and then count them. This is done by piping the output to the uniq command. The -c flag does the counting.

Sorting the messages by frequency

The last step is to sort the messages again by frequency and print them. This is done by the last sort command. The -n flag sorts by numbers and the flag -r does the sorting by descending order.

Posted in Unix/Server Stuff | Tagged , , , , , | Leave a comment

How To Unfreeze Frozen SSH Tabs In Mac Terminal App

I use the Mac Terminal App a lot to ssh into remote servers. It all works great except for a single annoyance. If I leave the ssh tab open without any activity for sometime or put my Mac to sleep, then the tab gets frozen and becomes unresponsive. It takes a couple of minutes for it to log out of the server and become responsive again, so that I can reuse the tab again.

Whenever this happens, I have to close the unresponsive tab and open a new one, or wait for 5-10 minutes. I did some research about it and found that Mac Terminal App provides an escape command that can be used to forcefully log out of the remote server. All you need to do is to type the following command in the unresponsive tab.

~.

(Tilda followed by a dot)

I tried it and it works like a charm. It immediately logs you out of the frozen ssh tab and you can reuse your tab again without waiting for it to become responsive again.

Hope this small trick is useful for others as well :)

Posted in Unix/Server Stuff | Tagged , , , | Leave a comment

How To Properly Create Tables In WordPress Multisite Plugins

Recently, I added full WordPress Multisite compatibility to my Email Log WordPress plugin and during the process, I learned a lot about how to create and handle tables in WordPress Multisite plugins. I thought of documenting it here so that it would be useful for other plugin authors.

Creating tables in Single site installations

Before we get into WordPress Multisite, first let’s see how we can create a table in a single site WordPress installation. You have to hook into the register_activation_hook action, which will be called every time you activate your plugin and then check if your table exists or not. If it doesn’t exist, then you can create your table.

The following code shows how you can do it.

The above code will work even in WordPress Multisite installations, if your plugin will be activated individually for each blog. But if your plugin is networked activated, then the above code will create the table only for the primary blog.

Creating tables for all blogs in a WordPress Multisite installation

Now that we know how to create the table for a single blog, let’s see how we can create the table for all the blogs in a WordPress Multisite installation. Even in this case, we have to hook into the same register_activation_hook action, but should loop through all the blogs in the network.

The following code shows how you can do it.

Creating table when a new blog is created

The above code will create the table only for the blogs that were created before the plugin got network activated. We should also make sure that we create the table for every new that gets created. In order to do that we can hook into the wpmu_new_blog action.

The following code shows how you can do it.

Deleting the table when a blog is deleted

Now that we are creating the table for every new blog, it is our job to make sure that the table is deleted when the blog is deleted. In order to do that, we can hook into the wpmu_drop_tables filter.

The following code shows how you can do it.

Querying the correct table

When we are querying the table, we should always use $wpdb->prefix . $table_name. If we do that, then WordPress will automatically query the correct table based on the current blog. We don’t have to manually find out the blog id and add it to the table name.

Now your plugin should be completely compatible with WordPress Multisite :)

Alternate approach

If you look into the above code closely, you will notice that we are creating one table for each blog in the network. For most WordPress Multisite installations, this shouldn’t be a problem. But some WordPress Multisite installation may have thousands and even hundreds of thousands of blogs. In those cases, we might end up creating huge number of tables which might become a bottleneck. Also my plugin needed just one table. Some plugins might need more than one table, which might also increase the number of tables that gets created.

One alternate approach is to create just one table for all blogs and then separate out data for each blog using a blog_id column. While querying the table, we can filter out based the blog_id column.

If I had started my plugin from scratch, I would have done that instead of creating separate tables.

Removing the tables when the plugin is deleted

The other thing to keep in mind is that we should delete all the tables when the plugin is deactivated and deleted. I will write a separate article explaining how we should do that.

With WordPress Multisite becoming more popular these days, I hope this information was useful to you. Do let me know if you have any question or comments. Also you can checkout the entire code of my Email Log plugin in github.

Posted in WordPress | Tagged , , , | 5 Comments

Email Log plugin is now fully WordPress Multisite compatible

I just released version 1.7 of my Email Log WordPress plugin, which adds full compatibility with WordPress Multisite.

About Email Log WordPress plugin

Email Log is a WordPress plugin that allows you to log every email sent through WordPress and provides a UI where you can view them. The logged emails can be searched based on date, email address or subject.

WordPress Multisite compatibility

Earlier versions of Email Log plugin had some compatibility with WordPress Mu (older version of multisite), but I didn’t test it for long time. Also when WordPress Mu got merged into WordPress core, certain things have changed and my plugin was not fully compatible after that.

I came to know about it recently and then started to work on it to make it fully compatible with WordPress Multisite, since I have moved this blog to WordPress Multisite as well. I also learned a lot about WordPress Multisite during the process and will write a separate blog article about it soon. Update: As promised, here is the new blog article explaining how to properly create tables in WordPress Multisite plugins.

Dev Time

This release took me about 2.5 hours of development time. You can find more details about the dev time tracking which I have recently started doing in a separate blog post.

Download

You can download the latest version of the Plugin from the Plugin’s home page.

Feedback

Try out the Plugin and if you have any comments or if you want to report any bugs, please leave a comment below.

Posted in WordPress | Tagged , , , , | 1 Comment

Prevent users from adding new terms to custom taxonomy in WordPress

Recently I faced a scenario where I had to prevent certain users from adding new terms to some custom taxonomies that I created. After some research I found that there isn’t a straight forward way to do this using user capabilities or roles.

The only way is to use the pre_insert_term hook and black-list taxonomies based on user capabilities. I am sharing the code here so that it would be useful for people who want to something similar.

In the above code change the taxonomy name and capability to suit your usecase.

Posted in WordPress | Tagged , | 2 Comments

Merged my blogs using WordPress Multisite

Most of you know that I have another website in addition to this one where I talk about my experience with hardware programming. When I started the website in late 2012, I decided to host it using another single instance of WordPress.

Recently one of the users of my Bulk Delete WordPress plugin asked me if I can add the ability to delete posts in bulk from multiple sites at once in a WordPress multisite installation. I have played around with WordPress MU a bit before it got merged with WordPress, but haven’t played much with WordPress multisite after it got merged. I did a WordPress multisite installation in my dev server and was very impressed with it. I also realized that it will save me lot of time by just maintaining one WordPress installation instead of two.

After testing out plugin compatibility, I converted both the blogs into a single WordPress multisite installation. The migration process was very simple and now both the sites are powered by a single WordPress multisite installation.

If you manage more than one WordPress installation, then do consider doing it using WordPress multisite. It might simplify lot of things for you.

Posted in WordPress | Tagged , , | 2 Comments

Looking back at 2013

Another year just ended and it’s time for publishing my year end review like the ones I did for the previous few years.

Personal Highlights

2013 was a milestone year for me in the personal front. I spawned a new process ;) (Translation for those who didn’t get it: I became a dad).

Like all parents know, this is the most rewarding as well as the most demanding experience. Luckily, my little one has kind of gotten into a proper sleeping schedule these days and so my (and my wife) zombie days are in the past.

This would explain why I was kind of out of action (less blog posts, not much public speaking etc) in the last few months. Hopefully, I should be back to action in 2014.

Open source contribution

Back in the early days of 2013, I decided to do at least one contribution per day to open source projects. I was able to maintain this streak for more than 4 months (124 days to be exact). A look at my github profile, should explain everything :)

I also took over the Arduino Makefile project and worked hard to steadily increase its popularity. Right now it has around 250 stars and close to 100 forks in Github.

I also moved most of my WordPress plugins to github. This also increase the popularity of my WordPress plugins and also user contributions.

Reading

Towards the end of the year, I decided to take a reading break and this helped me to complete around 10 novels from Isaac Asimov’s famous Foundation Series.

Blog Stats

I used my Year End stats WordPress plugin to generate the following stats.

2013-no-posts2013-no-comments2013-total-post-length 2013-avg-post-length

The total number of posts that I have written in 2013 has increase a little bit and I am happy about it.

I am planning to write a little more in 2014 as well. Hopefully my little one allows me to do so ;)

What’s in for 2014?

I am definitely planning to spend quality time with my family, especially the little one :) As most parents know, the first few years of a kid are the most rewarding ones and I don’t want to miss out on it.

Assuming that I will have time after that, I want to do the following as well.

Books

I definitely want to complete the Foundation Series that I started. I still have another 5 books to complete.

After that I am planning to read favorite person Chris Hadfield’s Autobiography: An Astronaut’s Guide to Life on Earth. I already have the book, I am just waiting to finish Foundation Series.

Also, if you have recommendation, please let me know. I am planning to read at least a dozen books this year.

Learn R

I am also planning to concentrate a little bit more on Statistic and Machine Learning and as part of it, I am planning to learn R. I am also planning to see if I compete on some competitions in Kaggle. I might also start writing about more about Machine Learning this year. So stay tuned :)

Hardwarefun

Regarding hardware stuff, I am planning expand HardwareFun.com and write more posts there and also continue working on Arduino and start playing around with Raspberry Pi as well. So it is going to be pretty exciting.

From the looks of it, 2014 seems to be an exciting as well as a busy year and I am really looking forward to it.

So how was your 2013?

Posted in Random/Personal | Tagged , , , | Leave a comment