Tuesday, 17 June 2014

GSOC Week 4: "This is not a coding contest"

Basing my first patch to Gluster as a stepping stone, I've written a small utility glusterfsiostat, in python which can be found at https://github.com/vipulnayyar/gsoc2014_gluster/blob/master/stat.py. Currently, the modifications done by my patch to io-stats which is under review as of now, dumps private information from the xlator object to the proper file for private info in the meta directory. This includes total bytes read/written along with read/write speed in the previous 10 seconds. The speed at every 1 second is identified by it's respective unix timestamp and hence given out in bytes/second. These values at discrete points of time can be used to generate a graph. 

The python tool first identifies all gluster mounts in the system, identifies the mount path and parses the meta xlator output in order to generate output similar to the iostat tool. Passing '-j' option gives you extra information in a consumable json format. By default, the tool pretty prints the basic stats which are human readable. This tool is supposed to be a framework on which other applications can be built upon. I've currently put this out in the gluster devel ML for community feedback so as to improve it further.

Note: In order to test this, you need to apply my patch(http://review.gluster.org/#/c/8030/) in your repo first, build and then mount a volume. Preferably perform a big read/write operation with a file on your Gluster mount before executing the python script. Then run it as 'python stat.py' or 'python stat.py -j'

Quoting my mentor Krishnan from our latest weekly hangout, "This is not a coding contest". What he meant was that just writing the code and pushing it is not the essence of open source development. I still need to interact and gain more feedback from the community regarding the work I've done till now, since our aim is not to complete the project for just the sake of doing it, but to build something that people actually use.

Tuesday, 10 June 2014

GSOC Week 3: Patches Ahoy!!

This week, I focused on modifying the io-stats xlator so that it has the capability to store the speeds of recent reads and writes. Another information needed by my tool, i.e the amount of data read and written is stored in the private section of the xlator object(this->private). The meta xlator has a feature to custom dump the private info of an xlator, but for that it requires an initialized dumpops structure just like the fops one.

So for io-stats, I initialized dumpops and added the definition of the custom dump function(.priv) which is called by meta when doing a `cat profile` in the .meta folder. In order to store the read/write speeds, two separate agnostic doubly linked lists are used which store a max of 10 elements at a time. Each element represents a 1 second interval and stores the data in bytes read/wriiten during that duration and this element is uniquely identified by their respective unix timestamps(seconds). The following read_speed and write_speed fields in the file represent speed in bytes/sec for an interval of 1 sec which can be identified by the unix timestamp in the parentheses.
/mnt/.meta/graphs/active/newvol 
[root@myfedora newvol]# cat private
write_speed(1402602084) = 12552555
write_speed(1402602085) = 18558756
write_speed(1402602086) = 23685425
write_speed(1402602087) = 9786084
write_speed(1402602088) = 9543367
write_speed(1402602089) = 796957833
write_speed(1402602090) = 8530576722
write_speed(1402602091) = 10028056272
write_speed(1402602092) = 10719120525
write_speed(1402602093) = 10528354767
read_speed(1402602084) = 8961522
read_speed(1402602085) = 8082654
read_speed(1402602086) = 7617477
read_speed(1402602087) = 9810846
read_speed(1402602088) = 10258556
read_speed(1402602089) = 193668615
read_speed(1402602090) = 261608047
read_speed(1402602091) = 29639965
read_speed(1402602092) = 47595000
read_speed(1402602093) = 39282929
data_read_cumulative = 729737216
data_read_incremental = 729737216
data_written_cumulative = 729737216
data_written_incremental = 729737216

This patch is under review as of now. It currently produces a successful build with rpms and smoke tests with Jenkins . Also, the new rackspace regression test gives a success build too.    

Tuesday, 3 June 2014

GSOC Week 2 : Bootstrapping

Based on recent discussions with my mentor KP, we've decided that instead of looking at the bigger picture of modifying multiple xlators, it's a better alternative to stick to the primary objectives of my proposal and start working on a 0.1 version of the application so as to present it soon in front of the Gluster community. Feedback is very important for our application since it's intended users will be the developers and Gluster users.

I've decided to go the python way for building glusterfsiostat. The further tasks needed to build this kind of thing are:
  1. Get location of every gluster mount in the system. This will be done with the help of the  mount command output by applying a bit of regex parsing. For now, I'll only be focusing on Unix based mount paths.
  2. Every glusterfs mount path contains a virtual directory called .meta which stores the meta information(name, type, structure of graph, latency) for every xlator. Reading the files in .meta will be my primary source of information. As of now, I could only find latency information for each of the 49 fops defined in Gluster, but in order to display total read/write amount and speed like the tool nfsiostat, I may need to modify the meta xlator itself.
  3. Pretty print this output with the application, or in a consumable form like JSON.
This initial version of the application might undergo many changes based on feedback from the community, and I'm ready for that because we're looking for a long haul!!