Jan 5, 2012

Probing Ruby apps with SystemTap in RHEL

There is one small'n'hidden feature of RHEL 6.2 that popped up recently as an enhancement errata (RHSA-2011-1581). With this update, SystemTap probes have been added to the Ruby package. I was waiting for this feature, because at the time I was reading Aaron Patterson's excelent article about Ruby probing, this was not possible in Fedora or Red Hat Enterprise Linux by default. I mean without compilation and installation of DTrace or SystemTap.

Imagine you have a Ruby application that has some performance issues on a production server and it's running RHEL 6.2 or newer. Now it is possible to perform very detailed probing using SystemTap, which is similar technology to DTrace supported and developed by Red Hat.

It is possible to create probing scripts that are compiled as kernel modules and start them even on production servers without any change of running applications. It is possible to "look" into running applications and search for bottlenecks. Both DTrace and SystemTap are fantastic technologies.

Installation of SystemTap is easy and straightforward and for our purposes we do not need to install kernel devel and debug info packages.

# yum -y install systemtap systemtap-runtime ruby

If you plan to trace kernel and other low level stuff, pay a visit the SystemTap Beginners Guide, but this is quite enough for us.

Before we start, we will create a small Ruby example. In real situations you will be probing complex applications like Ruby on Rails or Sinatra. Let's call it factorial.rb.

# cat factorial.rb
def factorial n
  f = 1; for i in 1..n; f *= i; end; f
end
puts factorial 42

The first example is the easiest, actually taken from the SystemTap wiki. It it able to print all Ruby method calls. Create a file called calls.stp with the following content:

# cat calls.stp 
probe ruby.function.entry
{
  printf("%s => %s.%s in %s:%d\n", thread_indent(1),
         classname, methodname, file, line);
}
probe ruby.function.return
{
  printf("%s <= %s.%s in %s:%d\n", thread_indent(-1),
         classname, methodname, file, line);
}

Let's have some fun now.

# sudo stap calls.stp -c "ruby factorial.rb"
1405006117752879898543142606244511569936384000000000
     0 ruby(16160): => Module.method_added in factorial.rb:1
    13 ruby(16160): <= Module.method_added in factorial.rb:1
     0 ruby(16160): => Object.factorial in factorial.rb:5
    25 ruby(16160):  => Range.each in factorial.rb:2
    61 ruby(16160):   => Fixnum.* in factorial.rb:2
    ... about twenty lines removed ...
   705 ruby(16160):   <= Bignum.* in factorial.rb:2
   712 ruby(16160):  <= Range.each in factorial.rb:2
   718 ruby(16160): <= Object.factorial in factorial.rb:2
     0 ruby(16160): => Object.puts in factorial.rb:5
    20 ruby(16160):  => Bignum.to_s in factorial.rb:5
    38 ruby(16160):  <= Bignum.to_s in factorial.rb:5
    53 ruby(16160):  => IO.write in factorial.rb:5
    74 ruby(16160):  <= IO.write in factorial.rb:5
    81 ruby(16160):  => IO.write in factorial.rb:5
    99 ruby(16160):  <= IO.write in factorial.rb:5
   106 ruby(16160): <= Object.puts in factorial.rb:5

Please note you have to run SystemTap under root account and also expect the first run to be a little bit slower, because SystemTap is compiling and inserting a kernel module under the hood.

By the way the big number (1405006117752879898543142606244511569936384000000000) is the actual output of our Ruby script. Yes, it's the world-famous 42! By the way it does not worth googling it - nothing special is found.

What about to count method calls now? I prepared a short script for that. If you are interested in the SystemTap syntax, google around or read whole Beginners Guide.

# cat rubycount.stp 
#!/usr/bin/stap 

global fn_calls;

probe ruby.function.entry
{ 
  fn_calls[classname, methodname] <<< 1;
}

probe end {
  foreach ([classname, methodname] in fn_calls- limit 30) {
    printf("%dx %s.%s\n",
        @count(fn_calls[classname, methodname]),
        classname, methodname);
  }

  delete fn_calls;
}

As you can see, it only counts all method calls and prints them when program ends. The output is much more useful.

# sudo stap rubycount.stp -c "ruby factorial.rb"
1405006117752879898543142606244511569936384000000000
21x Bignum.*
21x Fixnum.*
2x IO.write
1x Module.method_added
1x Range.each
1x Bignum.to_s
1x Object.puts
1x Object.factorial

We have a catch! The need to optimize multiply methods :-)

Let's be serious again. Do you want to see more? I offer nothing more than a recommendation for you. Write your own scripts. Probe anything you want. You take a deep look on Ruby classes, methods, files, lines, garbage collector, exceptions and user defined probe points. SystemTap is very flexible and easy to use.

I have one more script for you, which can be downloaded from the wiki. It's called rubyfuntop.stp and it is a comfortable way of watching guts of Ruby processes. Something like "top", but with more information like methods and files. It is also a good example of live SystemTap technique. To run it just give it an exec flag:

# chmod +x rubyfuntop.stp
# sudo ./rubyfuntop.stp

Then execute the factorial in a different console.

For your information SystemTap for Ruby did not hit Fedora yet, I hope it will be part of Ruby 1.8 as well as Ruby 1.9 that is planned for Fedora 17. Have fun!

Jan 4, 2012

First look at ThinkPad X220

As a ThinkPad X201 user, I was more than happy to check out new X220 model. At the first sitght it is still ThinkPad X-series. Similar size, feeling, quality. I am comparing my good-old X201 with this model 287-2SG.

Pros

  • The lid has no lock. Frankly, I found it annoying and didnt like it ever on all my ThinkPads.
  • Thank God the keyboard layout is standard with only few changes - bigger Escape key. Happy Vimmer.
  • Bigger Power button, smaller ThinkVantage button.
  • Much larger touchpad with macintosh-like "hidden" buttons. Even when I exclusively use trackpoint, it is very nice.
  • Quite silent fan under load, feels better.
  • Very fast BIOS, dunno what kind of technology is this, but few seconds saved everyday.
  • VGA and Display Port on the left side. At last!
  • Yeah, it has some faster components inside. And yeah, I dont really see any difference.

Cons

  • Display is not bad, but I was expecting a bit more from 12.5" IPS LED (1366x768). Standard angles.
  • I dont like bigger Delete key since Insert was moved to the left. Can live with it.
  • Power cord plug on the back side. I like it on the left more. Not a blocker.
  • Ethernet plug on the right side. Guys, most people are right-handed and ethernet cables are hard. Dont like it on this side.
  • USB ports swapped (two on the left and one on the right now).
  • Mic port is gone. Probably combined into the headphones. I am not sure how to connect my headphones and mic there.
  • Damn, incompatible UltraBay dock again! It has one more USB port, but this really sucks. Money, money, money...

Dec 29, 2011

How to get rid of GUID Partition Table

I made a default installation of Fedora 16 on my testing machine the other day and Anaconda has installed GPT on the hard drive. Now, I want to fall back to MBR. Wildly used tool fdisk was showing a message Warning!! Unsupported GPT (GUID Partition Table) detected. Use GNU Parted. Frankly, I did not like it at all.

My first idea was to get rid of the first few sectors on the hard drive:

dd if=/dev/zero of=/dev/sda bs=512 count=1

Guess what, it did not work. After some time spend on the internet I found out GPT has a "backup" entry at the very end of the disc. I tried to copy zeroes over it, but it did not work too. The trick is quite easy, but it was more difficult to google this time:


parted /dev/sda
mklabel msdos
quit

Finally, the disc is back to traditional MBR. I think I don't like new things, sometimes.

Disable animations Gnome Shell plugin now as extension

Heya, you may already noticed the new site extensions.gnome.org for Gnome Shell extensions. It is possible to install extensions with a single mouse click with the latest and greatest version.

I just put my Disable Window Animations extensions there, so you can easily install from the service.

It is available here: https://extensions.gnome.org/extension/119/disable-window-animations/

Enjoy and HNY!

Dec 7, 2011

Setting hostname properly in Fedora and Red Hat

I have been playing with hostnames yesterday and I had to stop for a while today doing a little research. Setting up a hostname is not a hard task, in Fedora and Red Hat it's a matter of editing one file:

# grep HOSTNAME /etc/sysconfig/network
HOSTNAME=test.lan

Here you set the whole FQDN. If you ever seen entries like DOMAINNAME in this file, they are wrong for Fedora or RHEL. If you want to change hostname without rebooting few steps are necessary:

# hostname test.lan
# service network restart

I skipped one simple, yet important step - setting /etc/hosts. It differs for situations when you use DHCP or not.

Clients with DHCP

With properly configured DHCP server, hostname should be propagated or defined by network administrator, there is no need of adding hostname to the /etc/hosts. For clients, it is really not necessary. Trust me. Keep only localhost there, walk your dog, enjoy life.

Servers with static IP

This is totally different scenario - server with static IP address must have entry in the /etc/hosts file in the following format (assuming a random class C address):

# grep test /etc/hosts
192.168.1.1 test.lan test

Please note the order matters. You must keep the following order: ipaddress fqdn hostname. I recommend to do a check-up after adding entries:

# hostname 
test.lan
# hostname -s
test
# hostname -f
test.lan

With the incorrect order you will see hostname -f returning non-fully qualified response:

# grep test /etc/hosts
192.168.1.1 test test.lan # incorrect order
# hostname -f
test

The key for this behavior is in the manual page of hostname command:

The function gethostname(2) is used to get the hostname. When the hostname -a, -d, -f or -i is called will gethostbyname(3) be called. The difference in gethostname(2) and gethostbyname(3) is that gethostbyname(3) is network aware, so it consults /etc/nsswitch.conf and /etc/host.conf to decide whether to read information in /etc/sysconfig/network or /etc/hosts.

Servers with DHCP

Wait a minute! A server without static IP address? Does it make sense? Well, in the cloud it is not unusual to provision servers configured with DHCP. But cloud providers usually allow customers to reserve IPs for a while so it does not change. If this is your case, I recommend to stick with the servers with static IP rule. If the IP address changes rapidly, I would recomment clients with DHCP rule, but make sure hostname command returns correct answers for both -s and -f options, as seen above.

In the latter case, contact your cloud provider asking how to set up hostname properly. Sometimes in testing environments servers with IPs delivered by DHCP servers do not have proper DNS setup. And you can have a service that insists on having hostname resolved. In this case, only in this case, I would recommend adding loop back hostname entry:

# grep test /etc/hosts
127.0.0.1 test.lan test

Again, keep the right order and make sure it does not mess up services on that box. Be prepared to see "Cannot determine hostname" warning when starting Apache for example.

Dec 2, 2011

Gpaste terminal integration

Sometimes I need to copy whole output of some program into my clipboard or selection. I use nice Gnome Shell extension called Gpaste (packages gpaste and gnome-shell-extension-gpaste in Fedora and Red Hats). It is a nice daemon, command line tool and Gnome Shell menu, everything well integrated.

And how to do that? It is easy. Just pipe it through the gpaste command, I am not kidding.

$ echo "Some very long output" | gpaste

The long test is now in your selection or clipboard (according to your gpaste configuration). That's it :-)

Nov 30, 2011

Git's crying Patch format detection failed


If you get the following error message while applying git patch:

$ git am /path/to/0001-My-awesome-change.patch
Patch format detection failed.

It means that the patch was not generated with git format-patch and transmitted correctly. Either ask the contributor to try again using the above patch creation instructions, or apply each patch separately using git apply:

$ git apply --whitespace=fix /path/to/0001-My-awesome-change.patch
$ git commit --author='Contributor Name <contributor@thedomain.com>'

If the patch does not fit, check out older version. I usually checkout a version from the same date (or hour), that usually works.

Remember; writing is learning. That's all folks. At least for today. Cheers.

Nov 28, 2011

Anatomy of Ruby Exception

Today I was trying to find how to properly override Ruby standard Exception. To my surprise, I was not able to find any tutorial or even paragraph about it. No recommendations, no design patterns. Everything was the same:

class MyError < StandardError; end

As a Java guy, I tend to encapsulate lots of things into Exceptions. It's not a bad thing, actually it is a good thing. So I asked myself - what should I override, how, and why?

Let's skip the general Ruby recommendation to override StandardError instead Exception. Different class, the very same interface. To my surprise again, both Exception and StandardError classes are defined in the Ruby C code.

So I have rewritten the Ruby Exception into this Ruby code. Please note I made my life a little easier and some of the methods do not return the exact output as Ruby does (e.g. inspect), but we get very good overview with it. Please note this is Ruby 1.8, there is one small change in 1.9 in regard to method to_str, but it makes no difference for us.

class Exception
  def initialize(msg)
    @mesg = msg
    @bt = nil
  end

  def to_s
    return self.class.name if @mesg.nil?
    @mesg
  end

  alias :message :to_s
  alias :to_str :to_s

  def set_backtrace(bt)
    @bt = bt
  end

  def backtrace
    @bt
  end

  def inspect
    "\#<#{self.class.name}>: #{to_s}"
  end

  def exception(msg)
    return self if msg == message
    self.class.new(msg)
  end
end

I think the code says it all. Now we can see what we actually need to redefine. Now, let's do a small example.

class MyError < StandardError
  attr_accessor :info

  def initialize(msg, my_additional_info = "")
    info = my_additional_info
    super(msg)
  end

  def message
    "#{to_s}: #{info}"
  end
end

As you can see from the Exception code, Ruby stores the message into instance variable called "mesg". Since this is native code, we are not able to access it. The only way to get it is to call to_s method. Please do not do this:

def to_s
  "#{message}: #{@info}"
end

This will end up with stack overflow. Special thanks to Ivan Nečas for helping me with this.

I have to admit this interface is not the best feature of Ruby. I can imagine an accessor for the message itself. And with a different name than "mesg". But that is life.

Nobody told us Ruby programming's gonna be easy. Take care.