Hidden feature of Fedora 24: Live PXE boot

Some time ago, I was trying to PXE-boot Fedora 20 LiveCD directly as a squashfs filesystem from TFTP server. It was not working, because one of the dracut modules was not included on the CD/DVD init RAM disk. I filed a bug, some time passed and it’s finally fixed.

Required steps to achieve Live Fedora from PXE:

• download Fedora 24 Live DVD (or older)
• extract squashfs.img
• put it on HTTP site (/var/www/htdocs)
• extract vmlinuz and initrd.img
• install TFTP server
• put them into TFTP folder (subfolder boot in my case)
• setup DHCP and PXE
• deploy PXELinux configuration (below)
• profit!

Example PXELinux configuration:

default menu
menu title PXE
prompt 0
timeout 200
ontimeout local

label local
  menu label ^Boot from local drive
  menu default
  localboot 0

label fedora-live
  menu label Fedora Workstation LiveBoot
  kernel boot/fedora-live/vmlinuz
  append initrd=boot/fedora-live/initrd.img root=live:http://nas.home.lan/xxx/squashfs.img ro rd.live.image rd.luks=0 rd.md=0 rd.dm=0

Boot Fedora Live from your network, install it from your network when your crying friends show up with their laptops upgraded to Windows 10. Do it like a boss. End of transmission.

Probing Ruby 2.0 apps with SystemTap in RHEL7

Few years ago, I wrote an article about SystemTap and Ruby in RHEL6. When RHEL 7.0 was released, things changed. It has Ruby 2.0 and the Ruby SystemTap API changed as well, therefore I am updating my old article today according to new changes.

Imagine you have a Ruby application that has some performance issues on a production server and it’s running RHEL 7.0 or newer. With SystemTap, you can easily peek into the running application and investigate bottlenecks or count memory objects. If you know DTrace from other operating systems, welcome home.

Installation of SystemTap is easy and straightforward and for our purposes we do not need to install kernel devel and debug info packages.

# yum -y install systemtap systemtap-runtime ruby

Let’s create a trivial application called factorial.rb.

# cat factorial.rb
def factorial n
  f = 1; for i in 1..n; f *= i; end; f
end
puts factorial ARGV[1].to_i

And a simple SystemTap script that shows method calls:

# cat rubycalls.stp
probe ruby.cmethod.entry
{
  if (file == "factorial.rb") {
    printf("%s => %s.%s in %s:%d\n", thread_indent(1), classname, methodname, file, line);
  }
}
probe ruby.cmethod.return
{
  if (file == "factorial.rb") {
    printf("%s <= %s.%s in %s:%d\n", thread_indent(-1), classname, methodname, file, line);
  }
}

Let’s just run it for now.)

# stap rubycalls.stp -c "ruby factorial.rb 4"
24
 0 ruby(18989): => IO.set_encoding in factorial.rb:0
 6 ruby(18989): <= IO.set_encoding in factorial.rb:0
 0 ruby(18989): => IO.set_encoding in factorial.rb:0
 5 ruby(18989): <= IO.set_encoding in factorial.rb:0
 0 ruby(18989): => #<Class:0x00000000af8c78>.core#define_method in factorial.rb:1
 5 ruby(18989):  => Module.method_added in factorial.rb:1
 8 ruby(18989):  <= Module.method_added in factorial.rb:1
39 ruby(18989): <= #<Class:0x00000000af8c78>.core#define_method in factorial.rb:1
 0 ruby(18989): => Range.each in factorial.rb:2
 4 ruby(18989): <= Range.each in factorial.rb:2
 0 ruby(18989): => Kernel.puts in factorial.rb:4
 6 ruby(18989):  => IO.puts in factorial.rb:4
14 ruby(18989):   => Fixnum.to_s in factorial.rb:4
17 ruby(18989):   <= Fixnum.to_s in factorial.rb:4
21 ruby(18989):   => IO.write in factorial.rb:4
39 ruby(18989):   <= IO.write in factorial.rb:4
42 ruby(18989):   => IO.write in factorial.rb:4
47 ruby(18989):   <= IO.write in factorial.rb:4
49 ruby(18989):  <= IO.puts in factorial.rb:4
51 ruby(18989): <= Kernel.puts in factorial.rb:4

Please note you have to run SystemTap under root account and also expect the first run to be a little bit slower, because SystemTap is compiling and inserting a kernel module under the hood.

SystemTap syntax is similar to C and the easiest way of learning it is reading SystemTap Beginners Guide. From the book:

SystemTap allows users to write and reuse simple scripts to deeply examine the activities of a running Linux system. These scripts can be designed to extract data, filter it, and summarize it quickly (and safely), enabling the diagnosis of complex performance (or even functional) problems.

The essential idea behind a SystemTap script is to name events, and to give them handlers. When SystemTap runs the script, SystemTap monitors for the event; once the event occurs, the Linux kernel then runs the handler as a quick sub-routine, then resumes.

There are several kinds of events; entering or exiting a function, timer expiration, session termination, etc. A handler is a series of script language statements that specify the work to be done whenever the event occurs. This work normally includes extracting data from the event context, storing them into internal variables, and printing results.

The following example will count method calls to quickly search for bottlenecks. If you don’t understand how it works, head over to the Beginners Guide for more details.

# cat rubystack.stp
#!/usr/bin/stap·

global fn_calls;

probe ruby.cmethod.entry
{
  fn_calls[classname, methodname] <<< 1;
}

probe end {
  foreach ([classname, methodname] in fn_calls- limit 30) {
    printf("%dx %s.%s\n", @count(fn_calls[classname, methodname]), classname, methodname);
  }

  delete fn_calls;
}

Everytime a Ruby method is entered, counter is incremented by one in a global associative array. It prints top thirty counters on exit. When we run it, it’s a surprise!

# stap rubystack.stp -c "ruby factorial.rb 42"
1405006117752879898543142606244511569936384000000000
2823x Module.===
1728x BasicObject.==
1728x Kernel.===
997x Symbol.to_s
980x Kernel.initialize_dup
979x Kernel.dup
960x Kernel.instance_variable_set
683x Hash.[]=
676x String.to_s
499x String.initialize_copy
445x Class.new
386x String.gsub
384x Array.initialize_copy
327x Kernel.class
320x Module.method_added
315x Array.each
286x Array.flatten
260x File.file?
242x #<Class:0x00000002402d90>.core#define_method
236x String.<=>
220x String.to_i
204x Enumerable.any?
179x File.join
179x String.strip
177x Regexp.=~
164x Kernel.respond_to?
164x Kernel.untaint
156x String.=~
149x Array.last
146x Array.compact!

I’d expect multiplying operation (Bignum.*) but we see equality of module instead. Since we count all the method calls, we can see what Ruby needs to done in the backround to load such a trivial example. It’s actually rubygems gem which ships with Ruby 2.0 that does the loading mechanics (and it’s poorly designed in my opinion).

Anyway, if you want to see the bignum thing, increase the parameter from 42 to let’s say 500. Now another example, slightly modified example from the SystemTap Wiki:

# cat rubytop.stp
#!/usr/bin/stap

global fn_calls;

probe ruby.cmethod.entry
{
  fn_calls[pid(), file, methodname, line] <<< 1;
}

probe timer.ms(1000) {
  ansi_clear_screen()
  printf("%6s %30s %6s %30s %6s\n", "PID", "FILENAME", "LINE", "FUNCTION", "CALLS")
  foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) {
    printf("%6d %30s %6d %30s %6d\n", pid, filename, lineno, funcname, @count(fn_calls[pid, filename, funcname, lineno]));
  }
  delete fn_calls;
}

Now run the example with huge input that will cause it to loop for some time:

# stap rubytop.stp -c "ruby factorial.rb 999999999"

You should see a top-like screen which refreshes every second. The initial page will be full of method calls while from second one you will only see increasing counter of the multiply method:

  PID                       FILENAME   LINE                       FUNCTION CALLS
21185                   factorial.rb      2                              * 9904

If you want to sum all calls instead, delete the delete statement on the last line of the timer probe block.

SystemTap Ruby markers in RHEL 7.0 offers the following probes:

ruby.array.create
ruby.cmethod.entry
ruby.cmethod.return
ruby.find.require.entry
ruby.find.require.return
ruby.gc.mark.begin
ruby.gc.mark.end
ruby.gc.sweep.begin
ruby.gc.sweep.end
ruby.hash.create
ruby.load.entry
ruby.load.return
ruby.method.entry
ruby.method.return
ruby.object.create
ruby.parse.begin
ruby.parse.end
ruby.raise
ruby.require.entry
ruby.require.return
ruby.string.create

To probe objects creation, replace ruby.cmethod.entry with ruby.object.create, delete the delete fn_calls statement and reload. It is also possible to attach to existing process:

# stap rubytop.stp -x 12345

When using Software Collections, note the correct SCL enable syntax (credit to Pavel Valena from Red Hat):

# scl enable rh-ruby22 -- stap rubystack.stp -c "ruby factorial.rb 5"

That’s all for now.

Brother MFC L27000DW and Fedora 23

I bought this new device in order to have network printing and scanning and it works just fine. It was a pain to get it working tho.

Downloaded brscan4 and brscan-skey RPM packages from Brother site and fed the DNF tool with them.

Then I visited printers built-in web interface and configured it for static IP. You can do the same on your DHCP server if you prefer to.

And then - aaarghhh - the hidden trick which caused me headache:

$ brsaneconfig4 -a name=PRACOVNA model=MFC-L2700DW ip=192.168.1.105

And done! Switch over to “Brother *PRACOVNA” scanner and you can scan over IP. I use “simple-scan” which is quick and nice tool that aims for easy scanning into PDF with multiple pages support. Highly recommended.

To all incoming documents, payslips and letters - you will be all archived!

Actually I haven’t configured printing because I use Google Cloud Print which is build in the device, but it should not be that hard. Frankly, I tried but it did not work. Drop me a line in comments if you know how to do that.

Hidden gem of Fedora: git xcleaner

Fedora 22+ now provides new tool called git-xcleaner which helps deleting unused topic branches using TUI (text user interface). It also offers mechanisms for pre-selecting branches that can be safely removed.

Possible actions:

Merged

Command git branch –merged is used to find a list of branches that are marked for deletion.

Messages

For each branch, tip commit message is compared against base history and if found, the branch is marked for deletion. Whole commit message is compared and it must fully match.

User enters base branch name (defaults to “master”).

Remote

Delete all remote branches in remote repo which are not present locally.

User enters remote name (defaults to current username).

Upstream

All branches which no longer exist in origin or specific remote repository are marked for deletion.

User enters specific remote name (defaults to current username).

Manual

User manually marks branches for deletion.

Go ahead and try it:

# dnf -y install git-xcleaner
# git xcleaner

If you mis-deleted a branch and ignored all the warnings in documentation and on the screen, check out the homepage for instructions how to checkout your branch back.

And one more thing. File bugs at https://github.com/lzap/git-xcleaner/issues

Human readable name generator

Out of ideas for incoming bare-metal host names in your cluster? I wrote a little generator which contains frequently occurring given names and surnames from the 1990 US Census (public domain data):

• 256 (8 bits) unique male given names
• 256 (8 bits) unique female given names
• 65,536 (16 bits) unique surnames

Given names were filtered to be 3-5 characters long, surnames 5-8 characters, therefore generated names are never longer than 14 characters (5+1+8).

This gives 33,554,432 (25 bits) total of male and female name combinations. The generator can either generate randomized succession, or generate combinations based on MAC adresses.

Random generator

The random name generator makes use of Fibonacci linear feedback shift register which gives deterministic sequence of pseudo-random numbers. Additionally, algorithm makes sure names with same first name (or gender) and last name are not returned in succession. Since there are about 1% of such cases, there are about 33 million unique total names. Example sequence:

• velma-pratico.my.lan
• angie-warmbrod.my.lan
• grant-goodgine.my.lan
• alton-sieber.my.lan
• velma-vanbeek.my.lan
• don-otero.my.lan
• sam-hulan.my.lan

The polynomial used in linear feedback shift register is

x^25 + x^24 + x^23 + x^22 + 1.

The key thing is to store register (a number) and use it for each generation in order to get non-repeating sequence of name combinations. See example below.

MAC generator

Examples of MAC-based names:

• 24:a4:3c:ec:76:06 -> bobby-louie-sancher-weeler.my.lan
• 24:a4:3c:e3:d3:92 -> bob-louie-sancher-rimando.my.lan

MAC addresses with same OID part (24:a4:3c in this case) generates the same middle name (“Louie Sancher” in the example above), therefore it is possible to guess server (or NIC) vendor from it, or it should be possible to shorten middle names (e.g. bobby-ls-weeler.my.lan) in homogeneous environments.

Comparison of types

MAC-based advantages

• reprovisioning the same server generates the same name
• middle names are same for unique hardware vendors

MAC-based disadvantages

• name is longer

Random-based advantages

• name is shorter

Random-based disadvantages

• reprovisioning the same server generates different name

Usage

Visit project homepage to see some examples. Also try the Foreman which will (hopefully) have a embedded generator based on Deacon library in the 1.12 release.

Why the?

Deacon is a ministry in the Christian Church that is generally associated with service of some kind, but which varies among theological and denominational traditions. In many traditions the “diaconate” (or deaconate), the term for a deacon’s office, is a clerical office; in others it is for laity. – Wikipedia