Delivering cron and logwatch emails to gmail in RHEL

One of my servers has no real DNS name, there is no MX record and I can hardly confiture postfix for e-mail delivery. While relaying e-mails to another SMTP server is an option, it’s actually not needed to configure MTA in order to get emails delivered from cron and logwatch. One can use MUA called mailx to do the job.

The goal is to avoid more complex configuration of postfix or (jeeeez) sendmail, so this is an alternative approach. I am not telling you this is the best thing you should do. It just works for few of my Linux servers. This will work both on RHEL6 and RHEL7 and probably even older or newer versions. And of course CentOS as well.

Vixie cron, the default cron in RHEL, uses sendmail command for all emails. This is actually part of postfix package, delivery is handled by the MTA which I actually wanted to avoid. In this tutorial, we will configure vixie cron in RHEL7 to send e-mails via mailx user agent. First of all, get mailx installed:

# yum -y install mailx

Then edit either /etc/mail.rc or /root/.mailrc as follows:

# cat /root/.mailrc
set name="Server1234"
set from="username@gmail.com"
set smtp=smtps://smtp.gmail.com
set smtp-auth=login
set smtp-auth-user=username@gmail.com
set smtp-auth-password=mysecretpassword
set ssl-verify=ignore
set nss-config-dir=/etc/pki/nssdb

Make sure that from address is same as smtp-auth-user address, gmail servers will insist on this. Server certificate is ignored, you may want to install it into the NSS database. We are ready to send a test e-mail:

# mailx -r username@gmail.com username@gmail.com
Subject: Test

This is a test
.
EOT

There will be a warning on the standard error about unkonwn certificate. I suggest to put google server CA into the NSS database, but it’s harmless and you can keep it as is if you don’t mind man-in-the-middle.

Error in certificate: Peer's certificate issuer is not recognized.

Now, create a wrapper script that will explicitly set from and to addresses:

# cat /usr/local/sbin/mailx-r
#!/bin/sh
exec mailx -r username@gmail.com username@gmail.com

Make sure the script is executable. Finally, set it via crond command line option:

# cat /etc/sysconfig/crond
CRONDARGS="-m /usr/local/sbin/mailx-r"

And restart crond:

# service crond restart

You are now receiving e-mails from cron, congratulations. The next step I usually do is installing logwatch. Since it also uses sendmail we want to disable it and run it manually from our own cron script feeding the output to the mailx command:

# yum -y install logwatch

Disable the built-in daily report because this one uses sendmail. While it could be possible to change this via some configuration option, I actually like my very own cron script.

# cat /etc/logwatch/conf/logwatch.conf
DailyReport = no

Now, create your own script and feed the output into mailx. For a weekly report do something like:

# cat /etc/cron.weekly/logwatch
#!/bin/bash
logwatch --print --range 'between -7 days and today' | mailx -s "Logwatch from XYZ" -r username@gmail.com username@gmail.com 2>/dev/null

For a daily report do this:

# cat /etc/cron.daily/logwatch
#!/bin/bash
logwatch --print --range yesterday | mailx -s "Logwatch from XYZ" -r username@gmail.com username@gmail.com 2>/dev/null

Make sure the cron script is executable and test it first. That’s all, enjoy all the e-mails!

Switching to Universal Ctags

After some time spent with exuberant-ctags and ripper-tags, I am switching over to universal-ctags. Ruby language and other languages like Golang is vastly improved over to exuberant-ctags and even ripper-tags. Here are my git hooks which I use for all my project checkouts. It’s fast, fluent and works across all my projects.

For navigation I stick with Vim, sometimes I use fzf.vim plugin together with fzf for terminal.

For longer coding sessions, I am trying GNOME Builder which looks great and every single release it is catching up with Submlime Text 3, except it is fully open source and it has usable Vim emulation. I’ve tried Atom and VSCode but these slow things are not for me.

Capturing Ruby backtrace in Sublime Text 3

I am a heavy Vim user for over two decades now, but from time to time I want to give break to my hands and just browse code with mouse while doing reviews. This is where Vim is not good at, I’ve never felt in love with NERDTree plugin or these kind of stuff. Directory tree simply doesn’t look good in terminal. From all the modern editors like Textmate, Atom, VSCode and similar I like Sublime Text the most.

Being able to execute Ruby code or tests from Sublime Text is nice, it provides a regular expression parser to make backtrace “clickable”. It comes with definition for Ruby: https://github.com/sublimehq/Packages/blob/master/Ruby/Ruby.sublime-build

{
  "shell_cmd": "ruby \"$file\"",
  "file_regex": "^(...*?):([0-9]*):?([0-9]*)",
  "selector": "source.ruby"
}

The thing is - it does not work in most cases.

There are multiple backtrace formatting in Ruby. In versions 1.8 - 2.4 the format is:

test.rb:7:in `partridges': Shouldn't this be recursive? (RuntimeError)
  from test.rb:3:in `turtle_doves'
  from test.rb:10:in `<main>'

Starting from Ruby 2.5 it’s reversed with index numbers:

Traceback (most recent call last):
  2: from test.rb:10:in `<main>'
  1: from test.rb:3:in `turtle_doves'
test.rb:7:in `partridges': Shouldn't this be recursive? (RuntimeError)

The regular expression in Sublime Text 3 (build from summer 2018) does only capture line 7 and completly ignores “from” lines. But there’s more, exceptions are often printed to console via backtrace call from Exception class from Ruby standard library. An example from our application:

E, [2017-01-16T12:15:59.678835 #32142] ERROR -- : undefined method `join' for #<String:0x007f1a587784a8> (NoMethodError)
/usr/share/sinatra-1.3.5/lib/sinatra/showexceptions.rb:37:in `rescue in call'
/usr/share/sinatra-1.3.5/lib/sinatra/showexceptions.rb:21:in `call'
/usr/share/sinatra-1.3.5/lib/sinatra/base.rb:124:in `call'
...

Testing frameworks does the same often prepending whitespace (tab usually) for each line:

NoMethodError: undefined method `render' for "test":String
    app/models/host/base.rb:351:in `render_template'
    /home/lzap/work/foreman_discovery/lock_templates.rb:14:in `block in lock_templates'
    /home/lzap/work/foreman_discovery/lock_templates.rb:10:in `each'
    /home/lzap/work/foreman_discovery/lock_templates.rb:10:in `lock_templates'

Sublime Text needs a regular expression which captures them all. I had to find how to do non-capturing group in Sublime regular parser and the rest was easy:

{
  "shell_cmd": "ruby \"$file\"",
  "file_regex": "^\\s*(?:\\d:)?\\s*(?:from )?([^:]+):(\\d+):in",
  "selector": "source.ruby"
}

I filed a PR in the Sublime Packages repo with the change: https://github.com/sublimehq/Packages/pull/1706

That should do it. Have fun!

Tracing Ruby apps with PCP

PCP offers two APIs for instrumented applications. The first one to mention is MMV agent which uses memory mapped files for capturing high-resolution data with minimum performance impact. Currently available languages for MMV instrumentation include C/C++, Python and Perl plus native Java,Golang and Rust ports. A second agent and approach is called PMDA trace with its higher level API. It uses TCP sockets and a simple API for capturing time spent, counters, trace points and raw value observations.

The tracing API is not ideal for measuring time spent in processing web requests, but it can still be useful for tracing things like cron jobs. The API (C, Fortran and Java) is described in pmdatrace(3) man page and it is trivial, therefore I decided to create a simple Ruby wrapper which only took one evening. The wrapper offers one-to-one mapping of all functions in Ruby and also higher-level Ruby approach with blocks and more user-friendly method naming.

The first step is to install trace PMDA (agent) and pcptrace rubygem (RPM not yet available, include files and compiler required to build the rubygem). Make sure that firewall is not blocking TCP port 4323 on localhost which is used to send data from the application. Also when using SELinux the pmcd daemon will be blocked from binding, therefore it is necessary to enable boolean flag:

setsebool -P pcp_bind_all_unreserved_ports 1

Then install and configure necessary software:

yum -y install pcp-pmda-trace pcp-devel @development-tools
cd /var/lib/pcp/pmdas/trace
./Install
gem install pcptrace

Use pcp_pmcd_selinux(8) man page for more details about SELinux booleans for PCP. If you encounter any SELinux problem with PCP, please let the PCP maintainers know as they will fix it promptly (pcp-team@redhat.com, or open a bugzilla).  Alternatively, the workaround is to put PCP daemons into permissive mode keeping the rest of the system confined:

semanage permissive -a pcp_pmcd_t

Using the trace API is fairly straightforward, each function returns a status code (integer), if non-zero function pmtraceerrstr can be used to find error message if needed:

cat ruby_trace_example.rb
#!/usr/bin/env ruby

require "pcptrace"

# reached a point in source code
PCPTrace::point("a_point")

# observation of an arbitrary value
PCPTrace::obs("an_observation", 130.513)

# a counter - increasing or decreasing
PCPTrace::counter("a_counter", 1)

# time spent in a transaction (or block)
PCPTrace::begin("a_transaction")
# ...
PCPTrace::end("a_transaction")

# transactions must be aborted (e.g. exception)
#PCPTrace::abort("a_transaction")

# all methods return non-zero code
result = PCPTrace::counter("a_counter", -5))
puts("Error: " + PCPTrace::errstr(result)) if result != 0

There is also more Ruby-friendly API available, see README file for more info: https://github.com/lzap/ruby-pcptrace

Tracing metrics are available in trace namespace:

pminfo trace

The trace agent uses a rolling window technique to calculate rate, total, average, min and max values for some metrics, which is very useful feature for tracing durations. By default the average is recomputed every five seconds for a period covering the prior 60 seconds. To see trace counts converted to count per second:

pmval trace.point.count

PCP also provides rate for each individual metric, let’s view that with three digits precision:

pmval -f3 trace.point.rate

Execute similar commands or use other tools like pmchart to see values for counter or observation (trace.observe.value, trace.counter.value) or to see count and rate (trace.transact.count, trace.transact.rate).

Transaction metrics (time spent) provide total_time value (trace.transact.total_time) but also ave_time, min_time and max_time aggregated values. These are quite handy for ad-hoc troubleshooting.

There is also a helper utility pmtrace for emitting tracing events from scripts like cron jobs. Although the trace API is limited and multiple syscalls are being made every measurement, it is a good starting point to go further.

On generating kickstart passwords

Although Fedora or RHEL kickstart accept root password in plain text, it’s not bad idea at all to encrypt them using standard Linux PAM mechanism. There used to be a utility which is no longer available. So I wrote a small script which does that. It is in Fedora 27+ now and on its way to EPEL7. Here is how to use it:

$ man pwkickstart

NAME

pwkickstart - generate kickstart passwords

SYNOPSIS

  # pwkickstart
  Password:

  # md5
  rootpw --iscrypted $1$SI$D964QyK1.Iz/
  # sha256
  rootpw --iscrypted $5$Ll/Q6$SEYqHGso3maNbtBc1wjyiyr2
  # sha512
  rootpw --iscrypted $6$MN7wS$1QE3FmSBrN71tXV8y.Blif1avdhTYt/

DESCRIPTION

Utility pwkickstart generates kickstart passwords from password input and
prints them to standard output in three different formats: md5, sha256 and
sha512.

Previously grub-crypt provided the same, but it is not longer available in
some Linux distributions.

The output in section was shortened to fit on screen in the SYNOPOSIS example
above.

LICENSE

MIT License

Or you can grab it from my github repo.

Have fun!