Finding the right cost for bcrypt/pbkdf2

Foreman uses bcrypt with variable cost as the default password hashing approach, but I learned that bcrypt is not approved for passwords by NIST the other day. Before finishing my patch, I wanted to see what are the sane iteration counts for PBKDF2-HMAC-SHA algorithm which is approved by NIST. Here are results to give you rough estimation from my Intel NUC i3 8th gen running i3-8109U, a CPU from 2018:

       user     system      total        real
PBKDF2 SHA-1 with 1 iters	  0.000024   0.000007   0.000031 (  0.000028)
PBKDF2 SHA-1 with 100001 iters	  0.064736   0.000000   0.064736 (  0.064842)
PBKDF2 SHA-1 with 200001 iters	  0.129461   0.000000   0.129461 (  0.129587)
PBKDF2 SHA-1 with 300001 iters	  0.195255   0.000000   0.195255 (  0.195449)
PBKDF2 SHA-1 with 400001 iters	  0.259314   0.000000   0.259314 (  0.259565)
PBKDF2 SHA-1 with 500001 iters	  0.324826   0.000000   0.324826 (  0.325150)
PBKDF2 SHA-1 with 600001 iters	  0.388826   0.000000   0.388826 (  0.389216)
PBKDF2 SHA-1 with 700001 iters	  0.453290   0.000000   0.453290 (  0.453753)
PBKDF2 SHA-1 with 800001 iters	  0.518169   0.000000   0.518169 (  0.518704)
PBKDF2 SHA-1 with 900001 iters	  0.583946   0.000000   0.583946 (  0.584603)

       user     system      total        real
PBKDF2 SHA-256 with 1 iters	  0.000041   0.000000   0.000041 (  0.000040)
PBKDF2 SHA-256 with 100001 iters	  0.100197   0.000000   0.100197 (  0.100307)
PBKDF2 SHA-256 with 200001 iters	  0.200945   0.000000   0.200945 (  0.201145)
PBKDF2 SHA-256 with 300001 iters	  0.301221   0.000000   0.301221 (  0.301541)
PBKDF2 SHA-256 with 400001 iters	  0.402317   0.000000   0.402317 (  0.402714)
PBKDF2 SHA-256 with 500001 iters	  0.502404   0.000000   0.502404 (  0.502949)
PBKDF2 SHA-256 with 600001 iters	  0.607353   0.000000   0.607353 (  0.607923)
PBKDF2 SHA-256 with 700001 iters	  0.702174   0.000000   0.702174 (  0.702893)
PBKDF2 SHA-256 with 800001 iters	  0.804487   0.000000   0.804487 (  0.805271)
PBKDF2 SHA-256 with 900001 iters	  0.904246   0.000000   0.904246 (  0.905123)

       user     system      total        real
PBKDF2 SHA-512 with 1 iters	  0.000020   0.000000   0.000020 (  0.000020)
PBKDF2 SHA-512 with 100001 iters	  0.069437   0.000000   0.069437 (  0.069507)
PBKDF2 SHA-512 with 200001 iters	  0.138220   0.000000   0.138220 (  0.138372)
PBKDF2 SHA-512 with 300001 iters	  0.206984   0.000000   0.206984 (  0.207207)
PBKDF2 SHA-512 with 400001 iters	  0.278088   0.000000   0.278088 (  0.278450)
PBKDF2 SHA-512 with 500001 iters	  0.344130   0.000000   0.344130 (  0.344481)
PBKDF2 SHA-512 with 600001 iters	  0.413116   0.000000   0.413116 (  0.413551)
PBKDF2 SHA-512 with 700001 iters	  0.482472   0.000000   0.482472 (  0.482973)
PBKDF2 SHA-512 with 800001 iters	  0.553838   0.000000   0.553838 (  0.554417)
PBKDF2 SHA-512 with 900001 iters	  0.620699   0.000000   0.620699 (  0.621316)

This is an output from a quick benchmark written in Ruby which uses OpenSSL library from Fedora 34 with 40 bytes salt, password and output. The script is below if you want to run it. Anyway.

Say I want to target 40ms password hashing time on this class of CPU, which should be a sane default for an on-premise intranet web application. In that case, these are the recommended iteration counts:

• PBKDF2-HMAC-SHA1: 600_000 iterations
• PBKDF2-HMAC-SHA256: 400_000 iterations
• PBKDF2-HMAC-SHA512: 600_000 iterations

For roughly 20ms calculation time you can go with 250_000 iterations which should be probably a safe but reasonable minimum in 2021 for a web app.

One thing is interesting tho, SHA256 is actually slower than SHA512. I would not expect that, it looks like some padding. Or maybe an error in my benchmark or the fact the test is written in Ruby? That should not be the case because one call into OpenSSL native library is 40ms. To verify, I have rewritten the code in Crystal, an LLVM Ruby-like language which recently hit the 1.0.0 version milestone. It was pretty much copy and paste, here is the result:

                                      user     system      total        real
PBKDF2 SHA-1 with 1 iters	        0.000006   0.000020   0.000026 (  0.000022)
PBKDF2 SHA-1 with 100001 iters	   0.064808   0.000050   0.064858 (  0.064933)
PBKDF2 SHA-1 with 200001 iters	   0.129531   0.000014   0.129545 (  0.129681)
PBKDF2 SHA-1 with 300001 iters	   0.194418   0.000000   0.194418 (  0.194606)
PBKDF2 SHA-1 with 400001 iters	   0.259126   0.000000   0.259126 (  0.259377)
PBKDF2 SHA-1 with 500001 iters	   0.324371   0.000000   0.324371 (  0.324689)
PBKDF2 SHA-1 with 600001 iters	   0.389384   0.000000   0.389384 (  0.389780)
PBKDF2 SHA-1 with 700001 iters	   0.454766   0.000000   0.454766 (  0.455242)
PBKDF2 SHA-1 with 800001 iters	   0.518768   0.000000   0.518768 (  0.519265)
PBKDF2 SHA-1 with 900001 iters	   0.584092   0.000000   0.584092 (  0.584682)
                                        user     system      total        real
PBKDF2 SHA-256 with 1 iters	        0.000015   0.000000   0.000015 (  0.000015)
PBKDF2 SHA-256 with 100001 iters	   0.100220   0.000000   0.100220 (  0.100331)
PBKDF2 SHA-256 with 200001 iters	   0.200525   0.000000   0.200525 (  0.200722)
PBKDF2 SHA-256 with 300001 iters	   0.301427   0.000000   0.301427 (  0.301713)
PBKDF2 SHA-256 with 400001 iters	   0.401965   0.000000   0.401965 (  0.402360)
PBKDF2 SHA-256 with 500001 iters	   0.501238   0.000000   0.501238 (  0.501742)
PBKDF2 SHA-256 with 600001 iters	   0.605589   0.000000   0.605589 (  0.606171)
PBKDF2 SHA-256 with 700001 iters	   0.702972   0.000000   0.702972 (  0.703676)
PBKDF2 SHA-256 with 800001 iters	   0.803881   0.000000   0.803881 (  0.804708)
PBKDF2 SHA-256 with 900001 iters	   0.902646   0.000000   0.902646 (  0.903572)
                                        user     system      total        real
PBKDF2 SHA-512 with 1 iters	        0.000015   0.000000   0.000015 (  0.000014)
PBKDF2 SHA-512 with 100001 iters	   0.068897   0.000000   0.068897 (  0.068960)
PBKDF2 SHA-512 with 200001 iters	   0.137699   0.000000   0.137699 (  0.137853)
PBKDF2 SHA-512 with 300001 iters	   0.206790   0.000000   0.206790 (  0.206982)
PBKDF2 SHA-512 with 400001 iters	   0.275695   0.000000   0.275695 (  0.275972)
PBKDF2 SHA-512 with 500001 iters	   0.344550   0.000000   0.344550 (  0.344882)
PBKDF2 SHA-512 with 600001 iters	   0.412838   0.000000   0.412838 (  0.413224)
PBKDF2 SHA-512 with 700001 iters	   0.482600   0.000000   0.482600 (  0.483100)
PBKDF2 SHA-512 with 800001 iters	   0.551040   0.000000   0.551040 (  0.551562)
PBKDF2 SHA-512 with 900001 iters	   0.619910   0.000000   0.619910 (  0.620500)

It is roughly the same. Which means you can take these numbers when building non-Ruby projects (C, Go, Python) too.

For “comparison”, here is the ouput from bcrypt (from ruby-bcrypt library which uses a copy-paste implementation) from my Intel NUC i3 2018 brick:

       user     system      total        real
bcrypt with cost 6	  0.003510   0.000000   0.003510 (  0.003549)
bcrypt with cost 7	  0.006642   0.000000   0.006642 (  0.006669)
bcrypt with cost 8	  0.013120   0.000000   0.013120 (  0.013149)
bcrypt with cost 9	  0.026097   0.000000   0.026097 (  0.026154)
bcrypt with cost 10	  0.052030   0.000000   0.052030 (  0.052104)
bcrypt with cost 11	  0.103912   0.000000   0.103912 (  0.104034)
bcrypt with cost 12	  0.207882   0.000000   0.207882 (  0.208111)
bcrypt with cost 13	  0.415320   0.000000   0.415320 (  0.415777)
bcrypt with cost 14	  0.830609   0.000000   0.830609 (  0.831516)
bcrypt with cost 15	  1.660890   0.000000   1.660890 (  1.662613)
bcrypt with cost 16	  3.321980   0.000000   3.321980 (  3.325642)
bcrypt with cost 17	  6.641207   0.000000   6.641207 (  6.647944)

It has an exponential characteristic, 40ms is roughly cost 13 and 20ms is cost 12. It is still a good choice, however keep in mind that bcrypt is not available in most Linux distributions (including Red Hat Enterprise Linux) and, again, not approved by NIST.

I do have another brick: Apple Mac Mini M1 16GB from 2021, just wondering how it compares to the i3 from 2018. I will not be pasting all tests because everything was pretty much the same - Apple M1 was a tad slower in the Ruby test. Unfortunately, Crystal is not yet available for the M1 chip.

       user     system      total        real
bcrypt with cost 6	  0.004288   0.000079   0.004367 (  0.004395)
bcrypt with cost 7	  0.007669   0.000018   0.007687 (  0.007686)
bcrypt with cost 8	  0.015171   0.000076   0.015247 (  0.015254)
bcrypt with cost 9	  0.030117   0.000248   0.030365 (  0.030366)
bcrypt with cost 10	  0.060343   0.000530   0.060873 (  0.060873)
bcrypt with cost 11	  0.119880   0.000926   0.120806 (  0.120805)
bcrypt with cost 12	  0.240576   0.002135   0.242711 (  0.242947)
bcrypt with cost 13	  0.478940   0.003749   0.482689 (  0.482706)
bcrypt with cost 14	  0.957543   0.007271   0.964814 (  0.964815)
bcrypt with cost 15	  1.914573   0.014784   1.929357 (  1.929367)
bcrypt with cost 16	  3.829219   0.028846   3.858065 (  3.858162)
bcrypt with cost 17	  7.669053   0.056666   7.725719 (  7.726456)

The Ruby script:

require 'benchmark'
require 'openssl'
require 'bcrypt'

asha = "bbd2a53e6feb515d644090c4fefba1c2756cc19b"

Benchmark.bm do |bench|
  (1..1000000).step(100000).each do |iters|
    bench.report("PBKDF2 SHA-1 with #{iters} iters\t") do
      OpenSSL::PKCS5.pbkdf2_hmac_sha1(asha, asha, iters, 40)
    end
  end
end

Benchmark.bm do |bench|
  (1..1000000).step(100000).each do |iters|
    bench.report("PBKDF2 SHA-256 with #{iters} iters\t") do
      OpenSSL::PKCS5.pbkdf2_hmac(asha, asha, iters, 40, OpenSSL::Digest.new("SHA256"))
    end
  end
end

Benchmark.bm do |bench|
  (1..1000000).step(100000).each do |iters|
    bench.report("PBKDF2 SHA-512 with #{iters} iters\t") do
      OpenSSL::PKCS5.pbkdf2_hmac(asha, asha, iters, 40, OpenSSL::Digest.new("SHA512"))
    end
  end
end

Benchmark.bm do |bench|
  (6..17).each do |cost|
    bench.report("bcrypt with cost #{cost}\t") do
      BCrypt::Password.create(asha, cost: cost)
    end
  end
end

The same script in Crystal language:

require "benchmark"
require "openssl"

asha = "bbd2a53e6feb515d644090c4fefba1c2756cc19b"

Benchmark.bm do |bench|
  (1..1000000).step(100000).each do |iters|
    bench.report("PBKDF2 SHA-1 with #{iters} iters\t") do
      OpenSSL::PKCS5.pbkdf2_hmac_sha1(asha, asha, iters, 40)
    end
  end
end

Benchmark.bm do |bench|
  (1..1000000).step(100000).each do |iters|
    bench.report("PBKDF2 SHA-256 with #{iters} iters\t") do
      OpenSSL::PKCS5.pbkdf2_hmac(asha, asha, iters, OpenSSL::Algorithm::SHA256, 40)
    end
  end
end

Benchmark.bm do |bench|
  (1..1000000).step(100000).each do |iters|
    bench.report("PBKDF2 SHA-512 with #{iters} iters\t") do
      OpenSSL::PKCS5.pbkdf2_hmac(asha, asha, iters, OpenSSL::Algorithm::SHA512, 40)
    end
  end
end

Well, there you have it. Drop me a comment on twitter @lzap and have a nice day!

Remap US key next to enter to enter in MacOS

The Czech keyboard layout on a physical US Mac keyboard has some keys which are pretty much useless. For example the key | aka \ next to the enter is actually also available on tilde key next to left shift in Czech layout and since I am used to wide enter key I end up pressing it when I want to hit enter. It renders to a weird “double tilde” character which I never use anyway. Well, an easy help. This can be remapped pretty easily:

hidutil property --set '{"UserKeyMapping":
    [{"HIDKeyboardModifierMappingSrc":0x700000031,
      "HIDKeyboardModifierMappingDst":0x700000058}]
}'

That’s all, really. No need to restart anything, but to do this after each boot a property list for launcher must be created. Here it is:

cat << EOF | sudo tee -a /Library/LaunchDaemons/org.custom.keyboard-remap.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>org.custom.keyboard-remap</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/bin/hidutil</string>
      <string>property</string>
      <string>--set</string>
      <string>{"UserKeyMapping": [{"HIDKeyboardModifierMappingSrc":0x700000031, "HIDKeyboardModifierMappingDst":0x700000058}] }</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <false/>
  </dict>
</plist>
EOF
sudo launchctl load -w /Library/LaunchDaemons/org.custom.keyboard-remap.plist

Worry do not, this is still a Linux blog. I just ended up using MacOS on desktop a bit more lately, I still run Linux remote shells :-)

Crop and resize video to get rid of borders

We stream our community demos on youtube via Google Meet and there are borders on each side which makes the content to be smaller and less readable. Luckily, it is in the middle of the screen, so the following command will crop the image to its 1/1.29 of the size, stretch it back to 720p and reencodes it for YouTube copying the audio stream.

ffmpeg -i intput.mp4 -vf "crop=iw/1.29:ih/1.29,scale=-1:720" -y output.mp4

If your source video is different, just play around with the 1.29 constant to get the desired output. Use -t option to encode just the first 10 seconds of the video to speed testing up:

ffmpeg -i intput.mp4 -vf "crop=iw/1.29:ih/1.29,scale=-1:720" -t 00:00:10.0 -y output.mp4

That is all.

FreeIPA and Foreman Proxy development setup

I have been avoiding this for like ten years now, but today is the day when I will setup a FreeIPA with Foreman Proxy for development and testing purposes and here are my notes.

The goal is to deploy a libvirt VM with IPA server and Foreman Proxy intergated with it. The domain will be ipa.lan and the host named ipa.ipa.lan. This is NOT how you should deploy production Foreman FreeIPA integration! For that, reading our official documentation and using foreman-installer is suggested instead.

We need a VM, let’s go with CentOS 8.

virt-builder centos-8.2 --output /var/lib/libvirt/images/ipa.img --root-password password:redhat --hostname ipa.ipa.lan
virt-install --name ipa.ipa.lan --memory 2048 --vcpus 2 --disk /var/lib/libvirt/images/ipa.img --import --os-variant rhel8.3 --update
virsh console ipa.ipa.lan

We need a static IP for this VM:

nmcli con modify enp1s0 \
  ip4 192.168.122.5/24 \
  gw4 192.168.122.1 \
  ipv4.dns 192.168.122.1
nmcli con down enp1s0
nmcli con up enp1s0

Make sure the hostname is correct:

hostnamectl set-hostname ipa.ipa.lan

Make sure to fix hosts file, FQDN must resolve to the IP address not localhost:

grep ipa /etc/hosts
192.168.122.5 ipa.ipa.lan ipa

The installation is very smooth, expect just couple of questions like administrator password or the actual domain:

dnf module enable idm:DL1
dnf module install idm:DL1/dns
ipa-server-install --setup-dns --auto-forwarder --auto-reverse

Ensure firewall ports are enabled:

firewall-cmd --add-service=http --add-service=https --add-service=ldap --add-service=ldaps \
    --add-service=ntp --add-service=kerberos --add-service=dns --add-port=8000/tcp --permanent

Next up, install Foreman Proxy:

dnf -y install https://yum.theforeman.org/releases/2.4/el8/x86_64/foreman-release.rpm
dnf -y install foreman-proxy

Create the foreman user with minimum required permissions to manage Foreman hosts, create and configure keytab file. When asked for admin password, use the one used when installing the IPA server:

foreman-prepare-realm admin realm-smart-proxy
mv freeipa.keytab /etc/foreman-proxy/freeipa.keytab
chown foreman-proxy:foreman-proxy /etc/foreman-proxy/freeipa.keytab

Configure and start the Foreman Proxy service. This is for development purposes, so let’s only use HTTP. You may also want to add some trusted_hosts entries to allow access from Foreman:

cat /etc/foreman-proxy/settings.yml
---
:settings_directory: /etc/foreman-proxy/settings.d
:http_port: 8000
:log_level: DEBUG

Enable Realm module:

cat /etc/foreman-proxy/settings.d/realm.yml
---
:enabled: true
:use_provider: realm_freeipa

And enable FreeIPA plugin:

cat /etc/foreman-proxy/settings.d/realm_freeipa.yml
---
:keytab_path: /etc/foreman-proxy/freeipa.keytab
:principal: realm-smart-proxy@IPA.LAN
:ipa_config: /etc/ipa/default.conf
:remove_dns: true
:verify_ca: true

And start it up:

systemctl enable --now foreman-proxy

Realm feature should be available:

curl http://ipa.ipa.lan:8000/features
["realm"]

To show a host entry in IPA via CLI:

kinit admin
ipa host-show rex-dzurnak.ipa.lan
  Host name: rex-dzurnak.ipa.lan
  Class: ipa-debian-10
  Password: True
  Keytab: False
  Managed by: rex-dzurnak.ipa.lan

Add the foreman proxy into Foreman and start developing or testing. Have fun!

The Lounge web IRC client in Fedora

My graphics card died and thanks to COVID and Bitcoin, it will be a long wait until it’s back. I am on Mac M1 at the moment and it looks like there are not many good IRC clients on MacOS.

Let’s run a simple web-based IRC client which can also work as a bouncer (no need of ZNC). I randomly selected one which is called The Lounge, looks nice and works okay for me. This one is written in NodeJS and since there is no package in Fedora, I’ve decided to build it via yarn. It needs only one native dependency I think - sqlite3 so do not expect any problems on that front:

# dnf install nodejs yarn sqlite-devel
# dnf groupinstall "Development Tools"
# mkdir ~/.thelounge
# cd ~/.thelounge
# yarn add thelounge

If you prefer installing it into /usr/local then run yarn global add thelounge.

Create a user service, I will be running it as a regular user:

# cat /home/lzap/.config/systemd/user/thelounge.service
[Unit]
Description=The Lounge IRC client
After=network-online.target
[Service]
ExecStart=/home/lzap/.thelounge/node_modules/.bin/thelounge start
[Install]
WantedBy=multi-user.target

Start it to create a default configuration file:

# systemctl --user daemon-reload
# systemctl --user enable --now thelounge

Optionally, stop the service for now and review the configuration. There are couple of things I recommend to tune. By default the service listens on HTTP (9090), no HTTPS is configured, it stores all logs in both sqlite3 and text files and it is configured as “private” instance, meaning you need to login with a username and password:

# vim ~/.thelounge/config.js

Create new user:

# ~/.thelounge/node_modules/.bin/thelounge add lzap

Visit http://localhost:9090 or https://localhost:9090 if you’ve configured SSL. There you can create one or more IRC connections, join all channels, everything will be written into a separate ~/.thelounge/users/user.js configuration file which is nice. If you disabled sqlite3 logging, everything is stored in text files which I appreciate a lot.

If you want a simple letsencrypt tutorial for Fedora, read my prevous blog post:

# grep "https: {" -A5  ~/.thelounge/config.js
  https: {
    enable: true,
    key: "/etc/pki/tls/private/home.zapletalovi.com.key",
    certificate: "/var/lib/acme/certs/home.zapletalovi.com.crt",
    ca: "",
  },

No intermediate CAs are needed for letsencrypt so you can leave the field blank. Have fun.

Letsencrypt a Fedora server

I was looking for a simple letsencrypt tutorial for my home server running Fedora but it looks like the official (and quite capable) certbot is not availble in Fedora repos. So I have decided to go a more simple route of using acme-tiny shell script which is present and does the same, at least if you are running Apache httpd.

First off, install Apache httpd, SSL support and acme script itself:

# dnf install httpd mod_ssl acme-tiny

Let’s assume that the Apache server is already serving some files and is available on the desired domain via HTTP (not HTTPS yet):

# systemctl enable --now httpd
# curl -s http://home.zapletalovi.com | grep -o "Test Page"
Test Page

We are almost there, trust me. Generate a new certificate request. OpenSSL tool will ask several questions like name, organization and this stuff. Make sure that the Common Name (CN) is correct.

# cd /etc/pki/tls
# ln -s /var/lib/acme/csr .
# openssl req -new -nodes -keyout private/home.zapletalovi.com.key -out csr/home.zapletalovi.com.csr
# chmod 0400 private/home.zapletalovi.com.key

The next step is the actual communication with the authority, putting the challenge hash into /var/www/challenges directory which is exported by Apache httpd and downloading the signed request:

# systemctl start acme-tiny

See system journal for any errors. If you encounter one, just start the script manually but make sure to use acme user account not root:

# su acme -s /bin/bash
# /usr/libexec/acme-tiny/sign

And that’s really all! You should have your certificate signed by letsencrypt now. Configure the desired software to use the new certificate and the key from the following paths:

# find /var/lib/acme /etc/pki/tls/private
/var/lib/acme
/var/lib/acme/certs
/var/lib/acme/certs/home.zapletalovi.com.crt
/var/lib/acme/csr
/var/lib/acme/csr/home.zapletalovi.com.csr
/var/lib/acme/private
/var/lib/acme/private/account.key
/etc/pki/tls/private/home.zapletalovi.com.key

For example I want to actually configure the Apache httpd itself:

# grep zapletalovi /etc/httpd/conf.d/ssl.conf
SSLCertificateFile /var/lib/acme/certs/home.zapletalovi.com.crt
SSLCertificateKeyFile /etc/pki/tls/private/home.zapletalovi.com.key

If you are like me and running under SELinux enforcing, make sure that the newly generated certificates have the proper label:

# semanage fcontext -a -f a -t cert_t '/var/lib/acme/certs(/.*)?'
# restorecon -rv /var/lib/acme/certs

The final and the most important step - enable systemd timer which will automatically extend the certificate for you:

# systemctl enable --now acme-tiny.timer

That was easy.

Enable serial console for libvirt

QEMU/KVM libvirt virtual machine can be acessed via serial console. When a new VM is created, serial console device is created. However to fully utilize this, several steps are needed on the guest machine.

The first option is to start getty service on serial console to get a login prompt when system finishes booting. This is as easy as:

# systemctl enable --now serial-getty@ttyS0.service

To access serial console via libvirt command line do:

# virsh console virtual_machine_name

This approach is simple enough, but when something goes wrong and VM does not boot, it is not possible to access the VM during early boot or even bootloader. In that case, perform the additional configuration:

# grep console /etc/default/grub
GRUB_TERMINAL_INPUT="console serial"
GRUB_TERMINAL_OUTPUT="console serial"
GRUB_CMDLINE_LINUX="... console=ttyS0"

Then write new grub configuration, for EFI systems do the following:

# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

For BIOS systems do:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot and connect early to access grub or to see early boot messages. These commands will work on Fedora, CentOS, Red Hat Enterprise Linux and clones.

Thunderbolt bridge connection in Fedora 33

My home network is extremely slow, because I have CAT5e cables everywhere. I was wondering if I can use Thunderbolt ports which I have both on the new Mac M1 and Intel NUC with Fedora. So without my breath, since some Thunderbolt docks are known to brick the new Macs, I connected the two guys. And it worked automatically!

Fedora’s (33) kernel automatically recognized thunderbolt0 device and NetworkManager created a new connection named “Wired connection 1”. There must be some autonegotiation in the spec, because the two devices created 169.254/16 network and picked some IP addresses. I was not expecting that, I mean maybe if this was Linux to Linux but with MacOS involved I thought this is not gonna work. Let’s see how fast is my 100Mbps connection:

mac$ nc -v -l 2222 > /dev/null

linux$ dd if=/dev/zero bs=1024K count=512 | nc -v 192.168.1.55 2222
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.1.5:2222.
512+0 záznamů přečteno
512+0 záznamů zapsáno
536870912 bajtů (537 MB, 512 MiB) zkopírováno, 45,8012 s, 11,7 MB/s
Ncat: 536870912 bytes sent, 0 bytes received in 45.86 seconds.

That’s expected on a 100Mbps ethernet. On a gigabit network, which I considered to upgrade to, we should see something like 117 MB/s for my ideal case (just a switch). But let’s see how Thunderbolt works for me:

linux$ dd if=/dev/zero bs=1024K count=512 | nc -v 169.254.145.73 2222
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to 169.254.145.73:2222.
512+0 záznamů přečteno
512+0 záznamů zapsáno
536870912 bajtů (537 MB, 512 MiB) zkopírováno, 0,788541 s, 681 MB/s
Ncat: 536870912 bytes sent, 0 bytes received in 0.79 seconds.

Holy Moly! It’s not 1.1 Gbps but almost 900 MB/s that’s insane. This is a USB-C cable which is the best thing I currently have (this came with my LG screen). I am dropping a proper Thunderbolt3 into a basket to see how faster this can be. I mean in theory, I don’t have that fast SSD in my Intel NUC server.

Allright, so that’s looks like should be my preferred connection between my desktop and Linux. Let’s rename the connection first:

nmcli con modify "Drátové připojení 1" connection.id thunderbolt0

Oh gosh, I need to switch back to English from Czech language. Next up, set static IP address.

nmcli con modify thunderbolt0 ipv4.method static ipv4.address 192.168.13.4/24
nmcli con down thunderbolt0
nmcli con up thunderbolt0

And after quick update in a MacOS network dialog and /etc/hosts change, the connection between my new desktop and my working Linux machine is 10Gbps.

What is waking my HDD up in Linux

When my disks wake up during the day, I am angry. I want silence, so I started investigating which process makes them to do that. I suspect that something is browsing Samba share, but to confirm I created this simple SystemTap script:

# cat syscall_open.stp 
#!/usr/bin/env stap
#
# System-wide strace-like tool for catching file open syscalls.
#
# Usage: stap syscall_open filename_regexp
#
probe syscall.open* {
  if (filename =~ @1) {
    printf("%s(%d) opened %s\n", execname(), pid(), filename)
  }
}

It’s as easy as starting this up and waiting until the process is found. It accepts regular expression, not a glob:

# dnf install systemtap systemtap-runtime
# stap syscall_open.stp '/mnt/int/data.*'

This will work on all SystemTap operating systems, I tested this on Fedora and any EL distribution should work too.

Use btrfs compression in Fedora 33

Btrfs have been available in Fedora for quite some time and starting from Fedora 33, new installations of Workstation edition use it by default. Btrfs is pretty capable file system with lots of options, let’s take a look on one aspect: transparent per-file compression.

There’s little bit of misunderstanding how this works and some people recommend to mount with compress option. This is actually not necessary and I would actually strongly suggest NOT to use this option. See, this option makes btrfs to attempt to compress all files that are being written. If the beginning of a file cannot be effectively compressed, it’s marked as “not for compression” and this is never attempted again. This can be even forced via a different option. This looks nice on paper.

The problem is, not all files are good candidates for compression. Compression takes time and it can dramatically worsen performance, things like database files or virtual machine images should never be compressed. Performance of libvirt/KVM goes terribly down by order of magnitude if an inefficient backing store is used (qcow2).

I suggest to keep the default mount options Anaconda installer deploys, note there is none related to compression. Instead, use per-file (per-directory) feature of btrfs to mark files and directories to be compressed. A great candidate is /usr which contains most of the system including binaries or documentation.

One mount option is actually useful which Anaconda does not set by default and that is noatime. Writing access times for copy on write file system can be very inefficient. Note this option implies nodiratime so it’s not necessary to set both.

To enable compression for /usr simply mark this directory to be compressed. There are several compression algorithms available: zlib (slowest, best ratio), zstd (decent ratio, good performance) and lzo (best performance, worse ratio). I suggest to stick with zstd which lies in the middle. There are also compression level options, unfortunately the utility does not allow setting those at the time of writing. Luckily, the default level (3) is reasolable.

# btrfs property set /usr compression zstd

Now, btrfs does not immediately start compressing contents of the directory. Instead, everytime a file is written data in those blocks is compressed. To explicitly compress all files recursively, do this:

# btrfs filesystem defragment -r -v -czstd /usr

Let’s find out how much space have we saved:

# compsize /usr
Processed 55341 files, 38902 regular extents (40421 refs), 26932 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       50%      1.1G         2.2G         2.3G       
none       100%      337M         337M         338M       
zstd        42%      844M         1.9G         1.9G  

Exactly half of space is saved on a standard installation of Fedora 33 Server when /usr is compressed using zstd algorithm with the default level. Note some files are not compressed, these are too small files when it does not make any sense (a block would be used anyway). Not bad.

To disable compression perform the following command:

# btrfs property set /usr compression ""

Unfortunately, at the time of writing it is not possible to force decompression of a directory (or files), there is no defragment command to do this. If you really need to do this, create a script which reads and writes all files but be careful.

Keep in mind the btrfs property command will force all files to be compressed, even if they do not compress well. This will work pretty well for /usr just make sure there is no 3rd party software installed there writing files. There is also a way to mark files for compression and if they don’t compress well btrfs could give up on it. You can do that by setting chattr +c on files or directories. Unfortunately, you can’t set compression algorithm that way - btrfs will default to slower zlib.

Remember: Do not compress everything, specifically directory /var should definitely not be compressed. If you happened to accidentally mark files within /var to be compressed, you can fix this with:

# find /var -exec btrfs property set {} compression "" \;

Again, this will only mark them not to be compressed, it’s currently not possible to explicitly decompress them. Use the compsize utility to find out how much of data is still compressed.

That’s all for today, I will probably sharing some more btrfs posts.