Git auto fetch script I run every day

I am “shutdowner”, meaning I always shutdown my laptop (now workstation) at the end of the day. I have a script to do that which sleeps 5 seconds (so I can change my mind - e.g. when I dig shell history incorrectly and quickly hit enter - it really happened yeah) and it is simple:

puts my monitors into standby mode
applies all OS updates
runs duplicity backup on my home folder
fetches git repos
filesystem sync call
fstrim root volume
poweroff

I learned a trick I want to write about today from colleague of mine Mirek Suchý, but I think he runs it from cron (not a “shutdowner” guy). The idea is simple:

find all directories containing .git/ and run on all of them:
git fetch –all
git gc

So every time I do git pull on a repo that I don’t use much (e.g. ruby language), I don’t need to wait seconds in order to pull all commits. Clever, now I’ve improved it a bit.

With my Ryzen 1700 8 core 16 threads CPU, I am able to leverage GNU parallel to do this in parallel. That will be faster. But how much? Let’s test against git repo I use the most: www.theforeman.org.

# git -c pack.threads=1 gc --aggressive
1m25.175s

# git -c pack.threads=16 gc --aggressive
0m16.321s

Initially I thought that running 16 GNU parallel worker processes of parallel will be fine, but git gc is really slow on one core (see above), so I usually end up with several very slow garbage tasks while all the others finished downloading. The sweet spot for git is around 4 threads where it always gives reasonable times even for bigger repos.

But I think little bit of CPU overcommit won’t kill, therefore I’ve decided to go with 8x4 which might sound crazy (32 threads in theory), but in practice garbage collect is executed only on few repositories I work regularly on.

Lot of words, I know. Here is the snippet:

find ~/work -name '.git' -type d | \
    parallel -j 6 'pushd "{}"; git fetch --all; git -c pack.threads=4 gc --aggressive --no-prune --auto; popd'

I think I could go further but this already gives me good experience and when my PC is doing this, I am already heading away from it. No biggie. Final notes for git flags I use:

aggressive - much slower collect giving better results
no-prune - I don’t want to loose any commits at any point in time
auto - git will decide when to actually run gc

Read on Comment

17 July 2017 | linux | fedora | git

Ryzen workstation build 2017

I’ve been waiting for so long, AMD RYZEN processors are here for some time and I was giving few months to vendors and community to settle down and release updates for their BIOSes, firmware and Linux drivers before I finally made my decision. My next machine for work is AMD RYZEN 1700, unbelievable value for the money! Eight cores, sixteen threads and only 65 watts with stock cooler that is silent and “cool” according to all reviews.

For motherboard, I was sure to get B350 chipset which matches my requirements and after reading my friend’s article there was no doubt about ASUS PRIME B350-PLUS that gives just enough of everything I need, including M.2 (4x lanes), 1G LAN and enough USB ports.

Now, I am going to use this as typical Linux programmer workstation with some decent virtualization workload. Since I am working on Red Hat Satellite 6 which is quite demanding (minimum requirements are 16 GB RAM for server, 8 GB RAM for Capsule), I need more than 24 GB. I wish I could max out mobo to 64 GB, but memory prices are ridiculous these days! It went up like 200%, so I just ordered 32GB of Kingston DDR4 2400MHz which seems to work just fine with this ASUS mobo. I do not want any surprises, therefore I will stay at stock 2400MHz and will not do any overclocking of either memory or CPU as I prefer stability and I really don’t know how to do this exactly to be honest.

I am going to experiment with dm-cache tho, so I picked Seagate Barracuda 4TB SATA drive along with entry-level NVMe SSD drive Intel 600p M.2 256GB where I am going to put root of my system and cache for home folder sitting on the Barracuda drive. I am really interested in how dm-cache is usable and what performance it will give me for large git checkouts.

This is definitely not a gaming rig, but I need a graphics card for sure and it looks like my old spare MSI GTX 750 Ti will service well. I know, this card totally does not match the powerful CPU, but it is very quiet and with its TDP of 60 watts it does not even require extra power connector. It has a DVI output, that’s all I need. Will it scroll my browser smoothly? I bet it will.

Next on the list we have Seasonic M12 EVO Bronze 520W which is total overkill for this build, but I have one spare which returned from RMA last year (I had to replace my wife’s PSU already), so I will use it. It’s a premium and modular PSU which will keep my case clean.

Speaking about case, I have an old case with DVD burner, but when I pulled it out this morning, I realized it’s way too old, the drive is an IDE one and it looks - well dated. I quickly made a purchase of Fractal Design Define R5 which is supposed to be most successful case these days according to all reviews. And it’s silent and extensible with many drives for the future, that’s what I aim for. No more hard drive space issues for huge Red Hat yum repository testing anymore for me!

Hardware installation was smooth, the stock cooler is heavy, looks nice but most importantly AM4 socket is back to good-old screws. I never liked these clips or click things and cooler installation was always clunky. Not with AM4 anymore, four screws with a philips screwer. Although my very first chip was MOS from my Commodore 64, the first x86 CPU was indeed AMD (i386 running at 40Mhz). It’s good to be back home.

The new case is brilliant, I never owned a “gaming” case, but this piece was given dozens of little details which are amazing. Everything is on the right place but you can move lots of parts inside the noise-cancellation cage, plenty of room for everything including HDD trays, extra SSD slots on the back, cable management, dust filters, good case fans included, manual fan control, most of the work was screwless, and there is more.

The rest of the build was pretty standard, except NVMe Intel 600p M.2 chip which corrupted my brand new Fedora installation, turned out to be bug in the Intel’s firmware which surprised me and I swapped it with Samsung 960 EVO M.2 chip of the same size. It was a pain to find out, because it also corrupted Grub2 loader on EFI volume and the system was booting into black screen. Had to figure it out from Fedora Live system and fsck.

I have been using laptops for work for more than 10 years now, I almost forgot how it feels to have much more powerful CPU, much more memory and no limitations of adding more and more drives to the system. During the next days I will be installing Windows 10 onto this. Lame, yes I know but I just want smooth BIOS and (SSD) firmware upgrade experience, it is absolutely necessary to update BIOS of AMD AM4 mobos these days before moving to real work. Then I will be installing Fedora and setting things up and finally writing an article about dm-cache setup and how well it hopefully performs.

Idle power consumption of my build is 50W which is nice for such a powerful system, CPU fan rotating at 1600 RPMs in stock BIOS settings. Everything is very silent, under load it is the same noise as my Lenovo ThinkPad T430s but in more comfortable lower frequencies, under idle or web browsing the new build wins because fan of the laptop never turns off.

Tomorrow is my first working day with the system, I am indeed hopping into a video conference with BlueJeans which makes my laptop crazy (load 1.50 with full-throttle fan). I expect much more smooth experience this time.

Some poor photos from my ASUS phone

$ lspci | grep Intel | wc -l
0

Read on Comment

07 June 2017 | linux | fedora | hardware

PulseAudio headset switcher

From time to time I encounter an embarrasing moment on meeting when new headset does not work. I purchased new one which only made some noise instead. I will return this back, but in the meantime I found a Logitech USB headset in my closet and it works pretty well, so I think I am gonna stick with it.

I created few aliases that will help me to switch PulseAudio default input and output to headset and back and forth. My aliases also switch all existing streams, which is great if I open up BlueJeans or WebEx before making the change.

This switches input and output to headset, volume is about 90 % for output, 100 % for input, unmutes input and says “Speak”:

auheadset

This puts everything back to laptop default soundcard setting full output volume and zero input volume:

aunormal

This mutes the card and says “Mute”:

aumute

And this unmutes it and says “Speak”:

auspeak

I have some key bindings in my i3 window manager to do mute/unmute using keyboard shortcut. I have to say this is finally a setup that works. Here is the script you need to put into your .bashrc:

export MY_NORMAL_OUTPUT=alsa_output.pci-0000_00_1b.0.analog-stereo
export MY_NORMAL_INTPUT=alsa_input.pci-0000_00_1b.0.analog-stereo
export MY_HEADSET_OUTPUT=alsa_output.usb-Logitech_Logitech_USB_Headset-00.analog-stereo
export MY_HEADSET_INPUT=alsa_input.usb-Logitech_Logitech_USB_Headset-00.analog-mono
alias auvolup="pactl set-sink-volume $MY_HEADSET_OUTPUT +10% && pactl play-sample blip"
alias auvoldown="pactl set-sink-volume $MY_HEADSET_OUTPUT -10% && pactl play-sample blip"
alias auvolfull="pactl set-sink-volume $MY_HEADSET_OUTPUT 65536 && pactl play-sample blip"
alias aumute="pactl set-source-mute $MY_HEADSET_INPUT yes && pactl play-sample mute"
alias auspeak="pactl set-source-mute $MY_HEADSET_INPUT no && pactl play-sample speak"
function pacmd-set-output-input {
  pacmd set-default-sink $1
  echo "Default output set $1 (volume $3)"
  pacmd set-default-source $2
  echo "Default intput set $2 (volume $4)"
  pacmd set-sink-volume $1 $3
  pacmd set-source-volume $2 $4
  pacmd list-sink-inputs | grep index | while read line; do
    INDEX=$(echo $line | cut -f2 -d' ')
    echo "Moving $INDEX to $1"
    pacmd move-sink-input $INDEX $1
  done
  pactl play-sample blip
  echo "Done, all set to $5"
  [[ "$5" == "HEADSET" ]] && auspeak
}
alias aunormal="pacmd-set-output-input $MY_NORMAL_OUTPUT $MY_NORMAL_INTPUT 65536 0 NORMAL"
alias auheadset="pacmd-set-output-input $MY_HEADSET_OUTPUT $MY_HEADSET_OUTPUT 40000 65536 HEADSET"

You need to edit MY_* variables, use pactl list to find your device names.

You also want to put this in your startup script of your WM or somewhere else where it is loaded just once when PulseAudio starts. In the worst case you can put this into your .bashrc as well, but it will slow down your shell start.

pactl upload-sample ~/.i3/blip.wav blip
pactl upload-sample ~/.i3/mute.wav mute
pactl upload-sample ~/.i3/speak.wav speak

I found USB headset to be much more cleaner than integrated laptop sound card.

Read on Comment

13 April 2017 | linux | fedora

Improved the blog theme

I’ve improved the navbar, it should now work on mobile devices. Thanks for reporting this.

I tried to update to Bootstrap 3 but it did not work well. I guess I will stay on the old version for now. Christmas time!

Read on Comment

02 December 2016 | blog

XTerm Zen mode

I do lot of talks these days with some amount of shell live demos. Although it is easy to configure xterm for beamer (black on white, big font, fullscreen), I created an alias for that:

xterm -cm -fa Monospace -fs 32 -bg white -fg black -cr red -bc \
    -bcf 200 -bcn 200 -xrm "allowColorOps false" -fullscreen \
    -e "PS1='$ ' LC_ALL=C /bin/bash --norc --noprofile"

And this is how it looks like:

xterm-zen