We were investigating issue today - something - some process - have changed permission of a very important file in one of our Katello testing installations. We only know it happened over the night. The task is to catch the instigator. We can use SystemTap for that. Prepare our environment:

yum -y install systemtap systemtap-runtime kernel-debuginfo-`uname -r` kernel-debuginfo-common-`uname -i`-`uname -r` kernel-devel-`uname -r`

Our target file will be /test:

touch /test

Let's get its inode:

ls -i /test
274

Using "mount" found the device the file resides on and find it's major and minor numbers: In my case it's:

ll /dev/md-0
brw-rw----. 1 root disk 253, 0 Apr 17 10:23 /dev/dm-0

Now create the following file:

cat filechange.stp
#!/usr/bin/env stap
global ATTR_MODE = 1
probe kernel.function("setattr_copy")!,
      kernel.function("generic_setattr")!,
      kernel.function("inode_setattr") {
  dev_nr = $inode->i_sb->s_dev
  inode_nr = $inode->i_ino

  if (dev_nr == MKDEV($1,$2) # major/minor device
      && inode_nr == $3
      && $attr->ia_valid & ATTR_MODE)
    printf ("%d %s(%d) %s 0x%x/%u %o %d\n",
      gettimeofday_us(), execname(), pid(), probefunc(),
      dev_nr, inode_nr, $attr->ia_mode, uid())
}

And run it in a screen session or something:

stap -v filechange.stp 253 0 274

After some time...

1334676922011223 chmod(6157) generic_setattr 0xfd00000/274 100600 0

...BUSTED! The fist one is timestamp, the second one is process name(pid).