In the first bash tutorial we went over brace expansions and a few history tricks. The goal of this post is to expand on that and teach more bash features.

Have you ever wondered when you typed a previous command in your history? This is easy to do with the $HISTTIMEFORMAT variable and bash >= 3.0. $HISTTIMEFORMAT accepts standard strftime(3) format strings.

jeff@beastmaster:~$ export HISTTIMEFORMAT="%m/%d/%y - %H:%M:%S "
jeff@beastmaster:~$ history | tail -n 75
411  06/11/08 - 15:36:09 sudo apt-get install network-manager-dev libmozjs-dev libmozjs0d
412  06/11/08 - 15:36:22 make clean && ./configure
413  06/11/08 - 15:36:30 make
414  06/11/08 - 15:36:54 apt-cache search dbus
415  06/11/08 - 15:36:59 apt-cache search dbus | grep dev

NOTE: The first time you set $HISTTIMEFORMAT, all previous commands inherit the current time. Everything afterwards is saved with a proper timestamp.

In the previous tutorial we covered usage of shell history modifiers like !vi or !?passwd to run previous commands. The only problem with those is that they execute the command directly without letting you edit them. Enabling the histverify shell option lets you edit the command before running it.

jeff@beastmaster:~$ shopt -s histverify
jeff@beastmaster:~$ !vi
jeff@beastmaster:~$ vi src/libproxy/src/Makefile

$CDPATH is a variable similar to $PATH, but it defines a search path for the cd command.

Given a directory structure like below, lets say that /home/jeff/svn/systems is one of your most frequently accessed directories.

/home/jeff/svn

/home/jeff/svn/systems

jeff@beastmaster:~$ export CDPATH=/home/jeff/svn
jeff@beastmaster:~$ pwd
/home/jeff
jeff@beastmaster:~$ cd systems
/home/jeff/svn/systems
jeff@beastmaster:~/svn/systems$ pwd
/home/jeff/svn/systems

Cool huh? The last set of tricks for today are bash's powerful parameter substitutions. Here is a non-exhaustive list along with a few examples.

jeff@beastmaster:~$ jeff@beastmaster:~$ parameter=digitalprognosis.com
jeff@beastmaster:~$ echo $parameter
digitalprognosis.com
jeff@beastmaster:~$ unset parameter
jeff@beastmaster:~$ echo $parameter # Emptyjeff@beastmaster:~$ echo ${parameter:-linux.com} # Display, but don't set the variable
linux.com
jeff@beastmaster:~$ echo $parameter # Empty

jeff@beastmaster:~$ echo ${parameter:=ubuntu.com} # Sets parameter to ubuntu.comubuntu.com
jeff@beastmaster:~$ echo $parameter
ubuntu.com
jeff@beastmaster:~$ echo ${parameter:+redhat.com} # Displays because parameter is set
redhat.com
jeff@beastmaster:~$ echo $parameter
ubuntu.com
jeff@beastmaster:~$ echo ${parameter/ubuntu/canonical} # Very simple sed feature-set
canonical.com
jeff@beastmaster:~$ echo $parameter
ubuntu.com

Overview

lsof or "List Open Files" is a favorite in my free software toolbox. It is so versatile there are few things you can't do without it.

Here are some examples of real world lsof usage and a few things things you might not know it is capable of doing.

What process is holding onto /var/log/messages? lsof is indispensable when troubleshooting what process is using a file or mount-point. Phil Dibowitz pointed out when troubleshooting hung mounts, there is a difference between lsof /mount/point and lsof /mount/point/.

root@terminated:~# lsof /var/log/messages
COMMAND   PID   USER   FD   TYPE DEVICE   SIZE  NODE NAME
syslogd 18623 syslog   15w   REG    8,6 870043 65289 /var/log/messages

Each process has a "fd" directory with symlinks to each open file-handle. Take a look under /proc/${PID}/fd to verify sysklogd has /var/log/messages open.

root@terminated:~# ls -l /proc/18623/fd | grep messages
l-wx------ 1 root root 64 2008-02-13 23:11 15 -> /var/log/messages

The top 10 processes with the most files open. Look for unusual processes with lots of open files. Thanks to Nathaniel Mccallum for an improved 1 liner. Script available here.

[root@terminated:~# lsof +c 15 | awk '{printf("%15s  (%s)\n", $1, $2)}' | sort | uniq -c | sort -rn | head
121          master  (3074)
48           snmpd  (16136)
33            sshd  (13715)
32           showq  (1906)
31            qmgr  (3084)
31          pickup  (2974)
28       syslog-ng  (2741)
26            ntpd  (5081)
25            sshd  (2977)
17           crond  (3085)

All tcp connections on port 22. By default, lsof only shows processes owned by the current user or all processes if ran as root. It also supports port aliases the same as in /etc/services. See services(5) for more information.

jeff@terminated:~$ lsof -i TCP:22
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
ssh     14559 jeff    3u  IPv4  44537       TCP terminated:44315->webhost:ssh (ESTABLISHED)
ssh     16637 jeff    3u  IPv4  53098       TCP terminated:37509->sentry.net:ssh (ESTABLISHED)

jeff@terminated:~$ sudo lsof -i TCP:ssh
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
sshd     5077 root    3u  IPv6  17429       TCP *:ssh (LISTEN)
ssh     14559 jeff    3u  IPv4  44537       TCP terminated:44315->webhost:ssh (ESTABLISHED)
ssh     16637 jeff    3u  IPv4  53098       TCP terminated:37509->sentry.net:ssh (ESTABLISHED)

Show processes with open files containing a link count less than 1. This is handy for seeing real system problems. An example would be poorly written applications that don't realize when the file being written to is yanked out from underneath them and continue on merrily writing to nothing. Modern versions of sysklogd realize this and will close the file handle until the file returns.

root@terminated:~# lsof +L1
COMMAND   PID  USER   FD   TYPE DEVICE SIZE NLINK  NODE NAME
mysqld  17016 mysql    5u   REG    8,8    0     0 48290 /tmp/ibYlPjD3 (deleted)
mysqld  17016 mysql    6u   REG    8,8    0     0 48291 /tmp/ib8Fo2za (deleted)
mysqld  17016 mysql    7u   REG    8,8    0     0 48292 /tmp/ibXG3Kwh (deleted)
mysqld  17016 mysql    8u   REG    8,8    0     0 48293 /tmp/ibIbs4wo (deleted)
mysqld  17016 mysql   12u   REG    8,8    0     0 48294 /tmp/ibfQqINv (deleted)

What files does pid 1 (init) have open?

root@terminated:~# lsof -p 1
COMMAND PID USER   FD   TYPE DEVICE    SIZE    NODE NAME
init      1 root  cwd    DIR    8,3    4096       2 /
init      1 root  rtd    DIR    8,3    4096       2 /
init      1 root  txt    REG    8,3   31216 1114187 /sbin/init
init      1 root  mem    REG    8,3   52400 1392748 /lib/libsepol.so.1
init      1 root  mem    REG    8,3  110984 1392653 /lib/ld-2.3.4.so
init      1 root  mem    REG    8,3 1526108 1394887 /lib/tls/libc-2.3.4.so
init      1 root  mem    REG    8,3   55000 1392711 /lib/libselinux.so.1
init      1 root   10u  FIFO   0,13            1108 /dev/initctl

For the curious, lsof gets this information from the maps and smaps files under /proc/$PID. Linux kernel 2.6.14 and newer  have smaps. See proc(5) for more information.

root@terminated:~# awk '/\//{print $NF}' /proc/1/maps | sort -u
/lib/ld-2.3.4.so
/lib/libselinux.so.1
/lib/libsepol.so.1
/lib/tls/libc-2.3.4.so
/sbin/init

As you can see, lsof(8) is a great tool in diagnosing system problems or just seeing whats going on. In my usage, lsof -i has mostly replaced netstat.

Eric Butler did a very good writeup of this years Southern California Linux Expo. Instead of writing a rehash, you can read his post.

One particular thing to note is his mention of Sun's true commitment towards Linux.

Unbeknownst to many, rpm supports random printf() style query strings via the --qf option. List all available tags with "rpm --querytags" and display them with "rpm -q --qf '%{QUERYVARIABLE}'. On some versions of rpm, you need to place multiple variables inside brackets ( [] ) to display them all properly.

Use the ARCH and NAME tags along with sort to display a list of all packages and the architecture they were built for:

$ rpm -qa --qf '[%{ARCH} %{NAME}\n]' | sort

i386 zip
i386 zlib
i686 glibc
i686 kernel
i686 kernel-smp
i686 openssl
noarch basesystem
noarch htmlview
noarch hwdata
noarch man-pages
...

Say you want to look at the licenses of every package installed on your system except gpl and bsd code. Rpm makes this very easy

$ rpm -qa --qf '[%{LICENSE} %{NAME}\n]' | egrep -v '(GPL|BSD)'
public domain setup
public domain basesystem
distributable ncurses
freeware vim-minimal
Freely Distributable cyrus-sasl
XFree86 redhat-menus
distributable symlinks
distributable zip
Artistic perl-RPM2
distributable perl-XML-Encoding
Public Domain expect
Artistic perl-XML-Simple
...

A fairly complex example script showing the power to rpm querystrings and how they can be useful in the real world is available here under the terms of the GNU GPLv2. It querys the rpm database using the verify function (-V). It shows you files that have changed permissions or ownership and optionally allows you to fix them.

Brace expansions save you lots of typing:

$ ls
file1

$ cp file{1,2}
$ ls
file1  file2

$ mv file{1,1.old}; ls
file1.old file2

You can even use brace expansions in shell scripts:

#!/bin/bash
scp /path/to/{file1,dir2,file2} user@otherserver:/dump/
if [ $? -eq 0 ]; then
rm /path/to/{file1,dir2,file2}
fi

NOTE: In posix land, an exclamation mark is often referred to as a "bang".

!! - aka "bang bang". Runs the previous command typed into the shell
!vi - Runs the most previous command starting with "vi"
!?passwd - Runs the most previous command that contained "passwd"
!vi:p - Prints the most previous command containing "vi" without running anything.
!vi:s/passwd/shadow/ - Runs the most previous command containing the word "vi" and replaces "passwd" with "shadow"

On to Part II --->

This is a place for me to contribute knowledge and solutions to problems found while working as a Linux Systems Administrator. If you find a {better,cleaner,faster} way to do anything posted on here please leave a comment so everyone can benefit.

This post won't contain much nor will this techblog until I finish setting up wordpress and creating a custom theme to match the rest of www.digitalprognosis.com.