8 CLI Tools that are Under-utilized, Sometimes Unknown, or Unappreciated and Yet Pack a Powerful Punch


8 CLI (That’s Command Line Interface For You Windows Types) Tools That Are Under-utilized, Sometimes Unknown, Or Unappreciated And Yet Pack A Powerful Punch

Editor’s Note: This article received 7 5-star ratings while posted on the original Cool Solutions site.

I watch a lot of the Linux boards and various distribution sites these days and notice the trend of wanting everything to be bigger, better and flashier. With great projects like Beryl (I love distracting Windows administrators with the cube…it’s like watching a bug fly into one of those electric blue lights), and developers in the open source community constantly improving and designing ever more functional GUI interfaces (think tools such as Yast2 or Synaptic) it is easy to start letting your CLI skills get rusty and forget about your little but mighty friends on the console. Unfortunately with the sheer amount of sensitivity in the world these days I need to be certain I preface that I use GUIs and am by no means belittling, demeaning, pointing fingers and laughing nor looking to taunt you a second time-ah, however, I do believe that maintaining CLI skills and commands are imperative to good Linux administration. Nothing like being woken at two in the morning by an urgent alarm that your server sporting Tux has gone down and will not boot back to a display manger (yes yes, I know this doesn’t happen…but let’s think about this for a moment strictly as pilots in a simulator…training). Below are a few command line tools that I have found to be very helpful and informative when it comes to administrating Linux. They may not always be readily visible or well known, but they pack a very powerful punch. Now to the nitty and gritty…

WATCH — watch is a neat and versatile tool. Ever sat there frustrated running a particular command over and over just to see what is changing? watch automates the command you wish to run and repeats it every x seconds. The format is watch -n <command>. The default time is 2 seconds if you leave out the -n flag. Lets say you want to watch connections on port 80. By issuing “watch lsof -i TCP:80” watch will run the command “lsof -i TCP:80” every 2 seconds and display the results in a terminal. You will see that I started out with just consulting the oracle (google), but then I figured I’d go see how my boys were doing up at Ft. Sill and cruise the news pages over on yahoo. Watch happily provided the results.

#watch lsof -i TCP:80
Every 2.0s: lsof -i TCP:80                                            Fri Jul 20 07:58:31 2007


iceweasel 10030 user  36u  IPv4  29054       TCP>hs-in-f99.google.com:www (ESTABLISHED)

iceweasel 10030 user   37u  IPv4  28547       TCP>yo-in-f103.google.com:www (ESTABLISHED)
Every 2.0s: lsof -i TCP:80                                       Fri Jul 20 07:58:38 2007   <---- notice the time


iceweasel 10030 user   34u  IPv4  31794       TCP>unknown.xeex.net:www (ESTABLISHED)

iceweasel 10030 user   42u  IPv4  32130       TCP>jc-in-f99.google.com:www (ESTABLISHED)

iceweasel 10030 user   43u  IPv4  33178       TCP>l1.ycs.vip.mud.yahoo.com:www (ESTABLISHED)

iceweasel 10030 user   44u  IPv4  32178       TCP>yo-in-f147.google.com:www (ESTABLISHED)

iceweasel 10030 user   45u  IPv4  32587       TCP> (ESTABLISHED)

iceweasel 10030 user   46u  IPv4  32190       TCP>sill-www.army.mil:www (ESTABLISHED)

iceweasel 10030 user   47u  IPv4  33177       TCP>l1.ycs.vip.mud.yahoo.com:www (ESTABLISHED)

iceweasel 10030 user   48u  IPv4  32907       TCP> (ESTABLISHED)

iceweasel 10030 user   52u  IPv4  33181       TCP>l1.ycs.vip.mud.yahoo.com:www (CLOSE_WAIT)

Maybe you want to watch a particular user and what processes they are running: watch -n 1 ‘ps aux | grep <user>’ . If you toss in the -d flag, each time the console refreshes it will highlight the differences from the last execution. You can get creative with watch, just remember it is actually executing the command.

JUMPGATE — jumpgate became my friend awhile back when I administered some wireless access points that did not have a connection to the backhaul due to specialized protocols and security. In effect I needed a TCP forwarder to send data back and forth between boxes. By issuing jumpgate you can take a machine and have it handle the forwarding and receiving for your client. Format: jumpgate -b <localhost or IP address> -l <local port> -a <remote host or IP address> -r <remote port>

For example, in my situation I needed my local client to authenticate to a certificate server to be issued a certificate to allow it to join a secure network. The problem was my client could not reach the internal network where this certificate server resided. However, it could reach the gateway linux server and that server could reach the certificate box. I ran jumpgate on the gateway to forward my request.

#jumpgate -b -l 80 -a -r 8080

Then when I opened my browser and was prompted for the address of the certificate server by my local linux box I gave it the gateway’s address instead ( Once the packet hits the gateway, jumpgate is listening on that particular port and forwards it to the remote destination ( where the certificate server was actually listening for requests on port 8080. Jumpgate includes options for logging or being interactive such as: jumpgate -i -l 32000 -f jumpgaterequest.log. This tells jumpgate to bind and listen for connections on port 32000, then interactively ask the user where they want the file transferred to when a connection is made and log the session to file jumpgaterequest.log.

LSOF — lsof is a personal favorite of mine. In fact I submitted a tip on lsof before. It’s versatility is great, especially when you are troubleshooting an issue and need more information about process or connection details. This command elegantly stands for list open files. Linux treats most everything as a file. Sockets, devices, directories, etc, can all be viewed as files. When a process or application interacts with these files it has to “open” them if you will. Using this command you can delve into and see what your system is up to.

For instance to show all the open TCP files - Will return what service is running, who is running it, the process ID and the connections on all TCP ports:
# lsof -i TCP
Show open TCP files on port 80 -
# lsof -i TCP:80 
returns --> httpd2-wo 7010 wwwrun    3u  IPv6  14787       TCP *:http (LISTEN)

Show open LDAP connections on TCP -
# lsof -i TCP:636

Want to know what files are open by a particular command (substitute your command after the c, and yes you can abbreviate it matches the closest command)
# lsof -c mysq
mysqld    991 admin  cwd    DIR        8,3      240 148743 /home/admin/novell/idm/mysql/data
mysqld    991 admin  rtd    DIR          8,3      536      2 /
mysqld    991 admin  txt    REG         8,3      5464060 148691 /home/admin/novell/idm/mysql/bin/mysqld
mysqld    991 admin    0r   CHR         1,3       41715 /dev/null
mysqld    991 admin    1w   REG        8,3     1250 149954 /home/admin/novell/idm/mysql/mysql.log
mysqld    991 admin    2w   REG        8,3     1250 149954 /home/admin/novell/idm/mysql/mysql.log
mysqld    991 admin    3u  IPv4      86990             TCP *:63306 (LISTEN)...
Want to know what files are open by a particular device?

#lsof /dev/cdrom
bash    30904 admin  cwd
DIR    3,0 2048 63692 /media/cdrecorder/linux/user_application_provisioning

You can change TCP to UDP and narrow down your requests to very specific items you want to target (i.e. is there an established connection from xyz.somesite.com?).
# lsof -i TCP@ (lists LDAP connections to my server)

returns --> java     890 root   18u  IPv6 8365030
TCP myserver.somecompany.com:42936->myserver.somecompany.com:ldaps (ESTABLISHED)

ndsd    6520 root  262u  IPv4 8390927
TCP myserver.somecompany.com:ldaps->myserver.somecompany.com:43123 (ESTABLISHED)

ATOP — As an administrator you are used to typing ‘top’ to see real-time tasks and system information. atop takes it a step further and injects some steroids (in a safe and humane fashion) to allow you to flex some muscle and watch your processes and system information in much greater detail. I like the fact that I can watch read/writes from the disk as well as that it offers a quick snapshot of my network device and memory usage (VGROW and RGROW). Another neat feature is that it takes samples and logs these in /var/log/ for later review. This can be very useful for system analysis (i.e. you keep hearing complaints that the server is sluggish during certain times of the day…you can review these samples watching for high utilization or memory leaks). When you fire up atop you can press h and it will bring up the help screen. There are many options for you to tweak this command.


ATOP  ?  MYLAPTOP                    2007/07/25        08:25:29          10 Seconds elapsed
PRC   | sys   0.04s | user   0.26s   | #thr     150      | #zombie    0        | #exit      0         |
CPU   | sys      1% | user      3%   | irq       2%       | idle    195%         | wait      0%      |
cpu     | sys      0% | user      2%   | irq       2%       | idle     96%          | cpu000 w  0% |
cpu     | sys      0% | user      1%   | irq       0%       | idle     99%         | cpu001 w  0%  |
MEM | tot    2.0G | free  982.6M | cache 491.5M | buff  123.1M       | slab   60.6M     |
SWP   | tot    3.0G | free    3.0G   |                         | vmcom 869.0M | vmlim   4.0G    |
DSK   |         sda    | busy      0%  | read       0        | write      4           | avio    2 ms      |
NET   |   transport | tcpi       1      | tcpo       1        | udpi       0           | udpo       0        |
NET   |     network | ipi        5      | ipo        1         | ipfrw      0          | deliv      1         |
NET   | dev    eth0 | pcki       6    | pcko       1        | in    0 Kbps        | out   0 Kbps      |

 5463   0.02s   0.13s     0K     0K root       1  --   -  R   2% Xorg
 6152   0.00s   0.07s     0K     0K user      1  --   -  S   1% metacity
10953   0.01s   0.03s     0K     0K user      8  --   -  S   0% firefox-bin
11100   0.01s   0.01s     0K     0K root       1  --   -  R   0% atop
 6143   0.00s   0.01s     0K     0K user      1  --   -  S   0% gnome-panel
 6701   0.00s   0.01s     0K     0K cupsys     1  --   -  S   0% cupsd
 6148   0.00s   0.00s     0K     0K user      1  --   -  S   0% nautilus
 6270   0.00s   0.00s     0K     0K user      2  --   -  S   0% gnome-terminal
 7269   0.00s   0.00s     0K     0K user      1  --   -  S   0% notification-d
 6167   0.00s   0.00s     0K     0K user      4  --   -  S   0% gnome-cups-ico
 6244   0.00s   0.00s     0K     0K user      1  --   -  S   0% gnome-screensa

IFTOP — Curious about what network traffic is flowing around you? Pop this command into your console and get instant feedback. Of interest (although it is not shown in my example below) is that bars will appear as bandwidth is used up. Since I’m on a test network while I write this, there is not enough traffic generation for iftop to scale up. I like it because you can toggle DNS resolution, show source or destination or both, sort by column or source or destination and a sweet feature is to freeze the order. If you have something you want to specifically watch, you can type o (the letter not the number) and it locks onto just those source/destination connections. Again, typing h brings up the help and options screen. To be honest I usually leave iftop up and running during the day to keep an eye on what is going on with my network.

                      1.91Mb           3.81Mb           5.72Mb    7.63Mb  9.54Mb
+-----------------------------------------------          =>              0b      0b      0b
                         <=                            256b    179b    166b            => user1.machine.net          0b      0b      0b
                         <=                            0b      169b    133b            =>              0b      0b      0b
                         <=                            0b      0b      55b            => user1.machine.net          0b      0b      0b
                         <=                            0b      0b      55b            => user2.machine.net          0b      0b      0b
                         <=                            0b      0b      55b              => ALL-SYSTEMS.MCAST.NET      0b      0b      6b
                         <=                                                                                   0b      0b      0b

TX:             cumm:    654KB           peak:    0b          rates:     0b      0b      0b
RX:                     5.59MB                2.02Kb                   256b    348b    470b
TOTAL:                  6.23MB                2.02Kb                   256b    348b    470b

PSTREE — Run ‘ps aux’ to have it spew out its results and you are stuck combing through the list checking which process is related to the another process. It’s ingrained to type ‘ps aux’ as an administrator but pstree simplifies the display by taking the process status and building it out as tree. The results are clean and it allows you to rapidly check parent and child relationships (good parenting skills are always a bonus in life…remember to teach your children Linux early on). The key word there is rapidly. If you need or desire to see the PID inside the tree insert the -p flag.


         ?                         +-6*[{amarokapp}]
         +-gnome-terminal---bash---iftop---3*[{iftop}]                  <--- check it out, it shows iftop and atop running
         ?                                +-bash---atop
         ?                                +-bash---su---bash
         ?                                +-bash---su---bash---pstree
         ?                                +-gnome-pty-helpe
         ?                                +-{gnome-terminal}
<truncated for space>

NETCAT (NC) ? netcat or nc is aptly nicknamed the swiss army utility knife of networking. This is another personal favorite. Although this tool is still popular often times folks forget about it (except those who wear b/w/g hats). Official sounding terminology from the man page, is: it is “a simple unix utility which reads and writes data across network connections, using TCP or UDP protocol. It is designed to be a reliable “back-end” tool that can be used directly or easily driven by other programs and scripts”.

Using an earlier scenario of machines that were separated, and now adding the fact I needed to flash images, I could combine the use of jumpgate and dd with netcat and have my remote system flash itself (no cops or court dates appear with this type of flashing).

Quick imaging: On the box I wish to flash I issue:
# nc -l -p 23000 | dd of=/dev/hda        <--- tells netcat to listen on port 23000 and pipe whatever it receives to dd

On the client-side (box I am sending the image from) I issue:
# dd if=/dev/hda | nc 23000                    <--- takes the output from dd and pipes it over to a listening server

Don't have nmap loaded? No worries, netcat will scan ports for you:
# netcat -vv -z 8000-9200                      <--- using -vv tells netcat to be very verbose and -z tells it to scan the range (8000-9200) you can
                                                                                            substitute your own port range values
(UNKNOWN) [] 9104 (?) : Connection refused
(UNKNOWN) [] 9103 (bacula-sd) : Connection refused       <--- interesting it has a network backup daemon
(UNKNOWN) [] 9102 (bacula-fd) : Connection refused
(UNKNOWN) [] 9101 (bacula-dir) : Connection refused
(UNKNOWN) [] 9100 (?) open                           <--- Hey, I discovered I have a device that is running print service using HP JetDirect
(UNKNOWN) [] 9099 (?) : Connection refused
(UNKNOWN) [] 9098 (xinetd) : Connection refused

If you do not care about or need a secure transmission you could skip using scp and modify nc to transfer files, on the server side:
# nc -l -vv -p 9000 > myfilename.txt       <--- sets up netcat to listen on port 9000, output whatever it receives to said filename, and be verbose about it so
                                                                         I can watch (if I have a console open on the server)

And on the client side I issue:
# nc -vv 9000 < myfilename.txt     <--- feed my file using nc to the server at port 9000 and tell me about it while doing it

STRACE — strace is a wicked little debugger. This command is fancy to ToTo in the Wizard of Oz, it will peel back the curtain and show you what levers and wheels the great Oz is working when you execute a command. Ever wonder why or specifically where your compile and makefile was vomiting? Pondering why your application hangs and just appears to be caught in a time/space vortex? strace allows you to watch step by step what the kernel is performing when you request action. Now the warning for you is that it reveals A LOT of information that will require a little patience on your part when combing through the data. However, this little bit of patience will pay off when you are able to find where you need to modify your code or add libraries or whatever the problem is when your command errors out. Below is an example output (truncated/edited for space) from syncing the hardware clock to system clock:

#strace hwclock ?hctosys

execve("/sbin/hwclock", ["hwclock", "--hctosys"], [/* 32 vars */]) = 0

brk(0)                                  = 0x608000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b7148539000

uname({sys="Linux", node="L01395", ...}) = 0

access("/etc/ld.so.nohwcap", F_OK)      = 0

mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b714853a000

access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)

open("/etc/ld.so.cache", O_RDONLY)      = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=78580, ...}) = 0

mmap(NULL, 78580, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b714853c000

close(3)                                = 0

access("/etc/ld.so.nohwcap", F_OK)      = 0

open("/lib/libc.so.6", O_RDONLY)        = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\331"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=1367432, ...}) = 0

mmap(NULL, 3473592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b714873a000

mprotect(0x2b7148881000, 2097152, PROT_NONE) = 0

brk(0)                                  = 0x608000

brk(0x629000)                           = 0x629000

open("/usr/lib/locale/locale-archive", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/locale.alias", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=2586, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b714853c000

read(3, "# Locale name alias data base.\n#"..., 4096) = 2586

stat("/etc/adjtime", {st_mode=S_IFREG|0644, st_size=46, ...}) = 0

open("/etc/adjtime", O_RDONLY)          = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=46, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b714865f000

read(3, "0.293746 1185223896 0.000000\n118"..., 4096) = 46

close(3)                                = 0

munmap(0x2b714865f000, 4096)            = 0

open("/dev/rtc", O_RDONLY)              = 3

ioctl(3, RTC_RD_TIME, {tm_sec=26, tm_min=51, tm_hour=7, tm_mday=24, tm_mon=6, tm_year=107, ...}) = 0

ioctl(3, RTC_RD_TIME, {tm_sec=26, tm_min=51, tm_hour=7, tm_mday=24, tm_mon=6, tm_year=107, ...}) = 0

close(3)                                = 0

munmap(0x2b714865f000, 4096)            = 0

stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=861, ...}) = 0

settimeofday({1185281487, 0}, {300, 103079215111}) = 0

exit_group(0)                           = ?

Process 9729 detached

These commands and suggestions are strictly my two cents (and when adjusted for inflation usually worth less than that). However, central to having an understanding of Linux is having an understanding of how to interact and interpret data that the kernel will serve up. These CLI tools will run on any Linux variation, SLES, SLED, OES, OpenSuSE or other distribution. Not all of these tools are included by default though, so included below are links to those that probably are not already installed. I like GUIs…so no I’m not writing this tip sheet in vi…but I do believe learning CLI and using the Linux console is akin to learning to drive a standard. Once you understand the clutch and gas routine you can drive anything. CLI is the standard that never lets you down. Happy computing…and remember:

Use the Force Young Jedi-Penguin


(Visited 1 times, 1 visits today)

Leave a Reply

Your email address will not be published. Required fields are marked *

No comments yet