Saturday 9 February 2013

Using Linux the CLI way - Cheat Sheet

Using Linux the CLI way - Cheat Sheet


Linux, like any other Unix system is case sensitive so type them exactly as they are written in this article.

The File Separator

An important feature that differentiates Unix systems from Windows systems is the way one refers to the File Separator. On UNIX based systems it’s as follows.
/usr/local/apache/httpd.conf -----> the forward file separator. While on Windows systems. C:\webserver\apache\httpd.conf -----> the backward file separator.

any_command --help | more
Display a brief help file on a command (works with most commands). –help works similar to DOS /h switch. The more pipe is needed if the output is longer than one
screen. | This isn’t an L, it is what we call the redirection operator. If you are trying to find it on your keyboard, you’ll probably see it on the top row with the numerical keys. The main purpose of the redirector is to redirect output from one process to another. It works at the shell level and is very handy to have around. More on this
later.

ls
Lists the content of the current directory. Under Linux, the command dir is an alias to ls. Many users have ls to be an alias to ls --color.

ls -al | more
List the content of the current directory, all files (also those starting with a dot), and in a long form. Pipe the output through the more command, so that the display
pauses after each screen fill.

cd /path_to_directory
Change directory. cd - will take you to your previous directory and is a convenient way to toggle between two directories.

cp /path_to_copy_form /path_to_copy_to
Copy files from source to destination.

mv source destination
Move or rename files. The same command is used for moving and renaming files and directories.

ln source destination
Create a hard link called destination to the file called source. The link appears as a copy of the original files, but in reality only one copy of the file is kept, just two (or more) directory entries point to it. Any changes in the file are automatically visible throughout. When one directory entry is removed, the others stay intact. The
limitation of the hard links is that the files have to be on the same file system. Hard links to directories or special files are impossible.

ln -s source destination
Create a symbolic (soft) link called destination to the file called source. The symbolic link just specifies a path where to look for the file. In contradistinction to hard links, the source and destination do not have to be on the same file system. Compared with hard links, the drawback of symbolic links is that if the original file is removed, the link is broken. Symbolic links can also create circular references (like circular references in spreadsheets or databases, ( e.g., a points to b and b points back to a).

rm files
Remove (delete) files. You must own the file in order to be able to remove it.

mkdir directory
Make a new directory.

rmdir directory
Remove an empty directory.

rm -r files
(Recursive remove) Remove files, directories, and their subdirectories. Be careful with this command when logged in as root-- you can easily remove all files on the system if you are not careful.

 cat filename | more
View the contents of a text file called filename, one page a time. The | is the pipe symbol (on many keyboards it is a shift \ ). The pipe makes the output stop after each screen. For long files, it is sometimes convenient to use the commands head and tail that display just the beginning and the end of the file. If you happened to use cat on a binary file and your terminal displays funny characters afterwards, you can restore it with the command reset.

less filename
Scroll through the contents of a text file. Press q when done. Less is roughly equivalent to more, the command you just read about.

pico filename
Edit a text file using the simple and standard text editor called pico. (comes with the pine mail program)

pico -w filename
Edit a text file, while disabling the long line wrap. Handy for editing configuration files. e.g. /etc/fstab. We advise you to disable the word wrap feature for the simple
reason that you can easily mess up the configuration files with Word Wrap on,  because the line length will differ from system to system.

lynx file.html
View an html file or browse the net from text mode.

tar -xvzf filename.tar.gz
Untar a tarred and compressed tarball (*.tar.gz or *.tgz) that you downloaded from the Internet.

tar -xvf filename.tar
Untar a tarred but uncompressed tarball (*.tar).

gunzip filename.gz
Decompress a zipped file (*.gz or *.z). Use gzip (also zip or compress) if you want to compress files.

bunzip2 filename.bz2
Decompress a file (*.bz2) zipped with bzip2 compression utility. Used for big files.

unzip filename.zip
Decompress a file (*.zip) zipped with a compression utility compatible with PKZIP for DOS.

 find / -name filename
Find the file called filename on your file system starting the search from the root  directory / . The filename may contain wildcards (*,?).

rsh server (remote shell)
Yet another way to connect to a remote machine. The login name/password from  your current session is used;  if it fails you are prompted for a password.

ftp server
Ftp to another machine.Ftp is good for copying files to/from a remote machine.
The essential ftp commands are:
ls - list the files on the remote system
ASCII - set the file transfer mode to text mode
binary set the file transfer mode to binary mode
get - copy a file from the remote system to the local system
mget - get many files at once - (use *.*)
put - copy a file from the local system to the remote system
mput - put many files at once
bye - disconnectfind / -name filename rsh server (remote shell)
For automation in a script, you may want to use ncftpput and ncftpget, for example: ncftpput -u my_user_name -p my_password -a remote.host.domain remote_dir *local.html

pine
A good text-mode mail reader. Pine will let you read your local mail, e.g. the mail you get from a computer on your home network.

talk username1
Talk to another user currently logged on your machine (or use talk username1@machinename to talk to a user on a different computer). To accept the invitation to the conversation, type the command talk username2. If somebody is trying to talk to you and it disrupts your work, your may use the command mesg n to refuse accepting messages. You may want to use who or rwho to determine the users who are currently logged-in.

mc
Launch the Midnight Commander file manager (looks like Norton Commander) for Linux.

telnet server
Connect to another machine using the TELNET protocol. Use a remote machine name or IP address.You will be prompted for your login name and password--you must have an account on the remote machine to login. Telnet will connect you to another machine and let you operate on it as if you were sitting at its keyboard (almost). Telnet is not very secure-- everything you type goes in open text, even your password!.

rlogin server
(Rremote login) Connect to another machine. The login name/password from your current session is used; if it fails you are prompted for a password. (Not very secure!)

minicom
Minicom program. It’s a communication program that somewhat resembles the famous old DOS communication program, Telix. You could always use your existing shell accounts via minicom.

./program_name
Run an executable in the current directory, which is not on your PATH

xinit
Start a barebone X-windows server (without a window manager).

startx
Start an X-windows server and the default window manager. Works like typing win under DOS with Win3.1

startx -- :1
Start another X-windows session on the display 1 (the default is opened on display 0). You can have several GUI terminals running concurrently. Switch between them using Ctrl+Alt+F7, Ctrl+Alt+F8, etc.

xterm (an X terminal)

Run a simple X-windows terminal. Typing exit will close it.

shutdown -h
Shut down the system to a halt. Allowed only when logged in as root. From the console you may use Ctrl+Alt+Del to issue a shutdown -r now. This can be done by any user.

halt
Halt the machine, simpler to type than the previous command.

man topic
Display the contents of the system manual pages (help) on the topic. Try man man first. Press q to quit the viewer. The command info works in a similar fashion and may contain more up-to-date information. Manual pages can sometimes be hard to read.

--help
Brings up a short, easy to digest help on a command. If you need more info, go to the directory /usr/doc and look for the command.


who
List the users logged in on the machine. --

rwho -a
List all users logged in on your network. The rwho service must be enabled for this command to work.

finger user_name
System info about a user. Try: finger root last. This lists the users last logged-in on your system.

history | more

Show the last (1000 or so) commands executed from the command line on the current account. The | more causes the display to stop after each screen fill.

pwd
Print working directory,

free
Memory info (in kilobytes)

hostname
Print the name of the local host

df -h
Print disk info about all the file systems in a human readable form.

whoami
Print your login name.


du / -bh | more
Print detailed disk usage for each subdirectory

killall -9 program_name
Kill program(s) by name.

lsmod
to see modules currently loaded.

xkill
Kill a GUI-based program

set
Show the current user environment.

lpc
(as root) Check and control the printer(s).
                          
id
Print user id (uid)

date
Print or change the operating system date and time. E.g., change the date and time to 2000-12-31 23:57 use this command date 123123572000

time
Determine the amount of time that it takes for a process to complete+ other info.

uptime
Amount of time since the last reboot

ps
List the processes that are beeing run by the current user.

echo $PATH
Show the content of the environment variable PATH. This command can be used to show other environment variables as well. Use set to see the full environment.

dmesg | less
Print kernel messages (the current content of the so-called kernel ring buffer). Press q to quit less. Use less /var/log/dmesg to see what dmesg dumped into the file right after bootup.

ps aux | more ps
List all the processes currently running, even those Display the list of currently running processes with their together with the name process IDs (PID) numbers.

top
Keep listing the currently running processes, sorted by cpu usage (top users first).


kill PID
Force a proc

lpq
Show the content of the printer queue.

lprm job_number
Remove a printing job job_number from the queue.

nice program_name
Run program_name adjusting its priority. Since the priority is not specified in this example, it will be adjusted by 10 (the process will run slower), from the default value (usually 0). The lower the number (of niceness to other users on the system), the higher the priority. The priority value may be in the range -20 to 19. Only root may specify negative values. Use top to display the priorities of the running processes.

fg
PID Bring a background or stopped process to the foreground.

uname -a
Info on your system.

Open-Source Management Systems

#Open-Source Management Systems

##Overview
This is a collection of open-source management systems tools.

###Monitoring
* [Nagios](http://www.nagios.org)
* [Icinga](http://www.icinga.org)
* [Zabbix](http://www.zabbix.com)
* [Hyperic](http://www.hyperic.com)
* [OpenNMS](http://www.opennms.org)
* [Reconnoiter](http://labs.omniti.com/labs/reconnoiter)
* [Flapjack](http://flapjack-project.com/)
* [Big Brother](http://bb4.com/)
* [Zenoss](http://www.zenoss.com)
* [OpenSmart](http://opensmart.sourceforge.net)
* [PandoraFMS](http://pandorafms.org)
* [Sensu](http://www.sonian.com/cloud-monitoring-sensu/)
* [Chukwa](http://incubator.apache.org/chukwa/)
* [Collectd](http://collectd.org)
* [Ganglia](http://ganglia.sourceforge.net)
* [Observium](http://www.observium.org/wiki/Main_Page)
* [Monit](http://mmonit.com/monit/)
* [Xymon (aka Hobbit)](http://xymon.sourceforge.net/)
* [Centreon](http://www.centreon.com/)
* [chinken](http://www.shinken-monitoring.org/)
* [OpenTSDB](http://opentsdb.net/)
* [Overwatch](https://github.com/danryan/overwatch)
* [Critical](https://github.com/danielsdeleo/critical)
* [ExtremeMon](https://extremon.org/)
* [Amon](https://github.com/martinrusev/amon)
* [Canopsis](http://www.canopsis.org/)
* [Riemann](http://aphyr.github.com/riemann/)
* [Diamond](http://opensource.brightcove.com/project/diamond)
* [Monitorix](http://www.monitorix.org/)

###Log Files
* [Flume](https://cwiki.apache.org/FLUME/)
* [Graylog2](http://graylog2.org)
* [LogStash](http://logstash.net/)
* [Kibana, UI for LogStash](http://kibana.org/)
* [Scribe](https://github.com/facebook/scribe)
* [Fluentd](http://fluentd.org/)
* [elasticsearch](http://www.elasticsearch.org/)
* [ossec](http://www.ossec.net/)
* [Lasso](http://sourceforge.net/projects/lassolog/)
* [rsyslog](http://www.rsyslog.com/)
* [log2timeline](https://code.google.com/p/log2timeline/)
* [plaso](http://log2timeline.kiddaland.net/)
* [LogHound](http://ristov.users.sourceforge.net/loghound/)
* [SLCT](http://ristov.users.sourceforge.net/slct/)
* [Logwatch](http://sourceforge.net/projects/logwatch/)
* [LogReport](http://www.logreport.org/)
* [LogSurfer](http://www.crypt.gen.nz/logsurfer/) - [(source)](http://sourceforge.net/projects/logsurfer/)

###Event Correlation
* [Esper](http://esper.codehaus.org/)
* [SEC](http://simple-evcorr.sourceforge.net/)

###Visualisation/Graphing
* [Graphite](http://graphite.wikidot.com)
* [Visage](https://github.com/auxesis/visage)
* [Cube](http://square.github.com/cube/)
* [Cacti](http://www.cacti.net/)
* [Dashku](https://github.com/Anephenix/dashku)
* [MRTG](http://oss.oetiker.ch/mrtg/)

###Packaging
* [0Install](http://0install.net/) - (all platforms)
* [WPKG](http://wpkg.org) - (Windows only)
* [Pakiti](http://pakiti.sourceforge.net) - (Linux only)
* [Npackd](http://code.google.com/p/windows-package-manager/) - (Windows only)
* [CoApp](http://coapp.org/) - (Windows only)
* [Pulp](http://pulpproject.org/) - (RH Linux only)
* [Pennyworth](https://github.com/heavywater/pennyworth)
* [FPM](https://github.com/jordansissel/fpm) - (Linux only)

###Configuration Management
* [Poni](http://melor.github.com/poni/index.html)
* [Puppet](http://puppetlabs.com)
* [Chef](http://www.opscode.com/chef/)
* [CFengine](http://www.cfengine.com)
* [Red Hat Spacewalk](http://spacewalk.redhat.com)
* [ZooKeeper](http://zookeeper.apache.org)
* [Noah](https://github.com/lusis/Noah)
* [Blueprint](http://devstructure.com/)
* [Ubuntu Landscape](http://www.canonical.com/enterprise-services/ubuntu-advantage/landscape) - the server component is not open-source
* [Bcfg2](http://trac.mcs.anl.gov/projects/bcfg2)
* [cDist](https://github.com/telmich/cdist)
* [Babushka](http://babushka.me/)
* [Ohai](http://wiki.opscode.com/display/chef/Ohai)
* [Facter](http://puppetlabs.com/puppet/related-projects/facter/)
* [Sprinkle](https://github.com/crafterm/sprinkle)
* [Ansible](http://ansible.github.com/)

###Deployment
* [glu](https://github.com/linkedin/glu)

###Provisioning
* [Cobbler](http://cobbler.github.com)
* [xCat](http://xcat.sourceforge.net)
* [Jumpsuit](http://jumpsuit.sysadmin.org.uk/)
* [Foreman](http://theforeman.org/)

###Orchestration
* [MCollective](http://docs.puppetlabs.com/mcollective/)
* [ControlTier](http://doc36.controltier.org/wiki/Main_Page)
* [RunDeck](http://rundeck.org)
* [Func](https://fedorahosted.org/func/)
* [Capistrano](http://en.wikipedia.org/wiki/Capistrano)
* [Mina](http://nadarei.co/mina/)
* [Salt](http://saltstack.org/)
* [Fabric](http://fabfile.org)
* [Nanite](https://github.com/ezmobius/nanite)
* [juju](https://juju.ubuntu.com/) (formerly Ensemble)
* [brooklyn](http://brooklyn.io)
* [Synapse](http://comodit.github.com/synapse-agent/) - [introduction blog post](http://www.comodit.com/2012/09/10/synapse---an-open-source-solution-to-remotely-manage-a-large-number-of-hosts/)

###IPAM/Asset Management
* [OpenNetAdmin](http://opennetadmin.com)
* [Fusion Inventory](http://www.fusioninventory.org)
* [Racktables](http://racktables.org/)
* [Netdot](https://osl.uoregon.edu/redmine/projects/netdot)
* [Netdisco](http://www.netdisco.org/)
* [nventory](http://sourceforge.net/apps/trac/nventory)

###Backup
* [Bacula](http://www.bacula.org/)
* [Amanda](http://www.amanda.org/)
* [Backup RubyGem](https://github.com/meskyanichi/backup) - See [getting started](https://github.com/meskyanichi/backup/wiki/Getting-Started) if ruby scares you

Tuesday 5 February 2013

Use of linux commands


#=======================================================================================================
# 1. PERL ONE-LINERS
#=======================================================================================================
#-----------------------------------------------------------------------
# Display Table of WORD in FILE*
#-----------------------------------------------------------------------
# Displays all occurrences of WORD in the FILE*(s), formatting as a
# table with filename, line number and line text.
#-----------------------------------------------------------------------
# Simple example for short filenames.  Seems to cause trouble in Windows command line
grep -in WORD FILE* | perl -ne 'if (/(.*):(\d*):\s*(.*)\s*/) { print $1, " " x (20 - length $1), " ", "0" x (5 - length $2), $2, "   $3\n"}'

# More complex example using egrep and assuming longer filenames (note the '50' in the spaces computation)
# Also seems to cause trouble in Windows command line.
egrep -in '(Alabama|"AL")' *.asp | perl -ne 'if (/(.*):(\d*):\s*(.*)\s*/) { print $1, " " x (50 - length $1), " ", "0" x (5 - length $2), $2, "   $3\n"}'

#-----------------------------------------------------------------------
# Text Trimming and Processing
#-----------------------------------------------------------------------
# Trim excess blanks from beginning to end of line. A Cygwin creature.
# cat myfile | perl -ne 'if (/^\s*(.*?)\s*$/s) { print "${1}\n" }' # seems less efficient in practice
perl -ne 'if (/^\s*(.*?)\s*$/s) { print "${1}\n" }' < temp.txt

# Don't forget the tr command! Not that we have use right now...


#=======================================================================================================
# 2. Unix FIND one liners
#=======================================================================================================
find .                    # Find all files under .
find . -type d                # Find all subdirectories.
find . -iregex ".*\(bas\|cls\|frm\)"    # Find all Visual Basic code files in a directory
                    # -iregex matches a case-insensitive regular expression
                    # backslashes necessary to protect special characters at shell

# Find all VB files containing a given string
# Note you must escape the {} and the ; because we are at the shell
find . -iregex ".*\(bas\|cls\|frm\)" -exec grep NIBRS \{\} \;                   

# Find all VB files containing comment lines
# Note you must escape the {} and the ; because we are at the shell
find . -iregex ".*\(bas\|cls\|frm\)" -exec egrep "^*'" \{\} \;

# Find all VB files containing NON comment lines
# Note you must escape the {} and the ; because we are at the shell
find . -iregex ".*\(bas\|cls\|frm\)" -exec egrep -v "^*'" \{\} \;

# Find all VB files containing NON comment, NON blank lines in a directory
find . -iregex ".*\(bas\|cls\|frm\)" -exec egrep -v "^*'|^[[:space:]]*$" \{\} \;

# Count the code in a directory hierarchy
find . -iregex ".*\(bas\|cls\|frm\)" -exec egrep -v "^*'|^[[:space:]]*$" \{\} \; | wc


#=======================================================================================================
# 3. Grep one liners
#=======================================================================================================
grep -R "Stuff" .            # Find all files in a subdirectory containing "Stuff"
grep -R --include "*.asp" "Stuff" .    # Find all .asp files in a directory tree containing "Stuff"

grep '"b"' * | cut -d":" -f1         # List of all filenames matching a pattern

# Long list of all the files matching a pattern
for file in $(grep '"b"' * | cut -d":" -f1 ); do ls -l $file; done

# Compare files with one extension with VI modified versions, redirecting diff error if ~ file not found
for file in *.cmd; do echo; echo $file; diff $file ${file}~ 2>&1; done > diffs.txt

#=======================================================================================================
# 4. File/Directory one liners
#=======================================================================================================
ls -R                    # Recursively list contents of subdirectories


                    # Two different formats for the subdirectories of just this directory
ls | xargs perl -e '@ARGV = grep( -d $_ , @ARGV); print "@ARGV\n"'   
ls | xargs perl -e '@ARGV = grep( -d $_ , @ARGV); while($dir = shift @ARGV) { print "$dir\n" }'

# Subdirectory sizes
du                    # Show sizes of subdirectories
du -h                    # Make the sizes more humanly readable
du -h --max-depth=1            # Display the sizes of just one level of subdirectories
du -h --summarize *             # Summarize the elements of this directory, humanly

# File Sizes

# Reversing a file
tac temp.txt > pmet.txt            # Reverse the order of the lines in a file using 'tac' (cat reverse) command
rev temp.txt > pmet.txt            # Reverse the text in each line of a file using 'rev' (reverse line) command

# cat all the files in a directory
for file in myDir/* ; do echo; echo "---$file--" ; cat $file ; done

# Sum the line counts of all the code files in a directory
find . -iregex ".*\(java\|html\|txt\)" -exec wc \{\} \; | gawk '{ print $1 "\t" $4; sum += $1 } END { print "--------"; print sum "\tTOTAL" }'


#=======================================================================================================
# 5. Path Parsing One Liners
#=======================================================================================================
# Can't use EXPR because it only matches at the beginning of a line.
if echo $PATH | grep -q "Program" ; then echo "present"; else echo "absent"; fi
if echo $PATH | grep -q "\." ; then echo ". present"; else export PATH=".:$PATH"; echo "added . to path";  fi
perl -e "print \"$PATH\" =~ /Program/"    # not good for testing

# See if a variable exists
if [ "$fram" ]; then echo "found a fram"; else echo "no fram here"; fi

# Use Python to print all the variable names on one line - note this can futz up some pathnames
python -c "for path in '%CLASSPATH%'.split(';'): print path"

# Use Perl to do the same thing
perl -e "print join qq(\n), split /;/, '%CLASSPATH%' "
perl -e "foreach (split ';', '%CLASSPATH%') { print; print qq(\n) }"

#=======================================================================================================
# 6. AWK File Parsing One Liners
#=======================================================================================================
# Get a sorted list of values from the first column of a tab-separated file
gawk '{ print $1 }' tabfile.txt | sort | uniq > firstcolvals.txt

# Get a sorted list of column values from a file with fields split by dashes
gawk 'BEGIN { FS="--" } ; { print $1 }' data.txt | sort | uniq > firstcolvals.txt

# Reverse fields in a tab-separated file and sort it by the numeric field
gawk 'BEGIN { FS="\t" } ; { print $2 "\t" $1 }' data.txt  | sort -nr > sorted.txt

# Reverse fields in a tab-separated file and sort it by the numeric field
gawk 'BEGIN { FS="\t" } ; { print $2 "\t" $1 }' data.txt  | sort -nr > sorted.txt

#
gawk '{ freq[$0]++ } ; END { sort = "sort --key=2" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' allprops.txt

# Extract the first field, collect the count of it, and output the word, counts sorted by name
gawk 'BEGIN { FS="--" } ; { freq[$1]++ } ; END { sort = "sort" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' data.txt

# Extract the first field, collect the count of it, and output the word, counts sorted by count
gawk 'BEGIN { FS="--" } ; { freq[$1]++ } ; END { sort = "sort --key=2 -nr" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' data.txt


#=======================================================================================================
# 7. DOS Fu One Liners
#=======================================================================================================
#-----------------------------------------------------------------------
# 7a. Windows Equivalents of Common UNIX Commands
#-----------------------------------------------------------------------
# Iterate through all the lines in a file, doing something
for /F %i IN (lines.txt) DO echo %i

#-----------------------------------------------------------------------
# 7b. OTHER CODE SNIPPETS
#-----------------------------------------------------------------------
# Windows Prompt
prompt $P $C$D$T$F: $_$+$G$S


#=======================================================================================================
# Z. ONE-LINES WRITTEN BY OTHER AUTHORS
#=======================================================================================================
#-----------------------------------------------------------------------
# FROM "Perl One-Liners" by Jeff Bay jlb0170@yahoo.com
#-----------------------------------------------------------------------
# Find the sum of the sizes of the non-directory files in a directory
ls -lAF | perl -e 'while (<>) { next if /^dt/; $sum += (split)[4] } print "$sum\n"'

#-----------------------------------------------------------------------
# Other One-Liners
#-----------------------------------------------------------------------
# Find all Perl modules - From Active Perl documentation
find `perl -e 'print "@INC"'` -name '*.pm' -print


Free-UP momory
echo 1 > /proc/sys/vm/drop_caches

kill pid
kill -9 `ps -ef |  grep http | grep -v grep  | awk '{print $2}'`

list only files
ls -l | awk 'NR!=1 && !/^d/ {print $NF}'

list only dirs
ls -l | awk 'NR!=1 && /^d/ {print $NF}'

log from to date
awk '$0>=from&&$0<=to' from="2009-08-10 00:00" to="2009-08-10 23:49" log4j.output.4

to see process
ps -U mailman  -o pid,%cpu,%mem,stime,time,vsz,args

to delete connection time out mails from postfix mail queue
cat /var/log/maillog | grep "Connection timed out" | awk '{print $6}' | cut -d: -f1 | /var/postfix/sbin/postsuper -d -

replace ipaddress
perl -pi -e "s/172.30.1.10/172.30.254.5/;" /etc/sysconfig/network-scripts/ifcfg-bond0

to show all file except 2 first lines
sed -n '3,$p'

delete many files
find . -type f -exec rm -v {} \;

remote incremental backup
ssh -l root main.server.com "tar -c -f - --incremental-listed=/var/tar-snapshot-file.txt /my/dir/to/backup" > my_local_backup.tar

to find which file has text examlestring from all folders
find . -type f -print0 | xargs -0 grep "examplestring"



To remove blank lines from a file using sed, use the following:

sed -e '/^$/d' filetoread >filetowrite

The ^ matches the beginning of a line and the $ matches the end.
The two of them together matches a line that begins and ends with
nothing in between (blank line).
The d just says delete the lines for which we have a match.
Since the standard operation of sed is to print every line,
all lines exept blank lines will be sent to filetowrite.
===============================================================
ZERO THOSE LOG FILES


Some programs write to multiple
log files in a directory and
need to be zeroed out sometimes
to save diskspace. The following
ksh shell script will zero out
all files with the ".log"
extension in a directory.

--- cut here ---
for object in *.log
do
> $object
print "$object has been zeroed!"
done
--- cut here ---

Just a little time saver when
you have 100 other things to
be doing.
================================================================
WHAT FILE IS THAT IN?

Did you ever just want to know
what files search patterns were
in but didn't want to see the
search results?

Turn this:
$ grep quick *

story1.txt: The quick brown fox
story1.txt: The turtle was not quick.
recipe.txt: How to make soup quick.

Into this:
$ grep -l quick *

story1.txt
recipe.txt
=================================================================
EDIT RUN EDIT

Here is a simple script that will
edit a file, run it and edit it again.
This is useful when you're doing alot
of testing against a script.

Replace "file" with the name of the script.

------------- cut here ---------------
while [ 1 ]
do
  vi file
  clear; echo
  chmod 755 file
  ./file
  echo; echo  -n "[pause] "
  read pause
done


------------- cut here ---------------

==================================================================

How to delete blank lines within vi?

Just type this baby in:

    <esc>:g/^$/d


NOTE:  This means that all the lines that just have a
       carriage return on them (NO Spaces), will be removed.




Ok, so I have some of those lines too.  How can I remove all of them as well?

    <esc>:g/^ *$/d


NOTE: There is a space after the '^' and before the '*'.
=================================================================
CHANGING PERMS RECURSIVELY

*** NOTE: Odly enough, the previous mailing
of this tip did not go through do to a
permissions problem. Sorry.

To change permissions recursively for all files in a
directory

find dirname -exec chmod xxx {} \; -print

where dirname is the directory you want to change permissions.
==================================================================
BASH HOTKEYS

Bash provides many hot keys to ease use. Like
ctrl-l  -- clear screen
ctrl-r  -- does a search in the previously given commands so that you don't
have to repeat long command.
ctrl-u  -- clears the typing before the hotkey.
ctrl-a  -- takes you to the begining of the command you are currently typing.
ctrl-e  -- takes you to the end of the command you are currently typing in.
esc-b   -- takes you back by one word while typing a command.
ctrl-c  -- kills the current command or process.
ctrl-d  -- kills the shell.
ctrl-h  -- deletes one letter at a time from the command you are typing in.
ctrl-z  -- puts the currently running process in background, the process
can be brought back to run state by using fg command.
esc-p  -- like ctrl-r lets you search through the previously given commands.
esc-.  -- gives the last command you typed.
======================================================================
KILLING MORE USERS

*** SORRY ABOUT THE PEVIOUS NULL TIP IF YOU RECEIVED IT****

To kill all processes of a particular user from root
at unix prompt type:

# kill -9 `ps -fu username |awk '{ print $2 }'|grep -v PID`

We can also use the username as an argument and pass it from
command line, if this command is put as a script.

====================================================
STRING REMOVAL

What the following does:

rm `ls -al | grep str | awk '{if ($9 !~ /^str/) {print $9}'`

Removes all files that contains the string "str" excepts
those that begin with it. Changing the !~ to =~ does the
opposite.

======================================================================
TRACKING OF LOGINS AND LOGOUTS


In the .login file add the commands:
------------------------------------

echo login time `date` >> .daylogs/masterlog

grep -i "sun" .daylogs/masterlog > .daylogs/sunday.log
grep -i "mon" .daylogs/masterlog > .daylogs/monday.log
grep -i "tue" .daylogs/masterlog > .daylogs/tuesday.log
grep -i "wen" .daylogs/masterlog > .daylogs/wensday.log
grep -i "thu" .daylogs/masterlog > .daylogs/thursday.log
grep -i "fri" .daylogs/masterlog > .daylogs/friday.log
grep -i "sat" .daylogs/masterlog > .daylogs/saturday.log


In the .logout file add this line
-----------------------------------
==================================================================
find . -name 'tmp*' -print0 | xargs -0 rm -rf


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
to change command type

export PS1='\u@\h - \w >> '
----------------------------------------------------------------
To Enable ipconntrack

modprobe ip_conntrack hashsize=131072
modprobe iptable_nat

----------------------------------------------------------------
to use proxy in linux command line
export http_proxy=http://'system:syst3m'@mails.poweryourtrade.com:3389

----------------------------------------------------------------

uniq(1) takes a stream of lines and collapses adjacent duplicate lines into one copy of the lines. So if you had a file called foo that looked like:
--------------------------------------------------------------------------------

davel
davel
davel
jeffy
jones
jeffy
mark
mark
mark
chuck
bonnie
chuck


--------------------------------------------------------------------------------
You could run uniq on it like this:
--------------------------------------------------------------------------------

% uniq foo
davel
jeffy
jones
jeffy
mark
chuck
bonnie
chuck


--------------------------------------------------------------------------------
Notice that there are still two jeffy lines and two chuck lines. This is because the duplicates were not adjacent. To get a true unique list you have to make sure the stream is sorted:
--------------------------------------------------------------------------------

% sort foo | uniq
jones
bonnie
davel
chuck
jeffy
mark


--------------------------------------------------------------------------------
That gives you a truly unique list. However, it's also a useless use of uniq since sort(1) has an argument, -u to do this very common operation:
--------------------------------------------------------------------------------

% sort -u foo
jones
bonnie
davel
chuck
jeffy
mark


--------------------------------------------------------------------------------
That does exactly the same thing as "sort | uniq", but only takes one process instead of two.
uniq has other arguments that let it do more interesting mutilations on its input:

-d tells uniq to eliminate all lines with only a single occurrence (delete unique lines), and print just one copy of repeated lines:
--------------------------------------------------------------------------------

% sort foo | uniq -d
davel
chuck
jeffy
mark


--------------------------------------------------------------------------------

-u tells uniq to eliminate all duplicated lines and show only those which appear once (only the unique lines):
--------------------------------------------------------------------------------

% sort foo | uniq -u
jones
bonnie


--------------------------------------------------------------------------------

-c tells uniq to count the occurrences of each line:
--------------------------------------------------------------------------------

% sort foo | uniq -c
   1 jones
   1 bonnie
   3 davel
   2 chuck
   2 jeffy
   3 mark


--------------------------------------------------------------------------------
I often pipe the output of "uniq -c" to "sort -n" (sort in numeric order) to get the list in order of frequency:
--------------------------------------------------------------------------------

% sort foo | uniq -c | sort -n
   1 jones
   1 bonnie
   2 chuck
   2 jeffy
   3 davel
   3 mark


--------------------------------------------------------------------------------

Finally, there are arguments to make uniq ignore leading characters and fields. See the man page for details.


for loop for finding IP

for i in `cat 1.txt `;do echo -n $i "  " ; ssh $i /sbin/ifconfig eth0 | sed -n 's/.*addr\:\([0-9\.]*\).*/\1/p' ;done

sed one liner


FILE SPACING:

 # double space a file
 sed G

 # double space a file which already has blank lines in it. Output file
 # should contain no more than one blank line between lines of text.
 sed '/^$/d;G'

 # triple space a file
 sed 'G;G'

 # undo double-spacing (assumes even-numbered lines are always blank)
 sed 'n;d'

 # insert a blank line above every line which matches "regex"
 sed '/regex/{x;p;x;}'

 # insert a blank line below every line which matches "regex"
 sed '/regex/G'

 # insert a blank line above and below every line which matches "regex"
 sed '/regex/{x;p;x;G;}'

NUMBERING:

 # number each line of a file (simple left alignment). Using a tab (see
 # note on '\t' at end of file) instead of space will preserve margins.
 sed = filename | sed 'N;s/\n/\t/'

 # number each line of a file (number on left, right-aligned)
 sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'

 # number each line of file, but only print numbers if line is not blank
 sed '/./=' filename | sed '/./N; s/\n/ /'

 # count lines (emulates "wc -l")
 sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:

 # IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # works on ssed, gsed 3.02.80 or higher

 # IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$/`echo -e \\\r`/"            # command line under ksh
 sed 's/$'"/`echo \\\r`/"             # command line under bash
 sed "s/$/`echo \\\r`/"               # command line under zsh
 sed 's/$/\r/'                        # gsed 3.02.80 or higher

 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$//"                          # method 1
 sed -n p                             # method 2

 # IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 # Can only be done with UnxUtils sed, version 4.0.7 or higher. The
 # UnxUtils version can be identified by the custom "--text" switch
 # which appears when you use the "--help" switch. Otherwise, changing
 # DOS newlines to Unix newlines cannot be done with sed in a DOS
 # environment. Use "tr" instead.
 sed "s/\r//" infile >outfile         # UnxUtils sed v4.0.7 or higher
 tr -d \r <infile >outfile            # GNU tr version 1.22 or higher

 # delete leading whitespace (spaces, tabs) from front of each line
 # aligns all text flush left
 sed 's/^[ \t]*//'                    # see note on '\t' at end of file

 # delete trailing whitespace (spaces, tabs) from end of each line
 sed 's/[ \t]*$//'                    # see note on '\t' at end of file

 # delete BOTH leading and trailing whitespace from each line
 sed 's/^[ \t]*//;s/[ \t]*$//'

 # insert 5 blank spaces at beginning of each line (make page offset)
 sed 's/^/     /'

 # align all text flush right on a 79-column width
 sed -e :a -e 's/^.\{1,78\}$/ &/;ta'  # set at 78 plus 1 space

 # center all text in the middle of 79-column width. In method 1,
 # spaces at the beginning of the line are significant, and trailing
 # spaces are appended at the end of the line. In method 2, spaces at
 # the beginning of the line are discarded in centering the line, and
 # no trailing spaces appear at the end of lines.
 sed  -e :a -e 's/^.\{1,77\}$/ & /;ta'                     # method 1
 sed  -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/'  # method 2

 # substitute (find and replace) "foo" with "bar" on each line
 sed 's/foo/bar/'             # replaces only 1st instance in a line
 sed 's/foo/bar/4'            # replaces only 4th instance in a line
 sed 's/foo/bar/g'            # replaces ALL instances in a line
 sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
 sed 's/\(.*\)foo/\1bar/'            # replace only the last case

 # substitute "foo" with "bar" ONLY for lines which contain "baz"
 sed '/baz/s/foo/bar/g'

 # substitute "foo" with "bar" EXCEPT for lines which contain "baz"
 sed '/baz/!s/foo/bar/g'

 # change "scarlet" or "ruby" or "puce" to "red"
 sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'   # most seds
 gsed 's/scarlet\|ruby\|puce/red/g'                # GNU sed only

 # reverse order of lines (emulates "tac")
 # bug/feature in HHsed v1.5 causes blank lines to be deleted
 sed '1!G;h;$!d'               # method 1
 sed -n '1!G;h;$p'             # method 2

 # reverse each character on the line (emulates "rev")
 sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

 # join pairs of lines side-by-side (like "paste")
 sed '$!N;s/\n/ /'

 # if a line ends with a backslash, append the next line to it
 sed -e :a -e '/\\$/N; s/\\\n//; ta'

 # if a line begins with an equal sign, append it to the previous line
 # and replace the "=" with a single space
 sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

 # add commas to numeric strings, changing "1234567" to "1,234,567"
 gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta'                     # GNU sed
 sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta'  # other seds

 # add commas to numbers with decimal points and minus signs (GNU sed)
 gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'

 # add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
 gsed '0~5G'                  # GNU sed only
 sed 'n;n;n;n;G;'             # other seds

SELECTIVE PRINTING OF CERTAIN LINES:

 # print first 10 lines of file (emulates behavior of "head")
 sed 10q

 # print first line of file (emulates "head -1")
 sed q

 # print the last 10 lines of a file (emulates "tail")
 sed -e :a -e '$q;N;11,$D;ba'

 # print the last 2 lines of a file (emulates "tail -2")
 sed '$!N;$!D'

 # print the last line of a file (emulates "tail -1")
 sed '$!d'                    # method 1
 sed -n '$p'                  # method 2

 # print the next-to-the-last line of a file
 sed -e '$!{h;d;}' -e x              # for 1-line files, print blank line
 sed -e '1{$q;}' -e '$!{h;d;}' -e x  # for 1-line files, print the line
 sed -e '1{$d;}' -e '$!{h;d;}' -e x  # for 1-line files, print nothing

 # print only lines which match regular expression (emulates "grep")
 sed -n '/regexp/p'           # method 1
 sed '/regexp/!d'             # method 2

 # print only lines which do NOT match regexp (emulates "grep -v")
 sed -n '/regexp/!p'          # method 1, corresponds to above
 sed '/regexp/d'              # method 2, simpler syntax

 # print the line immediately before a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{g;1!p;};h'

 # print the line immediately after a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{n;p;}'

 # print 1 line of context before and after regexp, with line number
 # indicating where the regexp occurred (similar to "grep -A1 -B1")
 sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

 # grep for AAA and BBB and CCC (in any order)
 sed '/AAA/!d; /BBB/!d; /CCC/!d'

 # grep for AAA and BBB and CCC (in that order)
 sed '/AAA.*BBB.*CCC/!d'

 # grep for AAA or BBB or CCC (emulates "egrep")
 sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d    # most seds
 gsed '/AAA\|BBB\|CCC/!d'                        # GNU sed only

 # print paragraph if it contains AAA (blank lines separate paragraphs)
 # HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'

 # print paragraph if it contains AAA and BBB and CCC (in any order)
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'

 # print paragraph if it contains AAA or BBB or CCC
 sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
 gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d'         # GNU sed only

 # print only lines of 65 characters or longer
 sed -n '/^.\{65\}/p'

 # print only lines of less than 65 characters
 sed -n '/^.\{65\}/!p'        # method 1, corresponds to above
 sed '/^.\{65\}/d'            # method 2, simpler syntax

 # print section of file from regular expression to end of file
 sed -n '/regexp/,$p'

 # print section of file based on line numbers (lines 8-12, inclusive)
 sed -n '8,12p'               # method 1
 sed '8,12!d'                 # method 2

 # print line number 52
 sed -n '52p'                 # method 1
 sed '52!d'                   # method 2
 sed '52q;d'                  # method 3, efficient on large files

 # beginning at line 3, print every 7th line
 gsed -n '3~7p'               # GNU sed only
 sed -n '3,${p;n;n;n;n;n;n;}' # other seds

 # print section of file between two regular expressions (inclusive)
 sed -n '/Iowa/,/Montana/p'             # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:

 # print all of file EXCEPT section between 2 regular expressions
 sed '/Iowa/,/Montana/d'

 # delete duplicate, consecutive lines from a file (emulates "uniq").
 # First line in a set of duplicate lines is kept, rest are deleted.
 sed '$!N; /^\(.*\)\n\1$/!P; D'

 # delete duplicate, nonconsecutive lines from a file. Beware not to
 # overflow the buffer size of the hold space, or else use GNU sed.
 sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

 # delete all lines except duplicate lines (emulates "uniq -d").
 sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'

 # delete the first 10 lines of a file
 sed '1,10d'

 # delete the last line of a file
 sed '$d'

 # delete the last 2 lines of a file
 sed 'N;$!P;$!D;$d'

 # delete the last 10 lines of a file
 sed -e :a -e '$d;N;2,10ba' -e 'P;D'   # method 1
 sed -n -e :a -e '1,10!{P;N;D;};N;ba'  # method 2

 # delete every 8th line
 gsed '0~8d'                           # GNU sed only
 sed 'n;n;n;n;n;n;n;d;'                # other seds

 # delete lines matching pattern
 sed '/pattern/d'

 # delete ALL blank lines from a file (same as "grep '.' ")
 sed '/^$/d'                           # method 1
 sed '/./!d'                           # method 2

 # delete all CONSECUTIVE blank lines from file except the first; also
 # deletes all blank lines from top and end of file (emulates "cat -s")
 sed '/./,/^$/!d'          # method 1, allows 0 blanks at top, 1 at EOF
 sed '/^$/N;/\n$/D'        # method 2, allows 1 blank at top, 0 at EOF

 # delete all CONSECUTIVE blank lines from file except the first 2:
 sed '/^$/N;/\n$/N;//D'

 # delete all leading blank lines at top of file
 sed '/./,$!d'

 # delete all trailing blank lines at end of file
 sed -e :a -e '/^\n*$/{$d;N;ba' -e '}'  # works on all seds
 sed -e :a -e '/^\n*$/N;/\n$/ba'        # ditto, except for gsed 3.02.*

 # delete the last line of each paragraph
 sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:

 # remove nroff overstrikes (char, backspace) from man pages. The 'echo'
 # command may need an -e switch if you use Unix System V or bash shell.
 sed "s/.`echo \\\b`//g"    # double quotes required for Unix environment
 sed 's/.^H//g'             # in bash/tcsh, press Ctrl-V and then Ctrl-H
 sed 's/.\x08//g'           # hex expression for sed 1.5, GNU sed, ssed

 # get Usenet/e-mail message header
 sed '/^$/q'                # deletes everything after first blank line

 # get Usenet/e-mail message body
 sed '1,/^$/d'              # deletes everything up to first blank line

 # get Subject header, but remove initial "Subject: " portion
 sed '/^Subject: */!d; s///;q'

 # get return address header
 sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

 # parse out the address proper. Pulls out the e-mail address by itself
 # from the 1-line return address header (see preceding script)
 sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'

 # add a leading angle bracket and space to each line (quote a message)
 sed 's/^/> /'

 # delete leading angle bracket & space from each line (unquote a message)
 sed 's/^> //'

 # remove most HTML tags (accommodates multiple-line tags)
 sed -e :a -e 's/<[^>]*>//g;/</N;//ba'

 # extract multi-part uuencoded binaries, removing extraneous header
 # info, so that only the uuencoded portion remains. Files passed to
 # sed must be passed in the proper order. Version 1 can be entered
 # from the command line; version 2 can be made into an executable
 # Unix shell script. (Modified from a script by Rahul Dhesi.)
 sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode   # vers. 1
 sed '/^end/,/^begin/d' "$@" | uudecode                    # vers. 2

 # sort paragraphs of file alphabetically. Paragraphs are separated by blank
 # lines. GNU sed uses \v for vertical tab, or any unique char will do.
 sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
 gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'

 # zip up each .TXT file individually, deleting the source file and
 # setting the name of each .ZIP file to the basename of the .TXT file
 # (under DOS: the "dir /b" switch returns bare filenames in all caps).
 echo @echo off >zipup.bat
 dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat

TYPICAL USE: Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:

 cat filename | sed '10q'        # uses piped input
 sed '10q' filename              # same effect, avoids a useless "cat"
 sed '10q' filename > newfile    # redirects output to disk

For additional syntax instructions, including the way to apply editing
commands from a disk file instead of the command line, consult "sed &
awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly,
1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty
and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst
distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power
of sed, one must understand "regular expressions." For this, see
"Mastering Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997).
The manual ("man") pages on Unix systems may be helpful (try "man
sed", "man regexp", or the subsection on regular expressions in "man
ed"), but man pages are notoriously difficult. They are not written to
teach sed use or regexps to first-time users, but as a reference text
for those already acquainted with these tools.

QUOTING SYNTAX: The preceding examples use single quotes ('...')
instead of double quotes ("...") to enclose editing commands, since
sed is typically used on a Unix platform. Single quotes prevent the
Unix shell from intrepreting the dollar sign ($) and backquotes
(`...`), which are expanded by the shell if they are enclosed in
double quotes. Users of the "csh" shell and derivatives will also need
to quote the exclamation mark (!) with the backslash (i.e., \!) to
properly run the examples listed above, even within single quotes.
Versions of sed written for DOS invariably require double quotes
("...") instead of single quotes to enclose editing commands.

USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used
the expression '\t' to indicate a tab character (0x09) in the scripts.
However, most versions of sed do not recognize the '\t' abbreviation,
so when typing these scripts from the command line, you should press
the TAB key instead. '\t' is supported as a regular expression
metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80.

VERSIONS OF SED: Versions of sed do differ, and some slight syntax
variation is to be expected. In particular, most do not support the
use of labels (:name) or branch instructions (b,t) within editing
commands, except at the end of those commands. We have used the syntax
which will be portable to most users of sed, even though the popular
GNU versions of sed allow a more succinct syntax. When the reader sees
a fairly long command such as this:

   sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d

it is heartening to know that GNU sed will let you reduce it to:

   sed '/AAA/b;/BBB/b;/CCC/b;d'      # or even
   sed '/AAA\|BBB\|CCC/b;d'

In addition, remember that while many versions of sed accept a command
like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which
contains space before the 's'. Omit the space when typing the command.

OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to
large input files or slow processors or hard disks), substitution will
be executed more quickly if the "find" expression is specified before
giving the "s/.../.../" instruction. Thus:

   sed 's/foo/bar/g' filename         # standard replace command
   sed '/foo/ s/foo/bar/g' filename   # executes more quickly
   sed '/foo/ s//bar/g' filename      # shorthand sed syntax

On line selection or deletion in which you only need to output lines
from the first part of the file, a "quit" command (q) in the script
will drastically reduce processing time for large files. Thus:

   sed -n '45,50p' filename           # print line nos. 45-50 of a file
   sed -n '51q;45,50p' filename       # same, but executes much faster

Shell One Liner

Numbers and Counting

Print the total number of lines that have the name Bill.

awk '/Bill/{n++}; END {print n+0}' filename

Print line numbers using a tab instead of a space.

awk '{print FNR "\t" $0}' filename

Use awk to pull the seventh field (ex. url string) of each line of the logfile (logfile should be separated by spaces). Sort the fields then find the unique fields and count them. Then do a reverse sort on numeric count. Filter out anything but JPEG files and only give me the top 10 of that list. This is for things like counting unique hostnames or urls from a logfile.

awk '{print $7}' logfile | sort | uniq -c | sort -rn | grep "\.jpg" | head
Fields

Print the last field of each line of the file.

awk '{ print $NF }' filename

Count the lines in a file. Just like "wc -l".

awk 'END{print NR}' filename

Total the number of lines that contain the name Jose

awk '/Jose/{n++}; END {print n+0}' filename

Add up the numbers at the eighth position (field) of each line. Print the total.

awk '{ sum += $8 } END { print sum }' filename

If the fourth field starts with a number then print that fourth field.

awk '$4 ~ /^[0-9]/ { print $4 }' filename
Text Conversion and Substitution

Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.

awk '{gsub(/dog|cat|bird,"pet");print}' filename

Find and replace "dog" with "cat" in every file with extension txt.

awk '{gsub("dog", "cat", $0); print > FILENAME}' *.txt

Find every line that begins with cat. In that line replace furry with nothing. Change the file called filename inline (-i).

sed -i '/^cat/{s/furry//}' filename

Find cat by itself on it's own line even if there are spaces or tabs before it or after it. Replace it with dog. Then print the line.

awk '{sub(/^[ \t]*cat .*$/,"dog");print}' filename

Find any line starting with the defined shell variable SHELLVAR (notice ' ' around it so it's evaluated). When the line is found substitute in foo or boo or coo with bar. Edit the file inline.

sed -i '/^'${SHELLVAR}'/s/\(foo\|boo\|coo\)/bar/' filename

From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).

awk '{sub(/\r$/,"");print}' filename

Remove all carriage returns from file and rewrite the edits back to same file. tr uses the octal form for cr.

tr -d "\015" < filename | tee > filename

From a unix os: Convert Unix newlines (LF) to DOS format.

awk '{sub(/$/,"\r");print} filename

Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.

awk '{sub(/^[ \t]+/, ""); print}' filename

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}' filename

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}' filename

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}' filename

Insert 6 blank spaces at beginning of each line.

awk '{sub(/^/, "      ");print}' filename

Substitute "dog" with "cat" on lines that don't contain the word "pet".

awk '!/pet/{gsub(/dog/, "cat")};{print}' filename

Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).

awk -F: '{print $1 " " $2}' filename

Swap the first 2 fields.

awk '{tmp = $1; $1 = $2; $2 = tmp; print}' filename

Remove the second field in each line and then print it.

awk '{ $1 = ""; print }' filename
Line Operations

Print the first 6 lines of a file.

awk 'NR <= 6' filename

Print the last line of a file

awk 'END{print}' filename

Print the lines matching the regular expression. (emulates grep).

awk '/regex_here/' filename

Print the lines that don't match the regular expression. (emulates grep -v).

awk '!/regex_here/' filename

Print the line before the regular expression match.

awk '/regex_here/{print x};{x=$0}' filename

Print the line after the regular expression match.

awk '/regex_here/{getline; print}' filename

Print the lines less than 50 characters.

awk 'length < 50' filename

Print the lines 20 through 30.

awk 'NR==20,NR==30' filename

Print the line 50.

awk 'NR==50 {print;exit}' filename

Print lines between the match starting at "Dog" and ending at "Cat".

awk '/Dog/,/Cat/' filename
File and Process Operations

Find a program by name from process listing that is not awk and kill it. Or try the programs pkill or killall.

ps aux | awk '/program_name/ && !/awk/ {print $2}' > kill

Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.

dd if=/dev/zero of=/tmp/zero bs=1k count=2000

From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.

find / -type f -name "*.txt" -print | xargs rm

To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.

find . -inum 128128 | xargs rm

Mark the end of each line with a dollar sign so you can see where the filename ends. Good for finding file names with spaces at the end.

ls -lA | cat -e

To delete files with control characters in them like ^M or ^L use the control-V shell feature. This tells many shells to interpret the next input character as a literal character (instead of as a control character). Like below to delete a file with a space before the ctrl-L " ^L" you would issue the following keystrokes in this order (separated by commas) r,m, ,\, ,ctrl-v,ctrl-l. The \ escapes the space. The command looks like:

rm \ ^L

Synchronize files in a directory between 2 hosts using the program rsync. host1's /disk01 (source) is the remote host and /disk01 (destination) is a local directory. The destination is always made to look like the source even if files need to be deleted in the destination (--delete). The source's data is never touched. The source is always named first and the destination is always named second. Trailing / on the source as means copy the contents of this directory to the destination. Without the trailing / on the source you get the directory name copied with all it's files in it. Below uses ssh as the remote-shell program as the transport. It also turns on the lowest grade encryption to speed up the transfer.

rsync -av --delete --rsh="ssh -c arcfour -l root" host1.domain.lan:/disk01/ /disk01/
Misc

Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is back grounded so all commands are executed one after another.

for HOST in $(< ListOfHosts); do ssh $HOST `df -h` & done

Set group id and sticky bits on a dir.

chmod g=swrx,+t DIR/

Use wget grab all of a certain type of file listed on a web page and put them in the current dir. The example will use jpeg's.

wget -r -l1 --no-parent -A "*.jpg" http://www.website.com/pictures/

ruby one liner

# double space a file
    $  ruby -pe 'puts' < file.txt
# triple space a file
    $  ruby -pe '2.times {puts}' < file.txt
# undo double-spacing (w/ and w/o whitespace in lines)
    $  ruby -lne 'BEGIN{$/="\n\n"}; puts $_' < file.txt
    $  ruby -ne 'BEGIN{$/="\n\n"}; puts $_.chomp' < file.txt
    $  ruby -e 'puts STDIN.readlines.to_s.gsub(/\n\n/, "\n")' < file.txt
# number each line of a file (left justified).
    $  ruby -ne 'printf("-%6s%s", $., $_)' < file.txt
# number each line of a file (right justified).
    $  ruby -ne 'printf("%6s%s", $., $_)' < file.txt
# number each line of a file, only print non-blank lines
    $  ruby -e 'while gets; end; puts $.' < file.txt
# count lines (emulates 'wc -l')
    $  ruby -ne 'END {puts $.}' < file.txt
    $  ruby -e 'while gets; end; puts $.' < file.txt

 # convert DOS newlines (CR/LF) to Unix format (LF)
  # - strip newline regardless; re-print with unix EOL
      $  ruby -ne 'BEGIN{$\="\n"}; print $_.chomp' < file.txt

  # convert Unix newlines (LF) to DOS format (CR/LF)
  # - strip newline regardless; re-print with dos EOL
      $  ruby -ne 'BEGIN{$\="\r\n"}; print $_.chomp' < file.txt

  # delete leading whitespace (spaces/tabs/etc) from beginning of each line
      $  ruby -pe 'gsub(/^\s+/, "")' < file.txt

  # delete trailing whitespace (spaces/tabs/etc) from end of each line
  # - strip newline regardless; replace with default platform record separator
      $  ruby -pe 'gsub(/\s+$/, $/)' < file.txt

  # delete BOTH leading and trailing whitespace from each line
      $  ruby -pe 'gsub(/^\s+/, "").gsub(/\s+$/, $/)' < file.txt

  # insert 5 blank spaces at the beginning of each line (ie. page offset)
      $  ruby -pe 'gsub(/%/, "   ")' < file.txt
      FAILS! $  ruby -pe 'gsub(/%/, 5.times{putc " "})' < file.txt

  # align all text flush right on a 79-column width
      $  ruby -ne 'printf("%79s", $_)' < file.txt

  # center all text in middle of 79-column width
      $  ruby -ne 'puts $_.chomp.center(79)' < file.txt
      $  ruby -lne 'puts $_.center(79)' < file.txt

  # substitute (find and replace) "foo" with "bar" on each line
      $  ruby -pe 'gsub(/foo/, "bar")' < file.txt

  # substitute "foo" with "bar" ONLY for lines which contain "baz"
      $  ruby -pe 'gsub(/foo/, "bar") if $_ =~ /baz/' < file.txt

  # substitute "foo" with "bar" EXCEPT for lines which contain "baz"
      $  ruby -pe 'gsub(/foo/, "bar") unless $_ =~ /baz/' < file.txt

  # substitute "foo" or "bar" or "baz".... with "baq"
      $  ruby -pe 'gsub(/(foo|bar|baz)/, "baq")' < file.txt

  # reverse order of lines (emulates 'tac') IMPROVE
      $  ruby -ne 'BEGIN{@arr=Array.new}; @arr.push $_; END{puts @arr.reverse}' < file.txt

  # reverse each character on the line (emulates 'rev')
      $  ruby -ne 'puts $_.chomp.reverse' < file.txt
      $  ruby -lne 'puts $_.reverse' < file.txt

  # join pairs of lines side-by-side (like 'paste')
      $  ruby -pe '$_ = $_.chomp + " " + gets if $. % 2' < file.txt

  # if a line ends with a backslash, append the next line to it
      $  ruby -pe 'while $_.match(/\\$/); $_ = $_.chomp.chop + gets; end' < file.txt
      $  ruby -e 'puts STDIN.readlines.to_s.gsub(/\\\n/, "")' < file.txt

  # if a line begins with an equal sign, append it to the previous line (Unix)
      $  ruby -e 'puts STDIN.readlines.to_s.gsub(/\n=/, "")' < file.txt

  # add a blank line every 5 lines (after lines 5, 10, 15, etc)
      $  ruby -pe 'puts if $. % 6 == 0' < file.txt

# print first 10 lines of a file (emulate 'head')
      $  ruby -pe 'exit if $. > 10' < file.txt

  # print first line of a file (emulate 'head -1')
      $  ruby -pe 'puts $_; exit' < file.txt

  # print the last 10 lines of a file (emulate 'tail'); NOTE reads entire file!
      $  ruby -e 'puts STDIN.readlines.reverse!.slice(0,10).reverse!' < file.txt

  # print the last 2 lines of a file (emulate 'tail -2'); NOTE reads entire file!
      $  ruby -e 'puts STDIN.readlines.reverse!.slice(0,2).reverse!' < file.txt

  # print the last line of a file (emulates 'tail -1')
      $  ruby -ne 'line = $_; END {puts line}' < file.txt

  # print only lines that match a regular expression (emulates 'grep')
      $  ruby -pe 'next unless $_ =~ /regexp/' < file.txt

  # print only lines that DO NOT match a regular expression (emulates 'grep')
      $  ruby -pe 'next if $_ =~ /regexp/' < file.txt

  # print the line immediately before a regexp, but not the regex matching line
      $  ruby -ne 'puts @prev if $_ =~ /regex/; @prev = $_;' < file.txt

  # print the line immediately after a regexp, but not the regex matching line
      $  ruby -ne 'puts $_ if @prev =~ /regex/; @prev = $_;' < file.txt

  # grep for foo AND bar AND baz (in any order)
      $  ruby -pe 'next unless $_ =~ /foo/ && $_ =~ /bar/ && $_ =~ /baz/' < file.txt

  # grep for foo AND bar AND baz (in order)
      $  ruby -pe 'next unless $_ =~ /foo.*bar.*baz/' < file.txt

  # grep for foo OR bar OR baz
      $  ruby -pe 'next unless $_ =~ /(foo|bar|baz)/' < file.txt

  # print paragraph if it contains regexp; blank lines separate paragraphs
      $  ruby -ne 'BEGIN{$/="\n\n"}; print $_ if $_ =~ /regexp/' < file.txt

  # print paragraph if it contains foo AND bar AND baz (in any order); blank lines separate paragraphs
      $  ruby -ne 'BEGIN{$/="\n\n"}; print $_ if $_ =~ /foo/ && $_ =~ /bar/ && $_ =~ /baz/' < file.txt

  # print paragraph if it contains foo AND bar AND baz (in order); blank lines separate paragraphs
      $  ruby -ne 'BEGIN{$/="\n\n"}; print $_ if $_ =~ /(foo.*bar.*baz)/' < file.txt

  # print paragraph if it contains foo OR bar OR baz; blank lines separate paragraphs
      $  ruby -ne 'BEGIN{$/="\n\n"}; print $_ if $_ =~ /(foo|bar|baz)/' < file.txt

  # print only lines of 65 characters or greater
      $  ruby -pe 'next unless $_.chomp.length >= 65' < file.txt
      $  ruby -lpe 'next unless $_.length >= 65' < file.txt

  # print only lines of 65 characters or less
      $  ruby -pe 'next unless $_.chomp.length < 65' < file.txt
      $  ruby -lpe 'next unless $_.length < 65' < file.txt

  # print section of file from regex to end of file
      $  ruby -pe '@found=true if $_ =~ /regex/; next unless @found' < file.txt

  # print section of file based on line numbers (eg. lines 2-7 inclusive)
      $  ruby -pe 'next unless $. >= 2 && $. <= 7' < file.txt

  # print line number 52
      $  ruby -pe 'next unless $. == 52' < file.txt

  # print every 3rd line starting at line 4
      $  ruby -pe 'next unless $. >= 4 && $. % 3 == 0' < file.txt

  # print section of file between two regular expressions, /foo/ and /bar/
      $  ruby -ne '@found=true if $_ =~ /foo/; next unless @found; puts $_; exit if $_ =~ /bar/' < file.txt

# print all of file except between two regular expressions, /foo/ and /bar/
      $  ruby -ne '@found = true if $_ =~ /foo/; puts $_ unless @found; @found = false if $_ =~ /bar/' < file.txt

  # print file and remove duplicate, consecutive lines from a file (emulates 'uniq')
      $  ruby -ne 'puts $_ unless $_ == @prev; @prev = $_' < file.txt

  # print file and remove duplicate, non-consecutive lines from a file (careful of memory!)
      $  ruby -e 'puts STDIN.readlines.sort.uniq!.to_s' < file.txt

  # print file except for first 10 lines
      $  ruby -pe 'next if $. <= 10' < file.txt

  # print file except for last line
      $  ruby -e 'lines=STDIN.readlines; puts lines[0,lines.size-1]' < file.txt

  # print file except for last 2 lines
      $  ruby -e 'lines=STDIN.readlines; puts lines[0,lines.size-2]' < file.txt

  # print file except for last 10 lines
      $  ruby -e 'lines=STDIN.readlines; puts lines[0,lines.size-10]' < file.txt

  # print file except for every 8th line
      $  ruby -pe 'next if $. % 8 == 0' < file.txt

  # print file except for blank lines
      $  ruby -pe 'next if $_ =~ /^\s*$/' < file.txt

  # delete all consecutive blank lines from a file except the first
      $  ruby -e 'BEGIN{$/=nil}; puts STDIN.readlines.to_s.gsub(/\n(\n)+/, "\n\n")' < file.txt

  # delete all consecutive blank lines from a file except for the first 2
      $  ruby -e 'BEGIN{$/=nil}; puts STDIN.readlines.to_s.gsub(/\n(\n)+/, "\n\n")' < file.txt

  # delete all leading blank lines at top of file
      $  ruby -pe '@lineFound = true if $_ !~ /^\s*$/; next if !@lineFound' < file.txt

perl one liner

collection of one-liners, a part is adapted from such a list by
Tom Christianson, one of the authors of "Programming Perl".

    # the always present hallo world program, adapted to a Math department
    perl -e 'print "Hello Mr Euler!\n"'

    # rename in each file name the string aaa by bbb
    ls | perl -ne 'chomp; next unless -e; $o = $_; s/aaa/bbb/; next if -e; rename $o, $_';

    # add first and last column in each line of file foo.txt and print it out
    perl -lane 'print $F[0] + $F[-1]' foo.txt

    # print lines 15 to 17 of file foo.txt
    perl -ne 'print if 15 .. 17' foo.txt

    # a second way to print lines 3 to 5 of file foo.txt
    perl -pe 'exit if 3<$. && $.<5' foo.txt

    # change all words "foo"s to "bar"s in every .c file and keep backups
    perl -p -i.bak -e 's/\bfoo\b/bar/g' *.c

    # the same but without backup. Remember the flags: "eat the pie"
    perl -p -i -e 's/foo/bar/g' *.c

    # changes ^M newline characters to newlines
    perl  -p -i -e 's/\012?\015/\n/g'  $1

    # the same but with all files with name filename
    perl -p -i -e  's/foo/bar' `find . -name "filename"`

    # substitution can also be applied to binary files like test.ppm
    perl -p -i -e 's/255/127/g' test.ppm

    # substitute "xyz.math" to "abc.math" in every .html file and keep backups
    perl -p -i.bak -e 's/xyz\.math/abc\.math/g' *.html

    # insert department name after each title and keep backup
    perl -p -i.bak -e 's#<title>#<title>Harvard .: #i' *.html

    # delete first 10 lines in foo.txt and keep backup foo.txt.bak
    perl -i.bak -ne 'print unless 1 .. 10' foo.txt

    # change isolated occurrence of aaa to bbb in each file *.c or *.h
    perl -p -i.bak -e 's{\baaa\b}{bbb}g' *.[ch]

    # reverses lines of file foo.txt and print it
    perl -e 'print reverse <>' foo.txt

    # find palindromes in a dictionary /usr/share/dict/words
    perl -lne 'print if $_ eq reverse' /usr/share/dict/words

    # reverses paragraphs in file foo.txt
    perl -00 -e 'print reverse <>' foo.txt

    # increments all numbers in foo.tx by 1
    perl -pe 's/(\d+)/ 1 + $1 /ge' foo.txt

    # reverses order of characters in each line of foo.txt
    perl -nle 'print scalar reverse $_' foo.txt

    # print lines beween START and END in foo.txt to STDOUT
    perl -ne 'print if /^START$/ .. /^END$/' foo.txt

    # delete lines beween START and END and backup original file
    perl -i.old -ne 'print unless /^START$/ .. /^END$/' foo.txt

    # look for duplicated words in a line
    perl -0777 -ne 'print "$.: doubled $_\n" while /\b(\w+)\b\s+\b\1\b/gi' foo.txt

    # start Perl debugger "stand-alone"
    perl -d -e 42

    # run a Perl program program.pl with warnings
    perl -w program.pl

    # run a Perl program program.pl with debugger
    perl -d program.pl

    # Run perl program program.pl, check syntax, print warnings
    perl -wc program.pl

HOT PERL ONLINERS

   Just enough perl to do most everything! Tom Christianson (spelling?)
   once posted a canonical list of one line perl programs to do many common
   command-line tasks.
   It included:
   # run contents of "my_file" as a program
   perl my_file

   # run debugger "stand-alone"
   perl -d -e 42

   # run program, but with warnings
   perl -w my_file

   # run program under debugger
   perl -d my_file

   # just check syntax, with warnings
   perl -wc my_file

   # useful at end of "find foo -print"
   perl -nle unlink

   # simplest one-liner program
   perl -e 'print "hello world!\n"'

   # add first and penultimate columns
   perl -lane 'print $F[0] + $F[-2]'

   # just lines 15 to 17
   perl -ne 'print if 15 .. 17' *.pod

   # in-place edit of *.c files changing all foo to bar
   perl -p -i.bak -e 's/\bfoo\b/bar/g' *.c

   # command-line that prints the first 50 lines (cheaply)
   perl -pe 'exit if $. > 50' f1 f2 f3 ...

   # delete first 10 lines
   perl -i.old -ne 'print unless 1 .. 10' foo.txt

   # change all the isolated oldvar occurrences to newvar
   perl -i.old -pe 's{\boldvar\b}{newvar}g' *.[chy]

   # command-line that reverses the whole file by lines
   perl -e 'print reverse <>' file1 file2 file3 ....

   # find palindromes
   perl -lne 'print if $_ eq reverse' /usr/dict/words

   # command-line that reverse all the bytes in a file
   perl -0777e 'print scalar reverse <>' f1 f2 f3 ...

   # command-line that reverses the whole file by paragraphs
   perl -00 -e 'print reverse <>' file1 file2 file3 ....

   # increment all numbers found in these files
   perl i.tiny -pe 's/(\d+)/ 1 + $1 /ge' file1 file2 ....

   # command-line that shows each line with its characters backwards
   perl -nle 'print scalar reverse $_' file1 file2 file3 ....

   # delete all but lines beween START and END
   perl -i.old -ne 'print unless /^START$/ .. /^END$/' foo.txt

   # binary edit (careful!)
   perl -i.bak -pe 's/Mozilla/Slopoke/g' /usr/local/bin/netscape

   # look for dup words
   perl -0777 -ne 'print "$.: doubled $_\n" while /\b(\w+)\b\s+\b\1\b/gi'

   # command-line that prints the last 50 lines (expensively)
   perl -e 'lines = <>; print @@lines[ $#lines .. $#lines-50' f1 f2 f3 ...

grep with regex

Searching Files on UNIX
On MPE you can display files using the :Print command, Fcopy, Magnet, or Qedit (with pattern match searches). On HP-UX you can display files using cat and even better using more (and string search using the slash "/" command), and Qedit (including searches of $Include files, and so on), but if you really want to search for patterns of text like a UNIX guru, grep is the tool for you.
Text version.

cat report.c {prints file on stdout, no pauses} 
cat -v -e -t dump {show non-printing characters too} 
cat >newfile {reads from stdin, writes to 'newfile'} 
cat rpt1.c inp.c test.s >newfile {combine 3 files into 1} 
more report.c {space for next page, q to quit} 
ps -a | more {page through the full output of ps} 
grep smug *.txt {search *.txt files for 'smug'}


MPE users will take a while to remember that more, like most UNIX tools, responds to a Return by printing the next line, not the next screen. Use the Spacebar to print the next page. Type "q" to quit. To scan ahead to find a string pattern, type "/" and enter a regular expression to match. For further help, type "h".

Searching Files Using UNIX grep
The grep program is a standard UNIX utility that searches through a set of files for an arbitrary text pattern, specified through a regular expression. Also check the man pages as well for egrep and fgrep. The MPE equivalents are MPEX and Magnet, both third-party products. By default, grep is case-sensitive (use -i to ignore case). By default, grep ignores the context of a string (use -w to match words only). By default, grep shows the lines that match (use -v to show those that don't match).
Text version.

% grep BOB tmpfile {search 'tmpfile' for 'BOB' anywhere in a line} 
% grep -i -w blkptr *  {search files in CWD for word blkptr, any case} 
% grep run[- ]time *.txt {find 'run time' or 'run-time' in all txt files} 
% who | grep root  {pipe who to grep, look for root}



Understanding Regular Expressions
Regular Expressions are a feature of UNIX. They describe a pattern to match, a sequence of characters, not words, within a line of text. Here is a quick summary of the special characters used in the grep tool and their meaning:
Text version.

^ (Caret) = match expression at the start of a line, as in ^A. 
$ (Question) = match expression at the end of a line, as in A$. 
\ (Back Slash) = turn off the special meaning of the next character, as in \^. 
[ ] (Brackets) = match any one of the enclosed characters, as in [aeiou]. Use Hyphen "-" for a range, as in [0-9]. 
[^ ] = match any one character except those enclosed in [ ], as in [^0-9]. 
. (Period) = match a single character of any value, except end of line. 
* (Asterisk) = match zero or more of the preceding character or expression. 
\{x,y\} = match x to y occurrences of the preceding. 
\{x\} = match exactly x occurrences of the preceding. 
\{x,\} = match x or more occurrences of the preceding.


As an MPE user, you may find regular expressions difficult to use at first. Please persevere, because they are used in many UNIX tools, from more to perl. Unfortunately, some tools use simple regular expressions and others use extended regular expressions and some extended features have been merged into simple tools, so that it looks as if every tool has its own syntax. Not only that, regular expressions use the same characters as shell wildcarding, but they are not used in exactly the same way. What do you expect of an operating system built by graduate students?

Since you usually type regular expressions within shell commands, it is good practice to enclose the regular expression in single quotes (') to stop the shell from expanding it before passing the argument to your search tool. Here are some examples using grep:

Text version.

grep smug files {search files for lines with 'smug'} 
grep '^smug' files {'smug' at the start of a line} 
grep 'smug$' files {'smug' at the end of a line} 
grep '^smug$' files {lines containing only 'smug'} 
grep '\^s' files {lines starting with '^s', "\" escapes the ^} 
grep '[Ss]mug' files {search for 'Smug' or 'smug'} 
grep 'B[oO][bB]' files {search for BOB, Bob, BOb or BoB } 
grep '^$' files {search for blank lines} 
grep '[0-9][0-9]' file {search for pairs of numeric digits}


Back Slash "\" is used to escape the next symbol, for example, turn off the special meaning that it has. To look for a Caret "^" at the start of a line, the expression is ^\^. Period "." matches any single character. So b.b will match "bob", "bib", "b-b", etc. Asterisk "*" does not mean the same thing in regular expressions as in wildcarding; it is a modifier that applies to the preceding single character, or expression such as [0-9]. An asterisk matches zero or more of what precedes it. Thus [A-Z]* matches any number of upper-case letters, including none, while [A-Z][A-Z]* matches one or more upper-case letters.

The vi editor uses \< \> to match characters at the beginning and/or end of a word boundary. A word boundary is either the edge of the line or any character except a letter, digit or underscore "_". To look for if, but skip stiff, the expression is \<if\>. For the same logic in grep, invoke it with the -w option. And remember that regular expressions are case-sensitive. If you don't care about the case, the expression to match "if" would be [Ii][Ff], where the characters in square brackets define a character set from which the pattern must match one character. Alternatively, you could also invoke grep with the -i option to ignore case.

Here are a few more examples of grep to show you what can be done:

Text version.

grep '^From: ' /usr/mail/$USER {list your mail} 
grep '[a-zA-Z]' {any line with at least one letter} 
grep '[^a-zA-Z0-9] {anything not a letter or number} 
grep '[0-9]\{3\}-[0-9]\{4\}' {999-9999, like phone numbers} 
grep '^.$' {lines with exactly one character} 
grep '"smug"' {'smug' within double quotes} 
grep '"*smug"*' {'smug', with or without quotes} 
grep '^\.' {any line that starts with a Period "."} 
grep '^\.[a-z][a-z]' {line start with "." and 2 lc letters}