Thursday, April 27, 2006

Basic Linux Commands

BASIC LINUX COMMANDS

alias    Create an alias
awk Find and Replace text, database sort/validate/index
break Exit from a loop
builtin Run a shell builtin

cal Display a calendar
case Conditionally perform a command
cat Display the contents of a file
cd Change Directory
cfdisk Partition table manipulator for Linux
chgrp Change group ownership
chmod Change access permissions
chown Change file owner and group
chroot Run a command with a different root directory
cksum Print CRC checksum and byte counts
clear Clear terminal screen
cmp Compare two files
comm Compare two sorted files line by line
command Run a command - ignoring shell functions
continue Resume the next iteration of a loop
cp Copy one or more files to another location
cron Daemon to execute scheduled commands
crontab Schedule a command to run at a later time
csplit Split a file into context-determined pieces
cut Divide a file into several parts

date Display or change the date & time
dc Desk Calculator
dd Data Dump - Convert and copy a file
declare Declare variables and give them attributes
df Display free disk space
diff Display the differences between two files
diff3 Show differences among three files
dir Briefly list directory contents
dircolors Colour setup for `ls'
dirname Convert a full pathname to just a path
dirs Display list of remembered directories
du Estimate file space usage

echo Display message on screen
ed A line-oriented text editor (edlin)
egrep Search file(s) for lines that match an extended expression
eject Eject CD-ROM
enable Enable and disable builtin shell commands
env Display, set, or remove environment variables
eval Evaluate several commands/arguments
exec Execute a command
exit Exit the shell
expand Convert tabs to spaces
export Set an environment variable
expr Evaluate expressions

factor Print prime factors
false Do nothing, unsuccessfully
fdformat Low-level format a floppy disk
fdisk Partition table manipulator for Linux
fgrep Search file(s) for lines that match a fixed string
find Search for files that meet a desired criteria
fmt Reformat paragraph text
fold Wrap text to fit a specified width.
for Expand words, and execute commands
format Format disks or tapes
free Display memory usage
fsck Filesystem consistency check and repair.
function Define Function Macros

gawk Find and Replace text within file(s)
getopts Parse positional parameters
grep Search file(s) for lines that match a given pattern
groups Print group names a user is in
gzip Compress or decompress named file(s)

hash Remember the full pathname of a name argument
head Output the first part of file(s)
history Command History
hostname Print or set system name

id Print user and group id's
if Conditionally perform a command
import Capture an X server screen and save the image to file
info Help info
install Copy files and set attributes

join Join lines on a common field

kill Stop a process from running

less Display output one screen at a time
let Perform arithmetic on shell variables
ln Make links between files
local Create variables
locate Find files
logname Print current login name
logout Exit a login shell
lpc Line printer control program
lpr Off line print
lprint Print a file
lprintd Abort a print job
lprintq List the print queue
lprm Remove jobs from the print queue
ls List information about file(s)

m4 Macro processor
man Help manual
mkdir Create new folder(s)
mkfifo Make FIFOs (named pipes)
mknod Make block or character special files
more Display output one screen at a time
mount Mount a file system
mtools Manipulate MS-DOS files
mv Move or rename files or directories

nice Set the priority of a command or job
nl Number lines and write files
nohup Run a command immune to hangups

passwd Modify a user password
paste Merge lines of files
pathchk Check file name portability
popd Restore the previous value of the current directory
pr Convert text files for printing
printcap Printer capability database
printenv Print environment variables
printf Format and print data
ps Process status
pushd Save and then change the current directory
pwd Print Working Directory

quota Display disk usage and limits
quotacheck Scan a file system for disk usage
quotactl Set disk quotas

ram ram disk device
rcp Copy files between two machines.
read read a line from standard input
readonly Mark variables/functions as readonly
remsync Synchronize remote files via email
return Exit a shell function
rm Remove files
rmdir Remove folder(s)
rpm Remote Package Manager
rsync Remote file copy (Synchronize file trees)

screen Terminal window manager
sdiff Merge two files interactively
sed Stream Editor
select Accept keyboard input
seq Print numeric sequences
set Manipulate shell variables and functions
shift Shift positional parameters
shopt Shell Options
shutdown Shutdown or restart linux
sleep Delay for a specified time
sort Sort text files
source Run commands from a file `.'
split Split a file into fixed-size pieces
su Substitute user identity
sum Print a checksum for a file
symlink Make a new name for a file
sync Synchronize data on disk with memory

tac Concatenate and write files in reverse
tail Output the last part of files
tar Tape ARchiver
tee Redirect output to multiple files
test Evaluate a conditional expression
time Measure Program Resource Use
times User and system times
touch Change file timestamps
top List processes running on the system
traceroute Trace Route to Host
trap Run a command when a signal is set(bourne)
tr Translate, squeeze, and/or delete characters
true Do nothing, successfully
tsort Topological sort
tty Print filename of terminal on stdin
type Describe a command

ulimit Limit user resources
umask Users file creation mask
umount Unmount a device
unalias Remove an alias
uname Print system information
unexpand Convert spaces to tabs
uniq Uniquify files
units Convert units from one scale to another
unset Remove variable or function names
unshar Unpack shell archive scripts
until Execute commands (until error)
useradd Create new user account
usermod Modify user account
users List users currently logged in
uuencode Encode a binary file
uudecode Decode a file created by uuencode

v Verbosely list directory contents (`ls -l -b')
vdir Verbosely list directory contents (`ls -l -b')

watch Execute/display a program periodically
wc Print byte, word, and line counts
whereis Report all known instances of a command
which Locate a program file in the user's path.
while Execute commands
who Print all usernames currently logged in
whoami Print the current user id and name (`id -un')

xargs Execute utility, passing constructed argument list(s)
yes Print a string until interrupted

Monday, April 24, 2006

Mirror Your Web Site With rsync

Mirror Your Web Site With rsync

Version 1.0
Author: Falko Timme
Last edited 04/20/2006

This tutorial shows how you can mirror your web site from your main web server to a backup server that can take over if the main server fails. We use the tool rsync for this, and we make it run through a cron job that checks every x minutes if there is something to update on the mirror. Thus your backup server should usually be up to date if it has to take over.

rsync updates only files that have changed, so you do not need to transfer 5 GB of data whenever you run rsync. It only mirrors new/changed files, and it can also delete files from the mirror that have been deleted on the main server. In addition to that it can preserve permissions and ownerships of mirrored files and directories; to preserve the ownerships, we need to run rsync as root which is what we do here. If permissions and/or ownerships change on the main server, rsync will also change them on the backup server.

In this tutorial we will tunnel rsync through SSH which is more secure; it also means you do not have to open another port in your firewall for rsync - it is enough if port 22 (SSH) is open. The problem is that SSH requires a password for logging in which is not good if you want to run rsync as a cron job. The need for a password requires human interaction which is not what we want.

But fortunately there is a solution: the use of public keys. We create a pair of keys (on our backup server mirror.example.com), one of which is saved in a file on the remote system (server1.example.com). Afterwards we will not be prompted for a password anymore when we run rsync. This also includes cron jobs which is exactly what we want.

As you might have guessed already from what I have written so far, the concept is that we initiate the mirroring of server1.example.com directly from mirror.example.com; server1.example.com does not have to do anything to get mirrored.

I will use the following setup here:

  • Main server: server1.example.com (server1) - IP address: 192.168.0.100
  • Mirror/backup server: mirror.example.com (mirror) - IP address: 192.168.0.175
  • The web site that is to be mirrored is in /var/www on server1.example.com.

rsync is for mirroring files and directories only; if you want to mirror your MySQL database, please take a look at these tutorials:

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

1 Install rsync

First we have to install rsync on both server1.example.com and mirror.example.com. For Debian systems, this looks like this:

server1/mirror:

(We do this as root!)

apt-get install rsync

On other Linux distributions you would use yum (Fedora/CentOS) or yast (SuSE) to install rsync.

2 Create An Unprivileged User On server1.example.com

Now we create an unprivileged user called someuser on server1.example.com that will be used by rsync on mirror.example.com to mirror the directory /var/www (of course, someuser must have read permissions on /var/www on server1.example.com).

server1:

(We do this as root!)

useradd -d /home/someuser -m -s /bin/bash someuser

This will create the user someuser with the home directory /home/someuser and the login shell /bin/bash (it is important that someuser has a valid login shell - something like /bin/false does not work!). Now give someuser a password:

passwd someuser

3 Test rsync

Next we test rsync on mirror.example.com. As root we do this:

mirror:

rsync -avz -e ssh someuser@server1.example.com:/var/www/ /var/www/

You should see something like this. Answer with yes:

The authenticity of host 'server1.example.com (192.168.0.100)' can't be established.
RSA key fingerprint is 32:e5:79:8e:5f:5a:25:a9:f1:0d:ef:be:5b:a6:a6:23.
Are you sure you want to continue connecting (yes/no)?

<-- yes

Then enter someuser's password, and you should see that server1.example.com's /var/www directory is mirrored to /var/www on mirror.example.com.

You can check that like this on both servers:

server1/mirror:

ls -la /var/www

You should see that all files and directories have been mirrored to mirror.example.com, and the files and directories should have the same permissions/ownerships as on server1.example.com.

4 Create The Keys On mirror.example.com

Now we create the private/public key pair on mirror.example.com:

mirror:

(We do this as root!)

mkdir /root/rsync
ssh-keygen -t dsa -b 2048 -f /root/rsync/mirror-rsync-key

You will see something like this:

Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase): [press enter here]
Enter same passphrase again: [press enter here]
Your identification has been saved in /root/cron/mirror-rsync-key.
Your public key has been saved in /root/cron/mirror-rsync-key.pub.
The key fingerprint is:
68:95:35:44:91:f1:45:a4:af:3f:69:2a:ea:c5:4e:d7 root@mirror

It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit enter!

Next, we copy our public key to server1.example.com:

mirror:

(Still, we do this as root.)

scp /root/rsync/mirror-rsync-key.pub someuser@server1.example.com:/home/someuser/

The public key mirror-rsync-key.pub should now be available in /home/someuser on server1.example.com.

Mirror Your Web Site With rsync

Mirror Your Web Site With rsync

Version 1.0
Author: Falko Timme
Last edited 04/20/2006

This tutorial shows how you can mirror your web site from your main web server to a backup server that can take over if the main server fails. We use the tool rsync for this, and we make it run through a cron job that checks every x minutes if there is something to update on the mirror. Thus your backup server should usually be up to date if it has to take over.

rsync updates only files that have changed, so you do not need to transfer 5 GB of data whenever you run rsync. It only mirrors new/changed files, and it can also delete files from the mirror that have been deleted on the main server. In addition to that it can preserve permissions and ownerships of mirrored files and directories; to preserve the ownerships, we need to run rsync as root which is what we do here. If permissions and/or ownerships change on the main server, rsync will also change them on the backup server.

In this tutorial we will tunnel rsync through SSH which is more secure; it also means you do not have to open another port in your firewall for rsync - it is enough if port 22 (SSH) is open. The problem is that SSH requires a password for logging in which is not good if you want to run rsync as a cron job. The need for a password requires human interaction which is not what we want.

But fortunately there is a solution: the use of public keys. We create a pair of keys (on our backup server mirror.example.com), one of which is saved in a file on the remote system (server1.example.com). Afterwards we will not be prompted for a password anymore when we run rsync. This also includes cron jobs which is exactly what we want.

As you might have guessed already from what I have written so far, the concept is that we initiate the mirroring of server1.example.com directly from mirror.example.com; server1.example.com does not have to do anything to get mirrored.

I will use the following setup here:

  • Main server: server1.example.com (server1) - IP address: 192.168.0.100
  • Mirror/backup server: mirror.example.com (mirror) - IP address: 192.168.0.175
  • The web site that is to be mirrored is in /var/www on server1.example.com.

rsync is for mirroring files and directories only; if you want to mirror your MySQL database, please take a look at these tutorials:

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

1 Install rsync

First we have to install rsync on both server1.example.com and mirror.example.com. For Debian systems, this looks like this:

server1/mirror:

(We do this as root!)

apt-get install rsync

On other Linux distributions you would use yum (Fedora/CentOS) or yast (SuSE) to install rsync.

2 Create An Unprivileged User On server1.example.com

Now we create an unprivileged user called someuser on server1.example.com that will be used by rsync on mirror.example.com to mirror the directory /var/www (of course, someuser must have read permissions on /var/www on server1.example.com).

server1:

(We do this as root!)

useradd -d /home/someuser -m -s /bin/bash someuser

This will create the user someuser with the home directory /home/someuser and the login shell /bin/bash (it is important that someuser has a valid login shell - something like /bin/false does not work!). Now give someuser a password:

passwd someuser

3 Test rsync

Next we test rsync on mirror.example.com. As root we do this:

mirror:

rsync -avz -e ssh someuser@server1.example.com:/var/www/ /var/www/

You should see something like this. Answer with yes:

The authenticity of host 'server1.example.com (192.168.0.100)' can't be established.
RSA key fingerprint is 32:e5:79:8e:5f:5a:25:a9:f1:0d:ef:be:5b:a6:a6:23.
Are you sure you want to continue connecting (yes/no)?

<-- yes

Then enter someuser's password, and you should see that server1.example.com's /var/www directory is mirrored to /var/www on mirror.example.com.

You can check that like this on both servers:

server1/mirror:

ls -la /var/www

You should see that all files and directories have been mirrored to mirror.example.com, and the files and directories should have the same permissions/ownerships as on server1.example.com.

4 Create The Keys On mirror.example.com

Now we create the private/public key pair on mirror.example.com:

mirror:

(We do this as root!)

mkdir /root/rsync
ssh-keygen -t dsa -b 2048 -f /root/rsync/mirror-rsync-key

You will see something like this:

Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase): [press enter here]
Enter same passphrase again: [press enter here]
Your identification has been saved in /root/cron/mirror-rsync-key.
Your public key has been saved in /root/cron/mirror-rsync-key.pub.
The key fingerprint is:
68:95:35:44:91:f1:45:a4:af:3f:69:2a:ea:c5:4e:d7 root@mirror

It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit enter!

Next, we copy our public key to server1.example.com:

mirror:

(Still, we do this as root.)

scp /root/rsync/mirror-rsync-key.pub someuser@server1.example.com:/home/someuser/

The public key mirror-rsync-key.pub should now be available in /home/someuser on server1.example.com.

Sunday, April 23, 2006

How To Set Up Database Replication In MySQL

How To Set Up Database Replication In MySQL

Version 1.1
Author: Falko Timme
Last edited: 01/14/2006

This tutorial describes how to set up database replication in MySQL. MySQL replication allows you to have an exact copy of a database from a master server on another server (slave), and all updates to the database on the master server are immediately replicated to the database on the slave server so that both databases are in sync. This is not a backup policy because an accidentally issued DELETE command will also be carried out on the slave; but replication can help protect against hardware failures though.

In this tutorial I will show how to replicate the database exampledb from the master with the IP address 192.168.0.100 to a slave. Both systems (master and slave) are running Debian Sarge; however, the configuration should apply to almost all distributions with little or no modification.

Both systems have MySQL installed, and the database exampledb with tables and data is already existing on the master, but not on the slave.

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

1 Configure The Master

First we have to edit /etc/mysql/my.cnf. We have to enable networking for MySQL, and MySQL should listen on all IP addresses, therefore we comment out these lines (if existant):

#skip-networking
#bind-address = 127.0.0.1

Furthermore we have to tell MySQL for which database it should write logs (these logs are used by the slave to see what has changed on the master), which log file it should use, and we have to specify that this MySQL server is the master. We want to replicate the database exampledb, so we put the following lines into /etc/mysql/my.cnf:

log-bin = /var/log/mysql/mysql-bin.log
binlog-do-db=exampledb
server-id=1

Then we restart MySQL:

/etc/init.d/mysql restart

Then we log into the MySQL database as root and create a user with replication privileges:

mysql -u root -p
Enter password:

Now we are on the MySQL shell.

For mysql version 3.23 and below

GRANT LINE ON *.* TO 'slave_user'@'Host IP' IDENTIFIED BY ''; (Replace with a real password! and Host IP with ip adress or host.domainname will do)

For Mysql version 4 and up

GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY ''; (Replace with a real password!)
FLUSH PRIVILEGES;

* you could also use grant all privileges although not recommended

Next (still on the MySQL shell) do this:

USE exampledb;
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

The last command will show something like this:

+---------------+----------+--------------+------------------+
| File | Position | Binlog_do_db | Binlog_ignore_db |
+---------------+----------+--------------+------------------+
| mysql-bin.006 | 183 | exampledb | |
+---------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Write down this information, we will need it later on the slave!

Then leave the MySQL shell:

quit;


There are two possibilities to get the existing tables and data from exampledb from the master to the slave. The first one is to make a database dump, the second one is to use the LOAD DATA FROM MASTER; command on the slave. The latter has the disadvantage the the database on the master will be locked during this operation, so if you have a large database on a high-traffic production system, this is not what you want, and I recommend to follow the first method in this case. However, the latter method is very fast, so I will describe both here.

If you want to follow the first method, then do this:

mysqldump -u root -p --opt exampledb > exampledb.sql (Replace with the real password for the MySQL user root! Important: There is no space between -p and !)

This will create an SQL dump of exampledb in the file exampledb.sql. Transfer this file to your slave server!

If you want to go the LOAD DATA FROM MASTER; way then there is nothing you must do right now.


Finally we have to unlock the tables in exampledb:

mysql -u root -p
Enter password:
UNLOCK TABLES;
quit;

Now the configuration on the master is finished. On to the slave...

2 Configure The Slave

On the slave we first have to create the database exampledb:

mysql -u root -p
Enter password:
CREATE DATABASE exampledb;
quit;


If you have made an SQL dump of exampledb on the master and have transferred it to the slave, then it is time now to import the SQL dump into our newly created exampledb on the slave:

mysql -u root -p exampledb < /path/to/exampledb.sql (Replace with the real password for the MySQL user root! Important: There is no space between -p and !)

If you want to go the LOAD DATA FROM MASTER; way then there is nothing you must do right now.


Now we have to tell MySQL on the slave that it is the slave, that the master is 192.168.0.100, and that the master database to watch is exampledb. Therefore we add the following lines to /etc/mysql/my.cnf:

server-id=2
master-host=192.168.0.100
master-user=slave_user
master-password=secret
master-connect-retry=60
replicate-do-db=exampledb

Then we restart MySQL:

/etc/init.d/mysql restart


If you have not imported the master exampledb with the help of an SQL dump, but want to go the LOAD DATA FROM MASTER; way, then it is time for you now to get the data from the master exampledb:

mysql -u root -p
Enter password:
LOAD DATA FROM MASTER;
quit;

If you have phpMyAdmin installed on the slave you can now check if all tables/data from the master exampledb is also available on the slave exampledb.


Finally, we must do this:

mysql -u root -p
Enter password:
SLAVE STOP;

In the next command (still on the MySQL shell) you have to replace the values appropriately:

CHANGE MASTER TO MASTER_HOST='192.168.0.100', MASTER_USER='slave_user', MASTER_PASSWORD='', MASTER_LOG_FILE='mysql-bin.006', MASTER_LOG_POS=183;

  • MASTER_HOST is the IP address or hostname of the master (in this example it is 192.168.0.100).
  • MASTER_USER is the user we granted replication privileges on the master.
  • MASTER_PASSWORD is the password of MASTER_USER on the master.
  • MASTER_LOG_FILE is the file MySQL gave back when you ran SHOW MASTER STATUS; on the master.
  • MASTER_LOG_POS is the position MySQL gave back when you ran SHOW MASTER STATUS; on the master.

Now all that is left to do is start the slave. Still on the MySQL shell we run

START SLAVE;
quit;

That's it! Now whenever exampledb is updated on the master, all changes will be replicated to exampledb on the slave. Test it!

How do I scan my Linux system for rootkits, worms, trojans, etc.?

Either with ckrootkit or with rkhunter.

chkrootkit:

Either install the package that comes with your distribution (on Debian you would run

apt-get install chkrootkit

), or download the sources from www.chkrootkit.org and install manually:

wget --passive-ftp ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz

tar xvfz chkrootkit.tar.gz

cd chkrootkit-/

make sense

Afterwards, you can move the chkrootkit directory somewhere else, e.g. /usr/local/chkrootkit:

cd ..

mv chkrootkit-/ /usr/local/chkrootkit

Now you can run chkrootkit manually:

cd /usr/local/chkrootkit

./chkrootkit

(if you installed a chkrootkit package coming with your distribution, your chkrootkit might be somewhere else).

You can even run chkrootkit by a cron job and get the results emailed to you:

Run

crontab -e

to create a cron job like this:

0 3 * * * (cd /usr/local/chkrootkit-; ./chkrootkit 2>&1 | mail -s "chkrootkit output my server" you@yourdomain.com)

That would run chkrootkit every night a 3.00h.

rkhunter:

Download the latest rkhunter sources from www.rootkit.nl:

wget http://downloads.rootkit.nl/rkhunter-1.2.7.tar.gz

tar xvfz rkhunter-1.2.7.tar.gz

cd rkhunter/

./installer.sh

This will install rkhunter to the directory /usr/local/rkhunter. Now run

rkhunter --update

to download the latest chkrootkit/trojan/worm signatures (you should do this regularly).

Now you can scan your system for malware by running

rkhunter -c

Tuesday, April 18, 2006

TRACKING OF LOGINS AND LOGOUTS on LINUX / UNIX Distros

TRACKING OF LOGINS AND LOGOUTS


In the .login file add the commands:
------------------------------
------

echo login time `date` >> .daylogs/masterlog

grep -i "sun" .daylogs/masterlog > .daylogs/sunday.log
grep -i "mon" .daylogs/masterlog > .daylogs/monday.log
grep -i "tue" .daylogs/masterlog > .daylogs/tuesday.log
grep -i "wen" .daylogs/masterlog > .daylogs/wensday.log
grep -i "thu" .daylogs/masterlog > .daylogs/thursday.log
grep -i "fri" .daylogs/masterlog > .daylogs/friday.log
grep -i "sat" .daylogs/masterlog > .daylogs/saturday.log


In the .logout file add this line
-----------------------------------

echo logout time `date`>> .daylogs/masterlog

This script assumes you have a hidden
directory called .daylogs this helps keep it
out of sight and away from prying eyes and
if you keep root ownership of the directory
change the mode to:

chmod 744 .daylogs

This will not allow anyone to get in to the
directory to look around.

Saturday, April 15, 2006

CREATING TRASH

CREATING TRASH

Sometimes unwittingly we may delete
some important files and realise it
later. To avoid such a situation,
you can use this following idea.

Create a small bash script containing
the line

mv $1 ~/trash/

Save this file in your home say
".srm" (safe rm) and in your
".bashrc" enter this line:

alias rm='~/.srm'

Now whenever you delete any files
it will go to "trash" directory
instead of deletion. You will
have to create a "trash" directory
in your home directory.

Thursday, April 13, 2006

BASH SHELL OPTION

BASH SHELL OPTION

If you are using bash shell. There is
a way to cd a particular directory
even if you spelled incorrectly on the
command line. Set the shell option to:

shopt -s cdspell

eg:-
Suppose you want to cd to "cd /tmp"
and you have miss typed to "cd /pmp"
still it will cd to "cd /tmp".

This setting will be very usefull if
you have a long named directory.

Wednesday, April 12, 2006

NFS BETWEEN SOLARIS & LINUX

NFS BETWEEN SOLARIS & LINUX


If you receive error messages such
as "unknown version" when attempting to
mount a Linux based NFS server from
Solaris, you probably have an
incompatibility between the NFS versions
running on both of them. Linux uses
version 2, while Solaris uses version 3.
In order to get the machines to
communicate, you have to use the vers
option on the Solaris machine as follows:

mount -o vers=2 nfsserver:/remotedir /localdir

Tuesday, April 11, 2006

MAINTAINING LOG AND TMP FILE

MAINTAINING LOG AND TMP FILE

If you need to maintain an
application that generate
a lot of logfiles or tmp
files with running numbers
as part of the filename and
uses a common extension.
These two command together
will help to maintain it
for a window of time.

It compress those files that
are more than 24hrs and
have it remove after 120hrs.
You need to put it in
daily cron.

find $LOGDIR -name '*.ext' -mtime +0 -exec compress {} \;
find $LOGDIR -name '*.Z' -mtime +5 -exec rm -f {} \;

You can change the time to
suit your needs and use
wherever compressing utility
you have to save space.
If you need to maintain
directories created by
application here are help;

find $LOGDIR -type d -mtime +0 -exec compress -r {} \;
find $LOGDIR -type d -mtime +5 -exec rm -f {}
The compression is to save
space while waiting to be
deleted. Application
developers may need to read
these files/directories so
keep those files/directories
for a certain amount of time
before deleting.

EXTRACT CORRUPTED TAR FILE

EXTRACT CORRUPTED TAR FILE

In many case if there is a corrupted tar file,
the following command can be used in an attempt
to extract the file:

% cat [tar-filename] | tar -xvf -

NOTE: Where "-" is the STDOUT

Sunday, April 02, 2006

EFFICIENT COMMANDS

EFFICIENT COMMANDS

I cringe anytime I see someone code
inefficiently. Here are three of the
most common mistakes, followed by a
better way to do the same thing.

Bad: cat somefile | grep something
Better: grep something somefile
Why: You're running one program (grep) instead of two (cat and grep).

Bad: ps -ef | grep something | grep -v grep
Better: ps -ef | grep [s]omething
Why: You're running two commands (grep) instead of three (ps
and two greps).

Bad: cat /dev/null > somefile
Better: > somefile
Why: You're running a command (cat) with I/O redirection,
instead of just redirection.

Although the bad way will have the
same result, the good way is far
faster. This may seem trivial, but
the benefits will really show when
dealing with large files or loops.

Saturday, April 01, 2006

Removing Blank Line using SED

I SED BLANK

Using sed, you can remove blank lines, and
lines that contain only whitespace, from a
file using the following:

sed -e '/^[ ]*$/d' InputFile >OutputFile

Within the single quotes ('), the forward
slashes (/) delimit the regular expression
that will be interpreted by sed. The "d"
before the closing single quote, tells sed
to delete any lines that match the regular
expression.

Within the regular expression, the caret
(^) matches the beginning of a line.
The []* matches zero to many occurrences of
the character list between the open bracket
([) and the close bracket (]) (in the above
regular expression, you must insert a space
and a tab between the brackets). The dollar
sign ($) matches the end of a line.

These three constructs together match any
blank line or any line that contains only
spaces and tabs (in any combination).

Since the standard operation of sed is to
echo lines to stdout, all lines except blank
lines (or lines that only contain whitespace)
will be sent to OutputFile.