Archive for the ‘Tuning Linux’ Category

h1

Automated Process monitoring during high server load

August 26, 2010

root@myServer [/root]# cat load-process-monitor.sh
#!/bin/bash

# Define Variables
DT=`date +”%A %b %e %r”`
HOSTNAME=`hostname`

# Create dir to store data
mkdir -p /opt/loadcheck/

# Retrieve the load average of the past 1 minute
LAVG=`uptime | awk {‘print $10}’ | cut -d. -f1`
LCURRENT=`uptime | awk {‘print $10,$11,$12}’`

# Define Threshold. This value will be compared with the current load average. Set the value as per your wish.
LIMIT=-1

# Compare the current load average with Threshold and email the server administrator if threshold is greater.

if [ $LAVG -gt $LIMIT ]
then

#Save the current running processes in a file
/bin/ps -auxf >> /opt/ps_output

echo “Current Time :: $DT. >> /tmp/loadmon.txt
echo “Current Load Average :: $LCURRENT. >> /tmp/loadmon.txt
echo “current processes list attached with the email 1 instance. >> /tmp/loadmon.txt
echo “Also check loadps.txt :: loadtop.txt :: netstat_all.txt :: netstat_port80.txt inside /opt/loadcheck/ on the server” >> /tmp/loadmon.txt
# Send email to support
/usr/bin/mutt -s “Server Load ALERT!!! High 1 minute load average on ‘$HOSTNAME'” -a /opt/ps_output support@somedomain.com > /opt/ps_output

echo “Current Time :: $DT” >> /tmp/loadmon.txt
echo “Current Load Average :: $LCURRENT” >> /tmp/loadmon.txt
echo “current processes list attached with the email 1 instance” >> /tmp/loadmon.txt
echo “Also check loadps.txt :: loadtop.txt :: netstat_all.txt :: netstat_port80.txt inside /opt/loadcheck/ on the server” >> /tmp/loadmon.txt
# Send email to support
/usr/bin/mutt -s ” Server Load ALERT ::: High 1 minute load average on ‘$HOSTNAME’ ” -a /opt/ps_output support@integrityhost.com > /opt/loadcheck/loadps.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt
/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt
/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt
/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt
/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt
/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt
/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt
/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt
/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt
/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt
/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt
/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt
/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt
/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt
echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt
/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt
echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt
/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt
echo ” ######################################################################################################################### ” >> /opt/loadcheck/netstat_port80.txt

fi

# Remove residue logs
/bin/rm -f /tmp/loadmon.txt
/bin/rm -f /opt/ps_output

root@myServer [/root]#

Add a cron to run load-process-monitor.sh ( * * * * * /bin/sh /path-to/load-process-monitor.sh ) every min and when server load goes beyond 4, it will send you email and log some important details, which can help to some extent to find some pointers to load issue from process and netstat listings.

Advertisements
h1

Virtuozzo Upgrade : Upgrading from Virtuozzo3 to Virtuozzo4

August 18, 2008

Upgrading from Virtuozzo 3 to Virtuozzo 4
================================

First of all refer to :  http://www.parallels.com/en/products/virtuozzo/easy-upgrade/ for detailed explanation. When I decided to upgrade, there was not much available except the above docs. I used the inbuilt vzup2date for the upgrade purpose.

First of all apply all updates on your current Virtuozzo 3 using vzup2date , by selecting udate for virtuozzo 3.0.x only. Once updates are applied, it also does a kernel upgrade. It will finally ask if you want to reboot. Just select finish, and dont reboot. It is better to reboot manually, so that you can swith back to old kernel, in case the new kernel panics or hang up during boot.

Step 1
===================================================

[root@vpsMainNode virtuozzo]# vzup2date

Apply all updates on your current Virtuozzo 3.
Select option reboot manually and dont hit the reboot option, just select finish.

Next,

Edit /etc/grub.conf  and change the option

default=0 to default=1 , so that the current kernel is selected in grub , instead of the new kernel , that was just installed.

Next we configure grub to boot the new kernel only for the next reboot. If the reboot does not go well and the system does not boot properly, do a hard reset ( ask your DC, if

reqd. ) and you will successfully boot into the last working kernel.

[root@vpsMainNode virtuozzo]# grub shell
Probing devices to guess BIOS drives. This may take a long time.

GNU GRUB  version 0.95  (640K lower / 3072K upper memory)

[ Minimal BASH-like line editing is supported.  For the first word, TAB
lists possible command completions.  Anywhere else TAB lists the possible
completions of a device/filename.]

grub> savedefault –default=0 –once
grub> quit

[root@vpsMainNode virtuozzo]#

Reboot the server. If all goes fine, and the server comes up with the new kernel, edit /etc/grub.conf  and change the option

default=1 to default=0

Next,

Run vzup2date again and select updates for Virtuozzo 3 , and go next , next to finish install ( select option manual reboot )
[root@vpsMainNode virtuozzo]# vzup2date

If all Virtuozzo 3 are already applied, you will see no more update available message.

Now that Virtuozzo 3 has all its latest stuff in its version, we will update Virtuozzo 3 to Virtuozzo 4.

Run vzup2date again,
[root@vpsMainNode virtuozzo]# vzup2date

This time, select Virtuozzo 4 for upgrades, do the usual next , next to apply updates. ( select option manual reboot )

This process will take some time depending on speed. Usually 20 to 60 minutes.

Again this will install a newer kernel, so

Next we configure grub to boot the new kernel only for the next reboot. If the reboot does not go well and the system does not boot properly, do a hard reset ( ask your DC, if

reqd. ) and you will successfully boot into the last working kernel.

[root@vpsMainNode virtuozzo]# grub shell
Probing devices to guess BIOS drives. This may take a long time.

GNU GRUB  version 0.95  (640K lower / 3072K upper memory)

[ Minimal BASH-like line editing is supported.  For the first word, TAB
lists possible command completions.  Anywhere else TAB lists the possible
completions of a device/filename.]

grub> savedefault –default=0 –once
grub> quit

[root@vpsMainNode virtuozzo]#

Reboot the server. If all goes fine, and the server comes up with the new kernel, edit /etc/grub.conf  and change the option

default=1 to default=0

Voila !   you have upgraded your server from Virtuozzo 3 to Virtuozzo 4. You will see that all nodes will work properly, start the nodes, if any active node is down in the usual

way.

[root@vpsMainNode virtuozzo]# vzctl start VE_ID

Note VE_ID is now also called as CT_ID ( container ID )

You will find that the Service VE ( now called as Service Container )  will fail to start. If not, then fine. You have nothing left to do , except running a vzup2date again and

apply any newer updates for one more time. If any new kernel is installed , then modify grub.conf as already discussed, while rebooting.

In case Service VE ( now called as Service Container )  fails to start or does not start, you will be unable to access your virtuozzo control panel.

Note , I destroyed old Service VE and recreated it, if for any reason, you want backup of old service , you might chose to make a backup of it.

Here is what I did to install the service CT
###############################################
[root@vpsMainNode virtuozzo]# vzctl destroy 1
[root@vpsMainNode virtuozzo]# cd /root/
[root@vpsMainNode virtuozzo]# wget http://download.parallels.com/virtuozzo/virtuozzo4.0/linux/iso/lin-i386.iso
[root@vpsMainNode virtuozzo]# mkdir /mnt/iso
[root@vpsMainNode virtuozzo]# mount -o loop lin-i386.iso /mnt/iso
[root@vpsMainNode virtuozzo]# vzsveinstall -D /mnt/iso -s xx.yy.zz.IP

[root@vpsMainNode virtuozzo]# vzlist -a | grep CT
CTID      NPROC STATUS    IP_ADDR         HOSTNAME
1         77 running   xx.yy.zz.IP  ServiceCT
[root@vpsMainNode virtuozzo]#

[root@vpsMainNode virtuozzo]# lsof  -i :4643
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE NODE NAME
vzaproxy  531   root    4u  IPv4 10286471       TCP *:4643 (LISTEN)
vzcp      591   root    4u  IPv4 10286403       TCP *:4643 (LISTEN)
vzcp      595 apache    4u  IPv4 10286403       TCP *:4643 (LISTEN)
vzcp      596 apache    4u  IPv4 10286403       TCP *:4643 (LISTEN)
vzcp     3004 apache    4u  IPv4 10286403       TCP *:4643 (LISTEN)
[root@vpsMainNode virtuozzo]#

[root@vpsMainNode virtuozzo]# vzctl  restart 1
Restart Container
Stopping Container …
Container was stopped
[  OK  ] down vzagent: [  OK  ]
[  OK  ] vzagent: [  OK  ]
Container is unmounted
Starting Container …
Starting vzagent: [  OK  ]
Starting vzagent: [  OK  ]
Container is mounted
Setting devperms 20006 dev 0x7d00
Adding IP address(es): xx.yy.zz.IP
Hostname for Container set: ServiceCT
File resolv.conf was modified
Container start in progress…

[root@vpsMainNode virtuozzo]# vzlist  -a | grep CT
CTID      NPROC STATUS    IP_ADDR         HOSTNAME
1         77 running   xx.yy.zz.IP  ServiceCT
[root@vpsMainNode virtuozzo]#

[root@vpsMainNode log]# cd /root/
[root@vpsMainNode ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             9.9G  4.1G  5.3G  44% /
/dev/sda1             494M   81M  388M  18% /boot
none                  4.0G     0  4.0G   0% /dev/shm
/dev/sda3             448G  267G  158G  63% /vz
/dev/sdb1             459G  123G  313G  29% /backup
/root/lin-i386.iso    582M  582M     0 100% /mnt/iso

[root@vpsMainNode ~]# mkdir /root/virt4_source
[root@vpsMainNode ~]# cp -r /mnt/iso/* /root/virt4_source/
[root@vpsMainNode ~]# umount /mnt/iso

[root@vpsMainNode ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             9.9G  4.7G  4.8G  50% /
/dev/sda1             494M   81M  388M  18% /boot
none                  4.0G     0  4.0G   0% /dev/shm
/dev/sda3             448G  267G  158G  63% /vz
/dev/sdb1             459G  123G  313G  29% /backup
[root@vpsMainNode ~]#

Congrats !!! , now that you have got the Service Container Up and Running, time to explore/enjoy the new looks in Virtuozzo 4 control panel.

h1

iptables string match to drop malicious urls

August 5, 2008

iptables string match to drop malicious urls

==================================

Usually modsecurity rules can help filter many malicious url attack patterns combined with apache on apache port (http|https).  But what if a malicious attack using a vulnerable url pattern, that exposes or tries to break into your system is coming onto another port?

This is where iptables string match comes in handy.

/usr/local/sbin/iptables -I INPUT -p tcp -s 0.0.0.0/0 -m stringstring “download?file=%2e%2e” –algo bm -j DROP

[root@server ~]# iptables -L -v | grep STR
73 49908 DROP       tcp  —  any    any     anywhere             anywhere            STRING match “download?file=%2e%2e” ALGO name bm TO 65535

[root@server ~]#

The above iptable rule will block any url  that has the string “download?file=%2e%2e” on any port on your server.

Note: your iptables binary path may be /sbin/iptables

Say http://yourserverIP:9132/blah/download?file=%2e%2e

h1

tune2fs practical

July 1, 2008

How do I make the system perform a filesystem check after a certain number of mounts?

If the system is fairly reliable, it is safe to increase the maximum mount count. However, you should strongly consider the implications. Bad disk drives, cables, memory and kernel bugs can all corrupt a filesystem without marking the filesystem dirty or have it error out. A filesystem error detected by the kernel will force an fsck on the next reboot, but by that time, it may already be too late to prevent data loss.

With these in mind, here are the instructions to performing filesystem checks after a certain number of mounts. To adjust the maximum mount count, the filesystem should be unmounted before using tune2fs .

The example below will check the filesystem or partition after 50 mounts or 2 months, whichever comes first. You need to type in this command in a root shell:

tune2fs -c 50 -i 2m /dev/hda1

root@server61 [/etc/pam.d]# tune2fs -c 300 -i 11m /dev/sda8
tune2fs 1.35 (28-Feb-2004)
Setting maximal mount count to 300
Setting interval between check 28512000 seconds

root@server61 [/etc/pam.d]# tune2fs -c 300 -i 11m /dev/sda2
tune2fs 1.35 (28-Feb-2004)
Setting maximal mount count to 300
Setting interval between check 28512000 seconds

root@server61 [/etc/pam.d]# fdisk  -l

Disk /dev/sda: 250.8 GB, 250808893440 bytes
255 heads, 63 sectors/track, 30492 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          65      522081   83  Linux
/dev/sda2              66        2023    15727635   83  Linux
/dev/sda3            2024        3720    13631152+  83  Linux
/dev/sda4            3721       30492   215046090    5  Extended
/dev/sda5            3721        4112     3148708+  83  Linux
/dev/sda6            4113        4373     2096451   83  Linux
/dev/sda7            4374        4634     2096451   82  Linux swap
/dev/sda8            4635       30492   207704353+  83  Linux

Disk /dev/sdb: 250.8 GB, 250808893440 bytes
255 heads, 63 sectors/track, 30492 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       30492   244926958+  83  Linux
root@server61 [/etc/pam.d]# mkdir /backup
root@server61 [/etc/pam.d]# mount /dev/sdb1 /backup

tune2fs disable disk check
##############################

one of my disk using etx3,and as usual e2fsck will check the disk every 20 mounts or 180 days or whichever come first. This is annoying especially if you have very big volume in your system. We can disable this feature by using this command.
tune2fs -i 0 /dev/hdxx

root@server61 [/etc/pam.d]# tune2fs -i 0 /dev/sdb1
tune2fs 1.35 (28-Feb-2004)
Setting interval between check 0 seconds
root@server61 [/etc/pam.d]#

h1

Prevent non-root users from logging

May 9, 2007

Prevent non-root users from logging

Imagine that for some reason (i.e. maintenance tasks) you want to prevent non-root users from logging into the system. The next tip is a very simple way to achieve this goal.

If a file called /etc/nologin exists login will disable the begin of a session in this system. If you put some text into the file, users will be shown this text and their login attempts will be refused.

vi /etc/nologin

Server under maintenance. No access allowed at this moment.

h1

Tips and tricks

August 11, 2006

———————————————————————–

Q:- Find out top 10 directories eating up your disk space:
A:- du -csh * –max-depth=0 | sort -rn | head -10
———————————————————————–

Q:- Find Harddisk Capacity on the box.
A:- fdisk -l | grep -iE ‘mb|gb|tb’
———————————————————————–

Q:- Find out performance of your hard disk with following command:
A:- hdparm -t -T /dev/hda
———————————————————————–

Q:- You can block all login access with following command:
A:- touch /etc/nologin
———————————————————————–

Q:- It is good idea to encrypt backup made with tar command:
A:- tar -zcvf – *| openssl des3 -salt -k PASSWORD | dd of=mybackup.tbz
tar zcvf – /home |openssl des3 -salt -k PASSWORD | dd of=/dev/st0
To extract encrypted tar file use command:
dd if= mybackup.tbz |openssl des3 -d -k PASSWORD| tar zvxf –
dd if=/dev/st0|openssl des3 -d -k PASSWORD | tar xzf ————————————————————————

Q:- Delete a file securely, first overwriting it to hide its contents.
A:- $ shred -n 200 -z -u personalinfo.tar.gz
A:- srm filename
A:- wipe filename
—————————————
Q:- Delete file by inode:
A:- $ find . -inum 782263 -exec rm -i {} \;
————————————————————————

Q:- Forcefully unmount CD/DVD Rom or any other mounted partitions with
fuser command:
Ans:- fuser -km /dev/cdrom
fuser -km /mnt/cdrom\n
fuser -km /data2
————————————————————————

Q: List open files under user nobody

A:- lsof -u nobody
————————————————————————-

h1

What is /dev/shm and its practical usage ?

July 25, 2006

What is /dev/shm and its practical usage ?

/dev/shm is nothing but implementation of traditional shared memory concept. It is an efficient means of passing data between programs. One program will create a memory portion, which other processes (if permitted) can access. This will result into speeding up things on Linux.

If you type mount command you will see /dev/shm as a tempfs file system. Therefore, it is a file system, which keeps all files in virtual memory. Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. If you unmount a tmpfs instance, everything stored therein is lost. By default almost all distro configured to use /dev/shm.

Nevertheless, where can I use /dev/shm?
You can use /dev/shm to improve the performance of application software or overall Linux system performance. On heavily loaded system, it can make tons of difference. For example VMware workstation/server can be optimized to improve your Linux host’s performance (i.e. improve the performance of your virtual machines).

For example, if you have 8GB RAM then remount /dev/shm as follows:

# mount -o remount,size=8G /dev/shm

To be frank if you have more than 2GB RAM and if you running multiple Virtual machines this hack always improves performance.