Tag Archive: grep


Off lately their was buzz all around Internet regarding SSH Ebury trojan, lots of misconceptions!!

We have around 400 linux servers with over 8000 websites hosted on it, still remember the day when there was a bombard of abuse reports from providers.

I was like WTF!!! all servers with two factor authentication as 2nd layer of security and it got rooted?

We started with reading through Ebury with couple of commands already out on Internet with all the checks and memory dumps, we were all clean.

Why the hell we are receiving abuse alerts? That too indirectly from CBL?

Okay, by far I have read about Ebury and what I have understood is that their are only two ways to get this installed,

  1. server rooted
  2. yum repos via automate updates

Later part was what I worried about, I started with the commands already known to 90% of security freaks but all the theories ended up with two commands which according to CBL and providers and most of the people out there is enough to assume that server is infected by Ebury, commands are shown below,

  • ipcs -m (looking for shared memory segments with permission 666)
  • size of libkeyutils file greater than 25KB, for some, greater than 20KB and so on.

Let’s starts with “ipcs -m”,

Sample Output:

root@bt [/proc/8890]# ipcs -mp

—— Shared Memory Creator/Last-op ——–
shmid owner cpid lpid
3997696 root 1675 1675
4358145 root 2038 2057
3964930 root 1668 1668
4390915 root 2038 2057
4423684 root 2038 2057
5898245 root 2866 19322
5931014 root 2866 19322

root@bt

Now if I run ps aux | grep lpid or cpid as shown below,

root@bt [/proc/2866]# ps aux | grep 1668 | grep -v grep
root 2648 0.0 0.0 21668 900 ? Ss Apr10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root@bt [/proc/2866]#

so next we, cd /proc/2866 you’ll see cmdline file,

root@bt [/proc/2866]# cat cmdline
/usr/local/apache/bin/httpd-kstart-DSSL

this will show the actual command and exe linked to,

lrwxrwxrwx   1 root root 0 Apr 16 17:40 exe -> /usr/local/apache/bin/httpd

Their will be another file called as “smaps”

open the file and you will see to which bin/lib it’s linked to,

3449406000-3449606000 —p 00006000 08:02 2860848                        /usr/lib64/libltdl.so.3.1.4

344a000000-344a031000 r-xp 00000000 08:02 2868115                        /usr/lib64/libidn.so.11.5.19

from this you can grep the actual linking of libraries, this is only applicable if it’s 666 permission but this holds true only if it’s governed by SSHD process.

Having 666 permission is common and shouldn’t be concluded as hacked or effected.

we had these on servers not governed by any process, doing nothing.

  • Let’s check for “size of libkeyutils”,

This varies on linux flavors to flavors and from repos to repos, on cpanel servers yum update and /scripts/upcp update will always differ in size.

Same applies for amazon, “yum reinstall –enable remi libkeyutils” and “yum reinstall libkeyutils” which will take amazon repos will differ in size compared to remis.

In-case of any abuse reports, run through a series of commands and do not reload OS, that’s not the option.

By far the only reliable I found was starce command,

strace-sshd.sh  (Execute this script first)
 
STRACE=`ps aux | grep ssh[d] | awk ‘{print “-p ” $2}’`
echo $STRACE
strace -o out -f -s 5000 $STRACE
 
syscallparse.sh (from second window after executing ctrl-c on first terminal)
 
DIR=/root/syscalls
 
if [ ! -d “$DIR” ];
then
        mkdir $DIR
fi
 
for x in `awk ‘{print $2}’ /root/out | grep [a-z] | sed s/\(.*//g | sort | uniq`
do
grep -n “^[[:digit:]]\+[[:space:]]\+$x(” /root/out >> /root/syscalls/$x
done

Under /root/syscalls directory, you’ll find a file called as “connect”, this file will say to which outbound port it is trying to connect.

Sample:

479:32560 connect(3, {sa_family=AF_INET6, sin6_port=htons(22), inet_pton(AF_INET6, “::”, &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
483:32560 connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr(“0.0.0.0”)}, 16) = 0
488:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
492:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
589:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
593:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
633:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
652:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
671:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0

So what happens here?

the connection attempts to /var/run/nscd/socket, this means ssh first tried to connect to NSCD – the Name Service Cache Daemon — which is fine but the IP it attempts to connect should be one of the IPs from /etc/resolv.conf which is name server IP for name lookups.

IP other than name-server ip’s  then you should be  with WTF!!! face 😮

DNS is port 53 so “sin_port=htons(53)”, it doesn’t have “sendto” in above sample so nothing was being sent to port 53.

Never the less, I had prepared the script which checks almost everything apart from,

  • libkeyutils size
  • signatures of openssh rpms

which I will add this week, execute it from /root/, you’ll need hashes.txt file to which is at the bottom.

###Script Starts here###

#!/bin/bash

LIB64=/lib64/libkeyutils.so.1.9
LIB64_1=/lib64/libkeyutils-1.2.so.2
LIB32=/lib/libkeyutils.so.1.9
LIB32_1=/lib/libkeyutils-1.2.so.2
red=’\e[0;31m’
NC=’\e[0m’
green=’\e[0;32m’

if [ -f $LIB64 ]; then
echo The server is compromised, $LIB64 found
exit 0
fi

if [ -f $LIB64_1 ]; then
echo The server is compromised, $LIB64_1 found
exit 0
fi
if [ -f $LIB32 ]; then
echo The server is compromised, $LIB32 found
exit 0
fi

if [ -f $LIB32_1 ]; then
echo The server is compromised, $LIB32_1 found
exit 0
fi

echo -e “\t\t\t$HOSTNAME”
echo ” “
echo -e “${green}Cannot find compromised library: Clean${NC}”
#exit 1
SHMEM=`ipcs -m | grep 666`

if [ -z “$SHMEM” ];
then
echo -e “${green}Checking Shared Memory Segments with 666: Clean${NC}”
else
echo -e “${red}Checking Shared Memory Segments with 666: Infected${NC}”
ipcs -m
fi

ILLOPTION=`ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo “System clean” || echo “System infected”`
CHK=`echo $ILLOPTION | grep -ie infected`

if [ -z “$CHK” ];
then
echo -e “${green}Checking for SSH illegal option: $ILLOPTION ${NC}”
else
echo -e “${red}Checking for SSH illegal option: $ILLOPTION ${NC}”
fi
echo ” “
echo -e “\t\t\tChecking for Trojanized sshd, ssh, ssh-add:”
echo ” “
SHASUM=`which sha1sum`
SSH=`which ssh`
SSHD=`which sshd`
SSHADD=`which ssh-add`
SHASSH=`$SHASUM $SSH | awk ‘{ print $1 }’`
SHASSHD=`$SHASUM $SSHD | awk ‘{ print $1 }’`
SHASSHADD=`$SHASUM $SSHADD | awk ‘{ print $1 }’`
HASHSSH=`grep $SHASSH -F /root/hashes.txt`
HASHSSHD=`grep $SHASSHD -F /root/hashes.txt`
HASHSSHADD=`grep $SHASSHADD -F /root/hashes.txt`

if [ -z “$HASHSSH” ];
then
echo -e “${green}Hashes for $SSH not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSH matching with Linux/Ebury hashes: $HASHSSH,${NC}”
fi

if [ -z “$HASHSSHD” ];
then
echo -e “${green}Hashes for $SSHD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHD matching with Linux/Ebury hashes: $HASHSSHD,${NC}”
fi

if [ -z “$HASHSSHADD” ];
then
echo -e “${green}Hashes for $SSHADD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHADD matching with Linux/Ebury hashes: $HASHSSHADD,${NC}”
fi

echo ” “
echo -e “\t\t\tChecking for libkeyutils:”
echo ” “
for UTILS in `locate libkey | grep -v home`;
do
CHKHASH=`which sha1sum`
RES=`$CHKHASH $UTILS | grep -ie 98cdbf1e0d202f5948552cebaa9f0315b7a3731d -ie 4d12f98fd49e58e0635c6adce292cc56a31da2a2 -ie 0daa51519797cefedd52864be0da7fa1a93ca30b -ie 7314eadbdf18da424c4d8510afcc9fe5fcb56b39 -ie 575bb6e681b5f1e1b774fee0fa5c4fe538308814 -ie fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b -ie c4c28d0372aee7001c44a1659097c948df91985d -ie 267d010201c9ff53f8dc3fb0a48145dc49f9de1e -ie 471ee431030332dd636b8af24a428556ee72df37 -ie 58f185c3fe9ce0fb7cac9e433fb881effad31421 -ie 09c8af3be4327c83d4a7124a678bbc81e12a1de4 -ie 2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b -ie 39ec9e03edb25f1c316822605fe4df7a7b1ad94a -ie 3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf -ie 74aa801c89d07fa5a9692f8b41cb8dd07e77e407 -ie 7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 -ie 899b860ef9d23095edb6b941866ea841d64d1b26 -ie 8daad0a043237c5e3c760133754528b97efad459 -ie 8f75993437c7983ac35759fe9c5245295d411d35 -ie 9bb6a2157c6a3df16c8d2ad107f957153cba4236 -ie a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 -ie adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f -ie b8508fc2090ddee19a19659ea794f60f0c2c23ff -ie bbce62fb1fc8bbed9b40cfb998822c266b95d148 -ie bf1466936e3bd882b47210c12bf06cb63f7624c0 -ie e14da493d70ea4dd43e772117a61f9dbcff2c41c -ie f1ada064941f77929c49c8d773cbad9c15eba322 -ie 9e2af0910676ec2d92a1cad1ab89029bc036f599 -ie 5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 -ie d552cbadee27423772a37c59cb830703b757f35e -ie 1a9aff1c382a3b139b33eeccae954c2d65b64b90 -ie 2e571993e30742ee04500fbe4a40ee1b14fa64d7 -ie e2a204636bda486c43d7929880eba6cb8e9de068`

if [ -z “$CHKHASH” ];
then
echo -e “${red}Hashes matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
else
echo -e “${green}Hashes not matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
fi

done

###hashes.txt###

98cdbf1e0d202f5948552cebaa9f0315b7a3731d Linux/Ebury . Version 0.4.4 . sshd
4d12f98fd49e58e0635c6adce292cc56a31da2a2 Linux/Ebury . Version 0.4.4 . sshd
0daa51519797cefedd52864be0da7fa1a93ca30b Linux/Ebury . Version 0.8.0 . sshd
7314eadbdf18da424c4d8510afcc9fe5fcb56b39 Linux/Ebury . Version 0.8.0 . sshd
575bb6e681b5f1e1b774fee0fa5c4fe538308814 Linux/Ebury . Version 0.8.0 . ssh-add
fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b Linux/Ebury . Version 0.8.0 . ssh
c4c28d0372aee7001c44a1659097c948df91985d Linux/Ebury . Version 0.8.0 . ssh
267d010201c9ff53f8dc3fb0a48145dc49f9de1e Linux/Ebury . Version 1.1.0 . libkeyutils.so
471ee431030332dd636b8af24a428556ee72df37 Linux/Ebury . Version 1.2.1 . libkeyutils.so
58f185c3fe9ce0fb7cac9e433fb881effad31421 Linux/Ebury . Version 1.3.1 . libkeyutils.so
09c8af3be4327c83d4a7124a678bbc81e12a1de4 Linux/Ebury . Version 1.3.2 . libkeyutils.so
2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b Linux/Ebury . Version 1.3.2 . libkeyutils.so
39ec9e03edb25f1c316822605fe4df7a7b1ad94a Linux/Ebury . Version 1.3.2 . libkeyutils.so
3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf Linux/Ebury . Version 1.3.2 . libkeyutils.so
74aa801c89d07fa5a9692f8b41cb8dd07e77e407 Linux/Ebury . Version 1.3.2 . libkeyutils.so
7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 Linux/Ebury . Version 1.3.2 . libkeyutils.so
899b860ef9d23095edb6b941866ea841d64d1b26 Linux/Ebury . Version 1.3.2 . libkeyutils.so
8daad0a043237c5e3c760133754528b97efad459 Linux/Ebury . Version 1.3.2a . libkeyutils.so
8f75993437c7983ac35759fe9c5245295d411d35 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9bb6a2157c6a3df16c8d2ad107f957153cba4236 Linux/Ebury . Version 1.3.2 . libkeyutils.so
a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 Linux/Ebury . Version 1.3.2 . libkeyutils.so
adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f Linux/Ebury . Version 1.3.2 . libkeyutils.so
b8508fc2090ddee19a19659ea794f60f0c2c23ff Linux/Ebury . Version 1.3.2 . libkeyutils.so
bbce62fb1fc8bbed9b40cfb998822c266b95d148 Linux/Ebury . Version 1.3.2 . libkeyutils.so
bf1466936e3bd882b47210c12bf06cb63f7624c0 Linux/Ebury . Version 1.3.2 . libkeyutils.so
e14da493d70ea4dd43e772117a61f9dbcff2c41c Linux/Ebury . Version 1.3.2 . libkeyutils.so
f1ada064941f77929c49c8d773cbad9c15eba322 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9e2af0910676ec2d92a1cad1ab89029bc036f599 Linux/Ebury . Version 1.3.3b . libkeyutils.so
5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 Linux/Ebury . Version 1.3.3 . libkeyutils.so
d552cbadee27423772a37c59cb830703b757f35e Linux/Ebury . Version 1.3.3 . libkeyutils.so
1a9aff1c382a3b139b33eeccae954c2d65b64b90 Linux/Ebury . Version 1.3.4b1 . libkeyutils.so
2e571993e30742ee04500fbe4a40ee1b14fa64d7 Linux/Ebury . Version 1.3.4b2 . libkeyutils.so
e2a204636bda486c43d7929880eba6cb8e9de068 Linux/Ebury . Version 1.3.5 . libkeyutils.so

Sample Output:

Ebury

Never Ever rely on what provider says, believe in what you are doing and if you are 100% sure than just reply logically and up-to the mark,

Our Reply:

our

Provider’s Reply:

provider

Trust me on this ,we didn’t reload OS and we just end up killing shared segment with perm 666 not governed by any process using ipcrm command.

I end up creating this signature from snort to iptables,

-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j LOG –log-prefix “Ebury SSH Rootkit:” –log-level 7
-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j DROP

Final note for “SSH -G” option, picture is worth a thousand words

ssh

 

Ref: http://roumenpetrov.info/openssh/

Cheers!!!

Clam-Scan Revisited

We already had discussions on how we can make clam-scan rock solid, it’s implementation.

Ref: http://ustechnica.com/2013/06/19/clam-scan-from-an-eye-of-an-ethical-hacker/

Recently, A customer came to us with an issue of intermittent downtime of their websites.

and we just had a clam-scan execution completed, we have 4 scans running on Daily, weekly and monthly basis.

We assumed that it was clam-scan but client doesn’t work on assumptions so we implemented a reporting thing which will

not only figure out the load and memory consumption on server but also will check status of websites on that server during

execution of clam-scan.

Short Description of what script does:

  1. Mark the START time of scan
  2. child script will be executed from main script
  3. child script will execute while it reads pidof main script after every loop
  4. child script will keep on checking the HTTP status code and will keep on storing RAW data
  5. once Main script finishes execution, it will mark END time of scan
  6. during the START and END time, it will fetch CPU usage and Memory usage from SAR logs
  7. After that, reporting script will prepare the report from RAW data generated during execution of child script
  8. In the end you will get something like below,

clam1

clam2

In case your domain goes down,

then,

clam3

To achieve this,

Prerequisite:

1. SAR command , installed and configure it to read data every 3 mins.

root@MJ [~]# cat /etc/cron.d/sysstat
# run system activity accounting tool every 3 minutes
*/3 * * * * root /usr/lib/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 -A

root@MJ [~]#

2. Folder Structure

cd /root/clam-scan-project

mkdir http-status-code/

under http-status-code, create two scripts,

create-report.sh

status.sh*

Actual Script:

##Main.sh (/etc/cron.monthly/add-signatures.sh)

file=/root/clam-scan-project/current.txt
if [ -f $file ];
then
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
diff -I ‘^#’ /root/clam-scan-project/current.txt /root/clam-scan-project/host.txt | grep “^>” | grep -v localhost | awk ‘{ print $3 }’ >> /root/clam-scan-project/add-domains.txt
if [ -s /root/clam-scan-project/add-domains.txt ];
then
for i in $(cat /root/clam-scan-project/add-domains.txt)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi
else
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
for i in $(cat /root/clam-scan-project/host.txt | grep -v “^#” | awk ‘{ print $2 }’ | grep -v localhost)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi

DIR=/root/clam-scan-project/http-status-code
rm -rf $DIR/*.txt
RESULT1=/root/clam-scan-project/http-status-code/HTTP_CODES.log
START=`date +%H:%M`
OUTPUT=/root/clam-scan-project/default-db-summary.log
OS=`uname -mrs`
PROCESSOR=`cat /proc/cpuinfo | grep -i processor | wc -l`
VENDOR=`cat /proc/cpuinfo | grep vendor_id | uniq | cut -d “:” -f2`
MODEL=`cat /proc/cpuinfo | grep -i model\ name | uniq | cut -d “:” -f2`
RAM=`grep ‘MemTotal:’ /proc/meminfo | awk ‘{ print $2 }’`
TOTALRAM=`echo “scale=2;$RAM/1024” | bc`
DOMAINS=`cat /etc/trueuserdomains | wc -l`
echo “<center>” >> $OUTPUT
echo “<TABLE BORDER=”5″ WIDTH=”50%” CELLPADDING=”4″ CELLSPACING=”3″ bgcolor=”#FAEBD7″>” >> $OUTPUT
echo “<TR>” >> $OUTPUT
echo “<TH COLSPAN=”2″><BR><H3>Server Information</H3>” >> $OUTPUT
echo “</TH>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>OS</TD>” >> $OUTPUT
echo “<TD>$OS</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Vendor ID</TD>” >> $OUTPUT
echo “<TD>$VENDOR</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Processor</TD>” >> $OUTPUT
echo “<TD>$PROCESSOR</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Model Name</TD>” >> $OUTPUT
echo “<TD><PRE>$MODEL</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>RAM</TD>” >> $OUTPUT
echo “<TD><PRE>$TOTALRAM MB</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Domains Hosted</TD>” >> $OUTPUT
echo “<TD><PRE>$DOMAINS</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “</TABLE>” >> $OUTPUT
echo “<br>” >> $OUTPUT
echo “<table>” >> $OUTPUT
echo “<caption>Storage</caption>” >> $OUTPUT
echo “<tr>” >> $OUTPUT
echo “<td><b><PRE>`df -h | column -t`</b></PRE></td>” >> $OUTPUT
echo “</tr>” >> $OUTPUT
echo “</table>” >> $OUTPUT
echo “<br>” >> $OUTPUT
echo “</center>” >> $OUTPUT
echo “<center><b>Clam-Scan Started at: $START</b></center>” >> /root/clam-scan-project/default-db-summary.log
/bin/sh $DIR/status.sh &
bPid=”$bPid $!”
#echo “<center><h1><i>Clam-Scan</i></h1></center>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<b><i>Clam-Scan Result using Default DB</i></b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<i>Updating Clam AV database</i>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
freshclam –no-warnings >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
clamscan –exclude-dir=mail –exclude-dir=virtfs -ir /home/* –log /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</font>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/default-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/default-db-summary.log
echo “<b>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<HR ALIGN=”LEFT” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<b><i>Clam-Scan Result using Custom DB</i></b>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
clamscan -d /root/clam-scan-project/signatures.ndb –exclude-dir=tmp –exclude-dir=log –exclude-dir=mail –exclude-dir=virtfs -ir /home/* –log /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</font>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/custom-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/custom-db-summary.log
echo “<b>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</b>” >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/default-db-summary.log /root/clam-scan-project/custom-db-summary.log >> /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/current.txt
#mv /root/clam-scan-project/host.txt /root/clam-scan-project/current.txt
#rm -f /root/clam-scan-project/hosts.txt
sleep 500
END=`date +%H:%M`
echo “<center><b>Clam-Scan Ended: $END</b></center>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>CPU Load Average during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>`/usr/bin/sar -u -q -s $START:00 -e $END:00`</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>Memory Usage during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>`/usr/bin/sar -r -s $START:00 -e $END:00`</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log

array=( $(sar -r -s $START:00 -e $END:00 | tail -1 | awk ‘{ print $2,$3,$5,$6,$7 }’) )
kbmemfree=`echo “scale=2;${array[0]}/1024” | bc`
kbmemused=`echo “scale=2;${array[1]}/1024” | bc`
kbbuffers=`echo “scale=2;${array[2]}/1024” | bc`
kbcached=`echo “scale=2;${array[3]}/1024” | bc`
kbswpfree=`echo “scale=2;${array[4]}/1024” | bc`
echo “<b><center><PRE>Memory Free: $kbmemfree\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Memory Used: $kbmemused\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Cached: $kbcached\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Free Swap: $kbswpfree\MB</center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
/bin/sh $DIR/create-report.sh

if [ -f “$RESULT1” ];
then
echo “<b>HTTP Status Code during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
cat /root/http-status-code-project/TABLE_STRUCTURE.log >> /root/clam-scan-project/clam-scan-result.log
else
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>HTTP Status Code Check for below HTTP Codes:</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<ul>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 500: Internal Error</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 503: Gateway timeout</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 502: Service temporarily overloaded</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 408: Request Timeout</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 407: Proxy Authentication Required</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “</ul>” >> /root/clam-scan-project/clam-scan-result.log
echo “<center><FONT COLOR=”GREEN”>No downtime for domains on $HOSTNAME during scan</FONT></center>” >> /root/clam-scan-project/clam-scan-result.log
fi
(echo -e “From: clam-scan@exa.com.au \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Clam-Scan on $HOSTNAME \nContent-Type: text/html \n”; cat /root/clam-scan-project/clam-scan-result.log) | sendmail -t
rm -f /root/clam-scan-project/default-db-summary.log
rm -f /root/clam-scan-project/custom-db-summary.log
rm -f /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/summary.log
#mv -f /root/clam-scan-project/add-domains.txt /root/
rm -rf $DIR/*.log
rm -rf $DIR/*.txt

##Main.sh ENDs##

##Child Script, status.sh

DIR=/root/clam-scan-project/http-status-code/
FILE=url.txt
z=`pgrep -f “add-signatures.sh” | grep -v grep`
OUT=`echo $?`

while [ “$OUT” != 1 ];
do

for URLs in $(cat /etc/trueuserdomains | cut -d “:” -f1)
do

HTTP_CODE=`curl -s -o /dev/null -w “%{http_code}” “$URLs”`

if [ “$HTTP_CODE” -ne 200 ];
then
echo “www.$URLs” >> $DIR$FILE
if [ -f “$DIR$URLs.txt” ];
then
FOUND=`grep $HTTP_CODE $DIR$URLs.txt`

if [ -z “$FOUND” ];
then
echo “HTTP-$HTTP_CODE:0” >> $DIR$URLs.txt
else
COUNT=`cat $DIR$URLs.txt | grep HTTP-$HTTP_CODE | cut -d “:” -f2`
NEWCOUNT=`expr $COUNT + 1`
sed -i “s/HTTP-$HTTP_CODE.*/HTTP-$HTTP_CODE:$NEWCOUNT/g” $DIR$URLs.txt
fi
else
echo “$URLs” >> $DIR$URLs.txt
echo “HTTP-$HTTP_CODE:0” >> $DIR$URLs.txt
fi

fi
done

z=`pgrep -f “add-signatures.sh” | grep -v grep`
OUT=`echo $?`

done

##Ends##

##Create Report##

DIR=/root/clam-scan-project/http-status-code/
FILE=url.txt
RESULT=/root/clam-scan-project/http-status-code/TABLE_STRUCTURE.log
RESULT1=/root/clam-scan-project/http-status-code/HTTP_CODES.log
echo “<center>” >> $RESULT
echo “<TABLE BORDER=4 ALIGN=center CELLPADDING=10 CELLSPACING=2>” >> $RESULT
echo “<TR>” >> $RESULT
echo “<TH WIDTH=”5%”>Domain Name</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>HTTP Status Code</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>Headers</TH>” >> $RESULT
echo “</TR>” >> $RESULT

for DOMAINS in $(cat $DIR$FILE | sed ‘s/^[^\.]\+\.//’)
do
HTTP_STATUS_CODE=`grep HTTP-* $DIR$DOMAINS.txt | sed ‘/^$/d’`
SHORT_CODE=`echo $HTTP_STATUS_CODE | awk -F”[-:]” ‘{ print $2}’`

if [ “$SHORT_CODE” == 503 ] || [ “$SHORT_CODE” == 502 ] || [ “$SHORT_CODE” == 408 ] || [ “$SHORT_CODE” == 500 ] || [ “$SHORT_CODE” == 407 ];
then
HEADERS=`curl -sI “www.$DOMAINS”`

echo “<TR>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$DOMAINS</PRE></TD>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$HTTP_STATUS_CODE</PRE></TD>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$HEADERS</PRE></TD>” >> $RESULT1
echo “</TR>” >> $RESULT1
fi
done

if [ -f “$RESULT1” ];
then
cat $RESULT1 | col -b >> $RESULT
fi

echo “</center>” >> $RESULT
echo “</TABLE>” >> $RESULT

##Ends##

Cheers!!!

Advanced Intrusion Detection Environment

What is AIDE?

AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker.

It is a host-based intrusion detection system (HIDS) for checking the integrity of files. It does this by creating a baseline database of files on an initial run, and then checks this database against the system on subsequent runs. File properties that can be checked against include inode, permissions, modification time, file contents, etc.

Scenario: Detect changes in Zone files in such a way that it should report changes with the difference between old record and new record.

Step by Step Installation:

1. Install Mhash

Mhash is a free (under GNU Lesser GPL) library which provides a uniform interface to a large number of hash algorithms. These algorithms can be used to compute checksums, message digests, and other signatures.

  • wget http://sourceforge.net/projects/mhash/files/mhash/0.9.9/mhash-0.9.9.tar.gz
  • tar -xzvf mhash-0.9.9.tar.gz
  • cd mhash-0.9.9.9/
  • ./configure

If error occurs like “configure error: you don’t have zlib properly installed” then,

  • ./configure –without-zlib
  • make && make install

2. Install AIDE

3. Run rsync command for the Initial setup

  • mkdir /home/dnsbackup
  • rsync /var/named/* /home/dnsbackup/

4. copy below conf file and replace it with file under root/aide-0.15/doc/aide.conf

Note: Before that, cp /root/aide-0.15/doc/aide.conf  /root/ ….(Backup) 🙂

#
# AIDE 0.15
#
# example configuration file
#
# IMPORTANT NOTE!! PLEASE READ
#
# This configuration file checks the integrity of the
# AIDE package.
#
# This file is not intended to be used as the primary aide.conf file for
# your system. This file is intended to be a showcase for different
# features for aide.conf file.
#
# WRITE YOUR OWN CONFIGURATION FILE AND UNDERSTAND WHAT YOU ARE WRITING
#
#
# Default values for the parameters are in comments before the
# corresponding line.
#

@@define TOPDIR /root/aide-0.15
@@define LOGDIR /var/log/aide

@@ifndef TOPDIR
@@define TOPDIR /
@@endif

@@ifdef DEBUG
@@define DEBUG ison
@@undef NOT_DEBUG
@@else
@@define NOT_DEBUG true
@@undef DEBUG
@@endif

@@ifhost korppi
@@define KORPPI yes
@@endif

@@ifnhost ftp
@@define BUMMER true
@@endif

# The location of the database to be read.
#database=file:aide.db
database=file:/home/aide/aide.db

# The location of the database to be written.
#database_out=sql:host:port:database:login_name:passwd:table
#database_out=file:aide.db.new
database_out=file:@@{TOPDIR}/doc/aide.db.new

# Whether to gzip the output to database
#gzip_dbout=yes

#verbose=5
verbose=20

report_url=stdout
#other possibilities
#report_url=stderr
#NOT IMPLEMENTED report_url=mailto:root@foo.com
#report_url=file:/tmp/aide.txt
#report_url=file:@@{LOGDIR}/aide.log
#report_url=syslog:LOG_AUTH
#report_url=stdout

# @@{TOPDIR} is replaced with /root/aide-0.15 when
# read by aide.
#p: permissions
#ftype: file type
#i: inode
#n: number of links
#l: link name
#u: user
#g: group
#s: size
#b: block count
#m: mtime
#a: atime
#c: ctime
#S: check for growing size
#I: ignore changed filename
#md5: md5 checksum
#sha1: sha1 checksum
#sha256: sha256 checksum
#sha512: sha512 checksum
#rmd160: rmd160 checksum
#tiger: tiger checksum
#haval: haval checksum
#crc32: crc32 checksum
#R: p+ftype+i+l+n+u+g+s+m+c+md5
#L: p+ftype+i+l+n+u+g
#E: Empty group
#>: Growing logfile p+ftype+l+u+g+i+n+S
#The following are available if you have mhash support enabled:
#gost: gost checksum
#whirlpool: whirlpool checksum
#The following are available and added to the default groups R, L and >
#only when explicitly enabled using configure:
#acl: access control list
#selinux SELinux security context
#xattrs: extended file attributes
#e2fsattrs: file attributes on a second extended file system

# Rule definition
All=R+a+sha1+rmd160+sha256+sha512+tiger+whirlpool

#Let Logfile grow

#LOG = >

# report_attributes is a special rule definition
# the attributes listed in it are alway displayed for changed files
# in the final report
report_attributes = u+g

# ignore_list is a special rule definition
# the attributes listed in it are not displayed in the
# final report, it overrules report_attributes where they conflict
#ignore_list = b

# Attributes that can be used to verify that aide in intact
# by people that have downloaded it from the web.
# Let’s be paranoid
Norm=l+s+n+b+u+g+a+m+c+ftype

# The commented rules are just examples the rest are used by
# make check

#AIDE check on following folders

/var/named Norm
!/var/named/data
!/var/named/chroot
!/var/named/slaves

#Selection regexp rule
@@{TOPDIR}/.* Norm
#Equals selection only the directory doc is checked and not it’s children
#=@@{TOPDIR}/doc L
#Negative selection no rule is necessary but ignored if there
!@@{TOPDIR}/.*~
!@@{TOPDIR}/src/.*\.o
!@@{TOPDIR}/src/(aide|core)$ L
!@@{TOPDIR}/.*RCS
!@@{TOPDIR}/.*CVS
!@@{TOPDIR}/.*aide\.db.*
!@@{TOPDIR}/.*\.cvsignore.*
# @@{TOPDIR}/doc/.* All

  • mkdir /home/aide
  • chown nobody:nobody /home/aide
  • /usr/local/bin/aide -c /root/aide-0.15/doc/aide.conf –init
  • echo yes | cp /root/aide-0.15/doc/aide.db.new /home/aide/aide.db
  • Check the permission of /home/aide/aide.db should be root:root, if not, use command:- chown root:root /home/aide/aide.db
  • Create file /root/aide_html.sh and copy below content in it.

 

Script:

##Starts Here##

#!/bin/bash
#echo “<body bgcolor=”#6D7B8D”>” >> /tmp/aide.log
echo “<h2><center><i>AIDE Report on changes done</i></center></h2>” >> /tmp/aide.log
echo “<HR ALIGN=”CENTER” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /tmp/aide.log
echo “<br>” >> /tmp/aide.log
echo “<pre>” >> /tmp/aide.log
/usr/local/bin/aide -c /root/aide-0.15/doc/aide.conf –check >> /tmp/aide.log
echo “</pre>” >> /tmp/aide.log
echo “</body>” >> /tmp/aide.log
sleep 10
grep -wi “All files match AIDE database” /tmp/aide.log
if [ $? -eq 1 ]
then
cat /tmp/aide.log | mail -s “$(echo -e “AIDE Changes under /var/named/ on $HOSTNAME\nContent-Type: text/html”)” youremailaddress@domain.com — -f AIDE@domain.com
grep ‘File’ /tmp/aide.log | awk ‘{ print $2 }’ | awk -F ‘/’ ‘{ print $4 }’ > /tmp/grep.txt
if [ -s /tmp/grep.txt ]
then
#echo “<body bgcolor=”#6D7B8D”>” >> /tmp/domains.txt
echo “<h2><center><i>Detailed AIDE Report</i></center></h2>” >> /tmp/domains.txt
echo “<HR ALIGN=”CENTER” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /tmp/domains.txt
for i in $(cat /tmp/grep.txt)
do
echo “<br>” >> /tmp/domains.txt
echo “<i><b>AIDE Report for $i</b></i>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<pre>” >> /tmp/domains.txt
sed -n “/File: \/var\/named\/$i/,/Ctime/p” /tmp/aide.log >> /tmp/domains.txt
echo “</pre>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<b><i><center>Difference found for $i under /var/named/</center></i></b>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
minus=$(diff -u –expand-tabs –ignore-all-space –ignore-blank-lines –ignore-space-change /home/dnsbackup/$i /var/named/$i | grep “^-” | grep -iv “cPanel” | grep -iv “serial number”)
plus=$(diff -u –expand-tabs –ignore-all-space –ignore-blank-lines –ignore-space-change /home/dnsbackup/$i /var/named/$i | grep “^+” |grep -v “cPanel” | grep -iv “serial number”)
echo “<TABLE BORDER=7 ALIGN=”center” CELLPADDING=”8″>” >> /tmp/domains.txt
echo “<caption>$i</caption>” >> /tmp/domains.txt
echo “<TR>” >> /tmp/domains.txt
echo “<TH WIDTH=”20%”>Before</TH>” >> /tmp/domains.txt
echo “<TH WIDTH=”20%”>After</TH>” >> /tmp/domains.txt
echo “</TR>” >> /tmp/domains.txt
echo “<TR>” >> /tmp/domains.txt
echo “<TD ALIGN=”char”><PRE>$minus</PRE></TD>” >> /tmp/domains.txt
echo “<TD ALIGN=”char”><PRE>$plus</PRE></TD>” >> /tmp/domains.txt
echo “</TR>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “</table>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<b><i><center>Difference ends for $i</center></i></b>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
done


(echo -e “From: aideReport@domain.com \nTo: youremailaddress@domain.com \nCc:youremailaddress@domain.com \nMIME-Version: 1.0 \nSubject: AIDE Report for $HOSTNAME \nContent-Type: text/html \n”; cat /tmp/domains.txt) | sendmail -t
fi
fi
rm -f /tmp/domains.txt
rm -f /tmp/aide.log
/usr/local/bin/aide -c /root/aide-0.15/doc/aide.conf –update
echo yes | cp /root/aide-0.15/doc/aide.db.new /home/aide/aide.db
rsync /var/named/* /home/dnsbackup

##Script Ends here##

Results:

aide

 

 

Cheers!!!!

Xtrabackup Script

Percona XtraBackup is an open-source hot backup utility for MySQL – based servers that doesn’t lock your database during the backup.

It can back up data from InnoDBXtraDB, and MyISAM tables on unmodified MySQL 5.0, 5.1 and 5.5 servers, as well asPercona Server with XtraDB.

Installation on Cent OS 5.x, 6.x series/ Red Hat Linux 5.x and 6.x series.

Step 1. uname -i

For 64 bit OS,

rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm

For 32 bit OS,

rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.i386.rpm

step 2. vi /etc/my.cnf, add below parameter

datadir=/var/lib/mysql (if not added already, MySql restart is not required)

step 3. root@bt [~]# yum -y install xtrabackup

step 4. Make sure Innodb is set to Yes

root@bt [~]#mysqladmin variables | grep have_innodb
| have_innodb | YES

if it is NO then vi /etc/my.cnf and remove “skip-innodb” from the file and restart mysql service and if still not able to see it YES.

Go to /var/lib/mysql/ and delete ibdata1 and its log files ib_logfile0 and ib_logfile1 and re ran mysql service.

Check ownership for iddata1 file at /var/lib/mysql as it should be mysql:mysql and if it is different then change it by below command

root@bt[~]#chown mysql:mysql /var/lib/mysql/ibdata1

Step 5. Copy and Paste below script,

Script STARTS here

rm -f /Backup/mount.txt
rm -f /Backup/errors.txt
rm -f /root/mysql119bck.log
BACKUPDIR=/Backup

if [ ! -d “$BACKUPDIR” ];
then
mkdir /Backup
fi

INNOBACKUPEX=innobackupex-1.5.1
TMPFILE=”/tmp/innobackupex-runner.$$.tmp”
AGE=4

/usr/bin/innobackupex-1.5.1 /Backup/ > $TMPFILE 2>&1

if [ -z “`tail -1 $TMPFILE | grep ‘completed OK!’`” ] ; then
echo “$INNOBACKUPEX failed:” >> /root/mysql119bck.log
echo “” >> /root/mysql119bck.log
echo “———- ERROR OUTPUT from $INNOBACKUPEX ———-” >> /root/mysql119bck.log
cat /root/mysql119bck.log | mail -s “$HOSTNAME Backup Report” abc@domain.com — -f mysql@domain.com

cat $TMPFILE
rm -f $TMPFILE
exit 1
fi

THISBACKUP=`cat $TMPFILE | grep ‘Backup created in directory’ | awk ‘{ print $6 }’ | sed “s/’//g”`

rm -f $TMPFILE

echo “Databases backed up successfully to: $THISBACKUP” >> /root/mysql119bck.log
echo ” ” >> /root/mysql119bck.log
echo “Now applying logs to the backuped databases” >> /root/mysql119bck.log

/usr/bin/innobackupex-1.5.1 –use-memory=2G –apply-log $THISBACKUP > $TMPFILE 2>&1

if [ -z “`tail -1 $TMPFILE | grep ‘completed OK!’`” ] ; then
echo “$INNOBACKUPEX –apply-log failed:” >> /root/mysql119bck.log
echo ” ” >> /root/mysql119bck.log
echo “———- ERROR OUTPUT from $INNOBACKUPEX –apply-log ———-” >> /root/mysql119bck.log
echo ” ” >> /root/mysql119bck.log
cat $TMPFILE >> /root/mysql119bck.log
cat /root/mysql119bck.log | mail -s “$HOSTNAME Backup Report” abc@domain.com — -f mysql@domain.com

rm -f $TMPFILE
exit 1
fi

echo “Logs applied to backuped databases” >> /root/mysql119bck.log
echo

#Compress backup

echo “Compressing backup files” >> /root/mysql119bck.log


tar -czvf /Backup/backup_Mysql_119_`date +%d-%m-%Y-%H`.tar.gz $THISBACKUP
rm -rf $THISBACKUP
# Cleanup

echo “Cleaning up old backups (older than $AGE days) and temporary files” >> /root/mysql119bck.log
rm -rf $TMPFILE
cd /tmp ; find $BACKUPDIR -maxdepth 1 -ctime +$AGE -exec echo “removing: “{} \; -exec rm -rf {} \;

echo >> /root/mysql119bck.log
echo “completed: `date`” >> /root/mysql119bck.log

#Moving to Remote Location (Assuming, Remote dir is mounted on the server)

ls -lh /mysqlM3hrs > $BACKUPDIR/mount.txt

if [ ! -s $BACKUPDIR/mount.txt ];
then
echo “Backup not mounted… Can’t move backup to Remote server” >> /root/mysql119bck.log
else
file=`ls /Backup/*.tar.gz | cut -d “/” -f3`
mv -f /Backup/$file /mysqlM3hrs/ 2> /Backup/errors.txt

if [ -s /Backup/errors.txt ];
then
cat /Backup/errors.txt >> /root/mysql119bck.log
echo “Copy failed on Remote server … Please check” >> /root/mysql119bck.log
else
if [ ! -f `ls /mysqlM3hrs/$file` ]; then
echo “Copy failed on Remote server … Please check” >> /root/mysql119bck.log
else
echo “Copied Successful to Remote server” >> /root/mysql119bck.log
echo >> /root/mysql119bck.log
echo `ls -lh /mysqlM3hrs/$file` >> /root/mysql119bck.log
echo >> /root/mysql119bck.log
fi
fi
fi
cat /root/mysql119bck.log | mail -s “$HOSTNAME Backup Report”  abc@domain.com — -f  mysql@domain.com

rm -f /root/mysql119bck.log
exit 0

Script ENDS here

what does the script do?

1. well, first it will check for all the prerequisite.

2. Starts with the backup and checks for “completed OK!” in the tmp file.

3. If failed then email and exit, if all good then,

4. It will use 2G of RAM and will apply logs on the backup.

5. It will check for “Completed OK!” in the tmp file

6. If all good then proceed to next step or else email and exit.

7. will compress the backup and will clean up the backup older than 4 days in the Backup DIR.

8. Will move the backup to the remote server. Also will check whether the mount point exists, if not then email that copy to remote location failed.

Restoration from the Backup taken from above script.

step 1. tar -xzvf backup.tar.gz

step2 . service mysql stop

step 3. mv /var/lib/mysql /root/ (Just in case :))

step 4. rm -rf /var/lib/mysql/*

step 5. innobackupex-1.5.1 –copy-back /root/untared_backup (From Step 1.)

step 6: chown -R mysql:mysql /var/lib/mysql

step 7. service mysql start

why do we apply logs?

To prepare the backup use the –apply-log option and specify the timestamped subdirectory of the backup. To speed up the apply-log process, we using the –use-memory option is recommended.

In short, we are Making a Local Full Backup (Create, Prepare and Restore)

PS: Tested and trust me on this, have saved our asses big time. 

ICMP v/s arping

It’s been a while since my last post, was busy with work.

I have been asked many times, how will I check whether remote server is up or down?

Most common & logical answer is by pinging!

But what if the ping request is disabled? How would you find out?

Imagining the lack of basic knowledge from people I came across who are CCNA, CCNP, RHCE certified + years of experience in networking. Majority of them bluffed or well, lets just put it; Did NOT have any answer!!

Let’s go through this and find the valid answer and a basics behind it.

As we say, “Common sense is not common” just like “Basics is not a basic thing, it’s a foundation of what you think and how you implement”.

In the span of my 5 years of experience, I’ve come across many projects and majority of them were stuck because of half knowledge and missing basics in the implementation.

Scenario 1: Where ping request is enabled.

root@bt:~# ping 10.2.30.226
PING 10.2.30.226 (10.2.30.226) 56(84) bytes of data.
64 bytes from 10.2.30.226: icmp_seq=1 ttl=64 time=1.02 ms
64 bytes from 10.2.30.226: icmp_seq=2 ttl=64 time=0.462 ms
^C
— 10.2.30.226 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.462/0.745/1.029/0.284 ms
root@bt:~# arping 10.2.30.226
ARPING 10.2.30.226
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=0 time=338.000 usec
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=1 time=212.000 usec
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=2 time=220.000 usec
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=3 time=257.000 usec
^C
— 10.2.30.226 statistics —
4 packets transmitted, 4 packets received,   0% unanswered (0 extra)

Let’s check arp cache,

root@bt:~# ip neigh show | grep 10.2.30.226
10.2.30.226 dev eth0 lladdr 26:fc:99:e8:47:42 REACHABLE

As you can see from apr cache, destination server is REACHABLE.

Scenario 2: Ping is disabled.

First, disable ping request on a destination server.

  • [root@10.2.30.226 ~]# echo “1” > /proc/sys/net/ipv4/icmp_echo_ignore_all
  • root@bt:~# ping 10.2.30.226
    PING 10.2.30.226 (10.2.30.226) 56(84) bytes of data.
    From 10.2.50.16 icmp_seq=1 Destination Host Unreachable
    From 10.2.50.16 icmp_seq=2 Destination Host Unreachable
    — 10.2.30.226 ping statistics —
    7 packets transmitted, 0 received, +6 errors, 100% packet loss, time 6018ms
  • Now let’s see what arping has to say,

root@bt:~# arping 10.2.30.226
ARPING 10.2.30.226
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=0 time=407.000 usec
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=1 time=217.000 usec
60 bytes from 26:fc:99:e8:47:42 (10.2.30.226): index=2 time=209.000 usec

  • root@bt:~# ip neigh show | grep 10.2.30.226
    10.2.30.226 dev eth0 lladdr 26:fc:99:e8:47:42 REACHABLE

Even ping request is disabled, we can check whether server is UP or Down without any monitoring tools.

what if the server is shutdown.

  • root@bt:~# ip neigh show | grep 10.2.30.226
    10.2.30.226 dev eth0 lladdr 26:fc:99:e8:47:42 STALE

As you can see, STALE is the status of the server.

Explanation:

arping sends TCP request to the server, we can disable ping request but we can’t disable TCP request as it will bring down majority services running on the server.

Let’s take an example of a server pinging but arp cache shows STALE, always a good idea to update cache entries.

  • root@bt:~# ip neigh show
    10.2.30.15 dev eth0 lladdr 00:1c:c0:7c:0f:f4 REACHABLE
    10.2.30.245 dev eth0 lladdr a2:81:17:5c:3c:35 STALE
    10.2.120.250 dev eth0 lladdr 00:1e:2a:49:93:cd STALE
  • 10.2.120.250 is STALE.

really???

  • root@bt:~# arping -u -I eth0 10.2.120.250 (updating arp cache)
    ARPING 10.2.120.250
    60 bytes from 00:1e:2a:49:93:cd (10.2.120.250): index=0/0 time=12.000 usec
    60 bytes from 00:1e:2a:49:93:cd (10.2.120.250): index=1/1 time=5.000 usec
  • root@bt:~# ip neigh show | grep 10.2.120.250
    10.2.120.250 dev eth0 lladdr 00:1e:2a:49:93:cd REACHABLE

It’s always a good habit to clear arp cache,

can be done using,

  • root@bt:~# arp -d ip-address
  • arp cache get’s information from the file, cat /proc/net/arp

In case, arp cache still shows deleted entries and throws error like “SIOCDARP(priv): Network is unreachable”  then theirs a work around

Example: IP is 10.2.30.226

then,

root@bt [~]# ip addr add 10.2.30.226 dev eth0

root@bt [~]# ip addr del 10.2.30.226 dev eth0

then check arp -n and it won’t show deleted entries any more.

Ref: https://lkml.org/lkml/2003/8/20/62

Cheers!!!