Tag Archive: wget


Clam-Scan Revisited

We already had discussions on how we can make clam-scan rock solid, it’s implementation.

Ref: http://ustechnica.com/2013/06/19/clam-scan-from-an-eye-of-an-ethical-hacker/

Recently, A customer came to us with an issue of intermittent downtime of their websites.

and we just had a clam-scan execution completed, we have 4 scans running on Daily, weekly and monthly basis.

We assumed that it was clam-scan but client doesn’t work on assumptions so we implemented a reporting thing which will

not only figure out the load and memory consumption on server but also will check status of websites on that server during

execution of clam-scan.

Short Description of what script does:

  1. Mark the START time of scan
  2. child script will be executed from main script
  3. child script will execute while it reads pidof main script after every loop
  4. child script will keep on checking the HTTP status code and will keep on storing RAW data
  5. once Main script finishes execution, it will mark END time of scan
  6. during the START and END time, it will fetch CPU usage and Memory usage from SAR logs
  7. After that, reporting script will prepare the report from RAW data generated during execution of child script
  8. In the end you will get something like below,

clam1

clam2

In case your domain goes down,

then,

clam3

To achieve this,

Prerequisite:

1. SAR command , installed and configure it to read data every 3 mins.

root@MJ [~]# cat /etc/cron.d/sysstat
# run system activity accounting tool every 3 minutes
*/3 * * * * root /usr/lib/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 -A

root@MJ [~]#

2. Folder Structure

cd /root/clam-scan-project

mkdir http-status-code/

under http-status-code, create two scripts,

create-report.sh

status.sh*

Actual Script:

##Main.sh (/etc/cron.monthly/add-signatures.sh)

file=/root/clam-scan-project/current.txt
if [ -f $file ];
then
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
diff -I ‘^#’ /root/clam-scan-project/current.txt /root/clam-scan-project/host.txt | grep “^>” | grep -v localhost | awk ‘{ print $3 }’ >> /root/clam-scan-project/add-domains.txt
if [ -s /root/clam-scan-project/add-domains.txt ];
then
for i in $(cat /root/clam-scan-project/add-domains.txt)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi
else
wget -O /root/clam-scan-project/hosts.txt http://www.malwaredomainlist.com/hostslist/hosts.txt
cat /root/clam-scan-project/hosts.txt | col -b >> /root/clam-scan-project/host.txt
for i in $(cat /root/clam-scan-project/host.txt | grep -v “^#” | awk ‘{ print $2 }’ | grep -v localhost)
do
x=$(echo $i | sigtool –hex-dump | sed ‘s/\(.*\)../\1/’)
echo “$i:0:*:$x” >> /root/clam-scan-project/signatures.ndb
done
fi

DIR=/root/clam-scan-project/http-status-code
rm -rf $DIR/*.txt
RESULT1=/root/clam-scan-project/http-status-code/HTTP_CODES.log
START=`date +%H:%M`
OUTPUT=/root/clam-scan-project/default-db-summary.log
OS=`uname -mrs`
PROCESSOR=`cat /proc/cpuinfo | grep -i processor | wc -l`
VENDOR=`cat /proc/cpuinfo | grep vendor_id | uniq | cut -d “:” -f2`
MODEL=`cat /proc/cpuinfo | grep -i model\ name | uniq | cut -d “:” -f2`
RAM=`grep ‘MemTotal:’ /proc/meminfo | awk ‘{ print $2 }’`
TOTALRAM=`echo “scale=2;$RAM/1024” | bc`
DOMAINS=`cat /etc/trueuserdomains | wc -l`
echo “<center>” >> $OUTPUT
echo “<TABLE BORDER=”5″ WIDTH=”50%” CELLPADDING=”4″ CELLSPACING=”3″ bgcolor=”#FAEBD7″>” >> $OUTPUT
echo “<TR>” >> $OUTPUT
echo “<TH COLSPAN=”2″><BR><H3>Server Information</H3>” >> $OUTPUT
echo “</TH>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>OS</TD>” >> $OUTPUT
echo “<TD>$OS</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Vendor ID</TD>” >> $OUTPUT
echo “<TD>$VENDOR</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Processor</TD>” >> $OUTPUT
echo “<TD>$PROCESSOR</TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Model Name</TD>” >> $OUTPUT
echo “<TD><PRE>$MODEL</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>RAM</TD>” >> $OUTPUT
echo “<TD><PRE>$TOTALRAM MB</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “<TR ALIGN=”CENTER”>” >> $OUTPUT
echo “<TD>Domains Hosted</TD>” >> $OUTPUT
echo “<TD><PRE>$DOMAINS</PRE></TD>” >> $OUTPUT
echo “</TR>” >> $OUTPUT
echo “</TABLE>” >> $OUTPUT
echo “<br>” >> $OUTPUT
echo “<table>” >> $OUTPUT
echo “<caption>Storage</caption>” >> $OUTPUT
echo “<tr>” >> $OUTPUT
echo “<td><b><PRE>`df -h | column -t`</b></PRE></td>” >> $OUTPUT
echo “</tr>” >> $OUTPUT
echo “</table>” >> $OUTPUT
echo “<br>” >> $OUTPUT
echo “</center>” >> $OUTPUT
echo “<center><b>Clam-Scan Started at: $START</b></center>” >> /root/clam-scan-project/default-db-summary.log
/bin/sh $DIR/status.sh &
bPid=”$bPid $!”
#echo “<center><h1><i>Clam-Scan</i></h1></center>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<b><i>Clam-Scan Result using Default DB</i></b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<i>Updating Clam AV database</i>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
freshclam –no-warnings >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
clamscan –exclude-dir=mail –exclude-dir=virtfs -ir /home/* –log /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</font>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/default-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/default-db-summary.log
echo “<b>” >> /root/clam-scan-project/default-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/default-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/default-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/default-db-summary.log
echo “</b>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/default-db-summary.log
echo “<HR ALIGN=”LEFT” SIZE=”3″ WIDTH=”70%” NOSHADE>” >> /root/clam-scan-project/default-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<b><i>Clam-Scan Result using Custom DB</i></b>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
echo “<font color=”red”>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
clamscan -d /root/clam-scan-project/signatures.ndb –exclude-dir=tmp –exclude-dir=log –exclude-dir=mail –exclude-dir=virtfs -ir /home/* –log /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</font>” >> /root/clam-scan-project/custom-db-summary.log
echo “<br>” >> /root/clam-scan-project/custom-db-summary.log
sed -n “/SCAN \SUMMARY/,/Time:/p” /root/clam-scan-project/custom-db-summary.log > /root/clam-scan-project/summary.log
sed -i “/SCAN \SUMMARY/,/Time:/d” /root/clam-scan-project/custom-db-summary.log
echo “<b>” >> /root/clam-scan-project/custom-db-summary.log
echo ‘<pre>’ >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/summary.log >> /root/clam-scan-project/custom-db-summary.log
echo ‘</pre>’ >> /root/clam-scan-project/custom-db-summary.log
echo “</b>” >> /root/clam-scan-project/custom-db-summary.log
cat /root/clam-scan-project/default-db-summary.log /root/clam-scan-project/custom-db-summary.log >> /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/current.txt
#mv /root/clam-scan-project/host.txt /root/clam-scan-project/current.txt
#rm -f /root/clam-scan-project/hosts.txt
sleep 500
END=`date +%H:%M`
echo “<center><b>Clam-Scan Ended: $END</b></center>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>CPU Load Average during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>`/usr/bin/sar -u -q -s $START:00 -e $END:00`</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>Memory Usage during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>`/usr/bin/sar -r -s $START:00 -e $END:00`</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log

array=( $(sar -r -s $START:00 -e $END:00 | tail -1 | awk ‘{ print $2,$3,$5,$6,$7 }’) )
kbmemfree=`echo “scale=2;${array[0]}/1024” | bc`
kbmemused=`echo “scale=2;${array[1]}/1024” | bc`
kbbuffers=`echo “scale=2;${array[2]}/1024” | bc`
kbcached=`echo “scale=2;${array[3]}/1024” | bc`
kbswpfree=`echo “scale=2;${array[4]}/1024” | bc`
echo “<b><center><PRE>Memory Free: $kbmemfree\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Memory Used: $kbmemused\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Cached: $kbcached\MB</PRE></center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b><center><PRE>Free Swap: $kbswpfree\MB</center></b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
/bin/sh $DIR/create-report.sh

if [ -f “$RESULT1” ];
then
echo “<b>HTTP Status Code during scan</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
cat /root/http-status-code-project/TABLE_STRUCTURE.log >> /root/clam-scan-project/clam-scan-result.log
else
echo “<br>” >> /root/clam-scan-project/clam-scan-result.log
echo “<b>HTTP Status Code Check for below HTTP Codes:</b>” >> /root/clam-scan-project/clam-scan-result.log
echo “<ul>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 500: Internal Error</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 503: Gateway timeout</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 502: Service temporarily overloaded</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 408: Request Timeout</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “<li>HTTP 407: Proxy Authentication Required</li>” >> /root/clam-scan-project/clam-scan-result.log
echo “</ul>” >> /root/clam-scan-project/clam-scan-result.log
echo “<center><FONT COLOR=”GREEN”>No downtime for domains on $HOSTNAME during scan</FONT></center>” >> /root/clam-scan-project/clam-scan-result.log
fi
(echo -e “From: clam-scan@exa.com.au \nTo: abc@gmail.com \nMIME-Version: 1.0 \nSubject: Clam-Scan on $HOSTNAME \nContent-Type: text/html \n”; cat /root/clam-scan-project/clam-scan-result.log) | sendmail -t
rm -f /root/clam-scan-project/default-db-summary.log
rm -f /root/clam-scan-project/custom-db-summary.log
rm -f /root/clam-scan-project/clam-scan-result.log
rm -f /root/clam-scan-project/summary.log
#mv -f /root/clam-scan-project/add-domains.txt /root/
rm -rf $DIR/*.log
rm -rf $DIR/*.txt

##Main.sh ENDs##

##Child Script, status.sh

DIR=/root/clam-scan-project/http-status-code/
FILE=url.txt
z=`pgrep -f “add-signatures.sh” | grep -v grep`
OUT=`echo $?`

while [ “$OUT” != 1 ];
do

for URLs in $(cat /etc/trueuserdomains | cut -d “:” -f1)
do

HTTP_CODE=`curl -s -o /dev/null -w “%{http_code}” “$URLs”`

if [ “$HTTP_CODE” -ne 200 ];
then
echo “www.$URLs” >> $DIR$FILE
if [ -f “$DIR$URLs.txt” ];
then
FOUND=`grep $HTTP_CODE $DIR$URLs.txt`

if [ -z “$FOUND” ];
then
echo “HTTP-$HTTP_CODE:0” >> $DIR$URLs.txt
else
COUNT=`cat $DIR$URLs.txt | grep HTTP-$HTTP_CODE | cut -d “:” -f2`
NEWCOUNT=`expr $COUNT + 1`
sed -i “s/HTTP-$HTTP_CODE.*/HTTP-$HTTP_CODE:$NEWCOUNT/g” $DIR$URLs.txt
fi
else
echo “$URLs” >> $DIR$URLs.txt
echo “HTTP-$HTTP_CODE:0” >> $DIR$URLs.txt
fi

fi
done

z=`pgrep -f “add-signatures.sh” | grep -v grep`
OUT=`echo $?`

done

##Ends##

##Create Report##

DIR=/root/clam-scan-project/http-status-code/
FILE=url.txt
RESULT=/root/clam-scan-project/http-status-code/TABLE_STRUCTURE.log
RESULT1=/root/clam-scan-project/http-status-code/HTTP_CODES.log
echo “<center>” >> $RESULT
echo “<TABLE BORDER=4 ALIGN=center CELLPADDING=10 CELLSPACING=2>” >> $RESULT
echo “<TR>” >> $RESULT
echo “<TH WIDTH=”5%”>Domain Name</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>HTTP Status Code</TH>” >> $RESULT
echo “<TH WIDTH=”5%”>Headers</TH>” >> $RESULT
echo “</TR>” >> $RESULT

for DOMAINS in $(cat $DIR$FILE | sed ‘s/^[^\.]\+\.//’)
do
HTTP_STATUS_CODE=`grep HTTP-* $DIR$DOMAINS.txt | sed ‘/^$/d’`
SHORT_CODE=`echo $HTTP_STATUS_CODE | awk -F”[-:]” ‘{ print $2}’`

if [ “$SHORT_CODE” == 503 ] || [ “$SHORT_CODE” == 502 ] || [ “$SHORT_CODE” == 408 ] || [ “$SHORT_CODE” == 500 ] || [ “$SHORT_CODE” == 407 ];
then
HEADERS=`curl -sI “www.$DOMAINS”`

echo “<TR>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$DOMAINS</PRE></TD>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$HTTP_STATUS_CODE</PRE></TD>” >> $RESULT1
echo “<TD ALIGN=”center”><PRE>$HEADERS</PRE></TD>” >> $RESULT1
echo “</TR>” >> $RESULT1
fi
done

if [ -f “$RESULT1” ];
then
cat $RESULT1 | col -b >> $RESULT
fi

echo “</center>” >> $RESULT
echo “</TABLE>” >> $RESULT

##Ends##

Cheers!!!

Proxychains, Backtrack

Proxychains is a tool that allows you to TCP traffic through multiple proxy servers.

It supports,

  • HTTP
  • socks4
  • socks5

BackTrack comes with Proxychains installed by default but let’s go through the installation process.

On BackTrack

  1. apt-get install proxychains

On Cent OS: 5.x and 6.x

  1. wget http://prdownloads.sourceforge.net/proxychains/proxychains-3.1.tar.gz
  2. tar -xzvf proxychains-3.1.tar.gz
  3. cd proxychains-3.1
  4. ./configure
  5. make && make install

Configuration is same for both Debian(Ubuntu) and Cent OS (RHEL)

  1.  vi /etc/proxychains.conf
  2. Three types of chaining in the configuration file,
  • dynamic_chain: chains the proxies in the same order that you placed them in the configuration file skipping any dead proxy servers that it encounters

Example of dynamic chaining:

socks5 192.168355.68 1080 lamer secret

http 192.58.85.74 8080 justu hidden

if 192.58.85.74 is dead then dynamic_chain will skip 192.58.85.74.

  • strict_chain: similar to dynamic_chain except that it will break when it encounters a non responding proxy server.
  • random_chain: chains the proxy servers in a random way from your configuration file.

uncomment the chaining type that suits your needs

3. if random_chain is the chaining type selected  than chain_len must be setup

chain_len=2 says that length of random proxy chains must not be greater than 2

Example of chain_len =2,

root@bt:~# proxyresolv http://www.cmyip.com
|R-chain|-<>-222.188.10.1:1080-<>-127.0.0.1:9050-<–denied
|R-chain|-<>-222.188.10.1:1080-<>-189.23.2.44:1080-<–denied

we have below entries in /etc/proxychains.conf,

socks4 127.0.0.1 9050
socks4 189.23.2.44 1080
socks4 222.188.10.1 1080

so chain_len =2 will randomly select 222.188.10.1:1080 and will chain to 127.0.0.1:9050 and if denied then

it will chain to 189.23.2.44:1080.

Another example where chain_len=1

root@bt:~# proxyresolv http://www.cmyip.com
|R-chain|-<>-222.188.10.1:1080-<><>-4.2.2.2:53-<><>-OK
174.132.254.58

4. I have configured proxychains with TOR so below in /etc/proxychains.conf should be untouched.

socks4 127.0.0.1 9050

How to configure TOR and Polipo?

https://ustechnica.wordpress.com/2013/01/24/tor-and-polipo-backtrack-5/

Examples of proxychains:

root@bt:~# proxychains elinks –dump http://www.example.com.au/testme.php | grep -e “PHP Version” -e “System” -C 1 -m 2
|DNS-request| http://www.example.com.au
|S-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| http://www.example.com.au is 116.240.194.24
|S-chain|-<>-127.0.0.1:9050-<><>-116.20.19.24:80-<><>-OK

   PHP Version 5.2.17

   System            Linux server12.x234.com.au 2.6.18-194.11.4.el5 #1 SMP
                     Tue Sep 21 05:04:09 EDT 2010 x86_64
root@bt:~#

Server Logs:

173.254.216.66 – – [29/Jan/2013:22:53:01 +1100] “GET /testme.php HTTP/1.1” 200 64414 “-” “ELinks/0.12~pre5-2ubuntu1 (textmode; Ubuntu; Linux 3.2.6 i686; -)”
173.254.216.66 – – [29/Jan/2013:22:53:01 +1100] “GET /testme.php HTTP/1.1” 200 64414 “-” “ELinks/0.12~pre5-2ubuntu1 (textmode; Ubuntu; Linux 3.2.6 i686; -)”

Spoofed IP location:

US,California

City: San Francisco 🙂