Tag Archive: Ebury


Off lately their was buzz all around Internet regarding SSH Ebury trojan, lots of misconceptions!!

We have around 400 linux servers with over 8000 websites hosted on it, still remember the day when there was a bombard of abuse reports from providers.

I was like WTF!!! all servers with two factor authentication as 2nd layer of security and it got rooted?

We started with reading through Ebury with couple of commands already out on Internet with all the checks and memory dumps, we were all clean.

Why the hell we are receiving abuse alerts? That too indirectly from CBL?

Okay, by far I have read about Ebury and what I have understood is that their are only two ways to get this installed,

  1. server rooted
  2. yum repos via automate updates

Later part was what I worried about, I started with the commands already known to 90% of security freaks but all the theories ended up with two commands which according to CBL and providers and most of the people out there is enough to assume that server is infected by Ebury, commands are shown below,

  • ipcs -m (looking for shared memory segments with permission 666)
  • size of libkeyutils file greater than 25KB, for some, greater than 20KB and so on.

Let’s starts with “ipcs -m”,

Sample Output:

root@bt [/proc/8890]# ipcs -mp

—— Shared Memory Creator/Last-op ——–
shmid owner cpid lpid
3997696 root 1675 1675
4358145 root 2038 2057
3964930 root 1668 1668
4390915 root 2038 2057
4423684 root 2038 2057
5898245 root 2866 19322
5931014 root 2866 19322

root@bt

Now if I run ps aux | grep lpid or cpid as shown below,

root@bt [/proc/2866]# ps aux | grep 1668 | grep -v grep
root 2648 0.0 0.0 21668 900 ? Ss Apr10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root@bt [/proc/2866]#

so next we, cd /proc/2866 you’ll see cmdline file,

root@bt [/proc/2866]# cat cmdline
/usr/local/apache/bin/httpd-kstart-DSSL

this will show the actual command and exe linked to,

lrwxrwxrwx   1 root root 0 Apr 16 17:40 exe -> /usr/local/apache/bin/httpd

Their will be another file called as “smaps”

open the file and you will see to which bin/lib it’s linked to,

3449406000-3449606000 —p 00006000 08:02 2860848                        /usr/lib64/libltdl.so.3.1.4

344a000000-344a031000 r-xp 00000000 08:02 2868115                        /usr/lib64/libidn.so.11.5.19

from this you can grep the actual linking of libraries, this is only applicable if it’s 666 permission but this holds true only if it’s governed by SSHD process.

Having 666 permission is common and shouldn’t be concluded as hacked or effected.

we had these on servers not governed by any process, doing nothing.

  • Let’s check for “size of libkeyutils”,

This varies on linux flavors to flavors and from repos to repos, on cpanel servers yum update and /scripts/upcp update will always differ in size.

Same applies for amazon, “yum reinstall –enable remi libkeyutils” and “yum reinstall libkeyutils” which will take amazon repos will differ in size compared to remis.

In-case of any abuse reports, run through a series of commands and do not reload OS, that’s not the option.

By far the only reliable I found was starce command,

strace-sshd.sh  (Execute this script first)
 
STRACE=`ps aux | grep ssh[d] | awk ‘{print “-p ” $2}’`
echo $STRACE
strace -o out -f -s 5000 $STRACE
 
syscallparse.sh (from second window after executing ctrl-c on first terminal)
 
DIR=/root/syscalls
 
if [ ! -d “$DIR” ];
then
        mkdir $DIR
fi
 
for x in `awk ‘{print $2}’ /root/out | grep [a-z] | sed s/\(.*//g | sort | uniq`
do
grep -n “^[[:digit:]]\+[[:space:]]\+$x(” /root/out >> /root/syscalls/$x
done

Under /root/syscalls directory, you’ll find a file called as “connect”, this file will say to which outbound port it is trying to connect.

Sample:

479:32560 connect(3, {sa_family=AF_INET6, sin6_port=htons(22), inet_pton(AF_INET6, “::”, &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
483:32560 connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr(“0.0.0.0”)}, 16) = 0
488:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
492:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
589:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
593:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
633:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
652:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
671:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0

So what happens here?

the connection attempts to /var/run/nscd/socket, this means ssh first tried to connect to NSCD – the Name Service Cache Daemon — which is fine but the IP it attempts to connect should be one of the IPs from /etc/resolv.conf which is name server IP for name lookups.

IP other than name-server ip’s  then you should be  with WTF!!! face 😮

DNS is port 53 so “sin_port=htons(53)”, it doesn’t have “sendto” in above sample so nothing was being sent to port 53.

Never the less, I had prepared the script which checks almost everything apart from,

  • libkeyutils size
  • signatures of openssh rpms

which I will add this week, execute it from /root/, you’ll need hashes.txt file to which is at the bottom.

###Script Starts here###

#!/bin/bash

LIB64=/lib64/libkeyutils.so.1.9
LIB64_1=/lib64/libkeyutils-1.2.so.2
LIB32=/lib/libkeyutils.so.1.9
LIB32_1=/lib/libkeyutils-1.2.so.2
red=’\e[0;31m’
NC=’\e[0m’
green=’\e[0;32m’

if [ -f $LIB64 ]; then
echo The server is compromised, $LIB64 found
exit 0
fi

if [ -f $LIB64_1 ]; then
echo The server is compromised, $LIB64_1 found
exit 0
fi
if [ -f $LIB32 ]; then
echo The server is compromised, $LIB32 found
exit 0
fi

if [ -f $LIB32_1 ]; then
echo The server is compromised, $LIB32_1 found
exit 0
fi

echo -e “\t\t\t$HOSTNAME”
echo ” “
echo -e “${green}Cannot find compromised library: Clean${NC}”
#exit 1
SHMEM=`ipcs -m | grep 666`

if [ -z “$SHMEM” ];
then
echo -e “${green}Checking Shared Memory Segments with 666: Clean${NC}”
else
echo -e “${red}Checking Shared Memory Segments with 666: Infected${NC}”
ipcs -m
fi

ILLOPTION=`ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo “System clean” || echo “System infected”`
CHK=`echo $ILLOPTION | grep -ie infected`

if [ -z “$CHK” ];
then
echo -e “${green}Checking for SSH illegal option: $ILLOPTION ${NC}”
else
echo -e “${red}Checking for SSH illegal option: $ILLOPTION ${NC}”
fi
echo ” “
echo -e “\t\t\tChecking for Trojanized sshd, ssh, ssh-add:”
echo ” “
SHASUM=`which sha1sum`
SSH=`which ssh`
SSHD=`which sshd`
SSHADD=`which ssh-add`
SHASSH=`$SHASUM $SSH | awk ‘{ print $1 }’`
SHASSHD=`$SHASUM $SSHD | awk ‘{ print $1 }’`
SHASSHADD=`$SHASUM $SSHADD | awk ‘{ print $1 }’`
HASHSSH=`grep $SHASSH -F /root/hashes.txt`
HASHSSHD=`grep $SHASSHD -F /root/hashes.txt`
HASHSSHADD=`grep $SHASSHADD -F /root/hashes.txt`

if [ -z “$HASHSSH” ];
then
echo -e “${green}Hashes for $SSH not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSH matching with Linux/Ebury hashes: $HASHSSH,${NC}”
fi

if [ -z “$HASHSSHD” ];
then
echo -e “${green}Hashes for $SSHD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHD matching with Linux/Ebury hashes: $HASHSSHD,${NC}”
fi

if [ -z “$HASHSSHADD” ];
then
echo -e “${green}Hashes for $SSHADD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHADD matching with Linux/Ebury hashes: $HASHSSHADD,${NC}”
fi

echo ” “
echo -e “\t\t\tChecking for libkeyutils:”
echo ” “
for UTILS in `locate libkey | grep -v home`;
do
CHKHASH=`which sha1sum`
RES=`$CHKHASH $UTILS | grep -ie 98cdbf1e0d202f5948552cebaa9f0315b7a3731d -ie 4d12f98fd49e58e0635c6adce292cc56a31da2a2 -ie 0daa51519797cefedd52864be0da7fa1a93ca30b -ie 7314eadbdf18da424c4d8510afcc9fe5fcb56b39 -ie 575bb6e681b5f1e1b774fee0fa5c4fe538308814 -ie fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b -ie c4c28d0372aee7001c44a1659097c948df91985d -ie 267d010201c9ff53f8dc3fb0a48145dc49f9de1e -ie 471ee431030332dd636b8af24a428556ee72df37 -ie 58f185c3fe9ce0fb7cac9e433fb881effad31421 -ie 09c8af3be4327c83d4a7124a678bbc81e12a1de4 -ie 2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b -ie 39ec9e03edb25f1c316822605fe4df7a7b1ad94a -ie 3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf -ie 74aa801c89d07fa5a9692f8b41cb8dd07e77e407 -ie 7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 -ie 899b860ef9d23095edb6b941866ea841d64d1b26 -ie 8daad0a043237c5e3c760133754528b97efad459 -ie 8f75993437c7983ac35759fe9c5245295d411d35 -ie 9bb6a2157c6a3df16c8d2ad107f957153cba4236 -ie a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 -ie adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f -ie b8508fc2090ddee19a19659ea794f60f0c2c23ff -ie bbce62fb1fc8bbed9b40cfb998822c266b95d148 -ie bf1466936e3bd882b47210c12bf06cb63f7624c0 -ie e14da493d70ea4dd43e772117a61f9dbcff2c41c -ie f1ada064941f77929c49c8d773cbad9c15eba322 -ie 9e2af0910676ec2d92a1cad1ab89029bc036f599 -ie 5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 -ie d552cbadee27423772a37c59cb830703b757f35e -ie 1a9aff1c382a3b139b33eeccae954c2d65b64b90 -ie 2e571993e30742ee04500fbe4a40ee1b14fa64d7 -ie e2a204636bda486c43d7929880eba6cb8e9de068`

if [ -z “$CHKHASH” ];
then
echo -e “${red}Hashes matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
else
echo -e “${green}Hashes not matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
fi

done

###hashes.txt###

98cdbf1e0d202f5948552cebaa9f0315b7a3731d Linux/Ebury . Version 0.4.4 . sshd
4d12f98fd49e58e0635c6adce292cc56a31da2a2 Linux/Ebury . Version 0.4.4 . sshd
0daa51519797cefedd52864be0da7fa1a93ca30b Linux/Ebury . Version 0.8.0 . sshd
7314eadbdf18da424c4d8510afcc9fe5fcb56b39 Linux/Ebury . Version 0.8.0 . sshd
575bb6e681b5f1e1b774fee0fa5c4fe538308814 Linux/Ebury . Version 0.8.0 . ssh-add
fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b Linux/Ebury . Version 0.8.0 . ssh
c4c28d0372aee7001c44a1659097c948df91985d Linux/Ebury . Version 0.8.0 . ssh
267d010201c9ff53f8dc3fb0a48145dc49f9de1e Linux/Ebury . Version 1.1.0 . libkeyutils.so
471ee431030332dd636b8af24a428556ee72df37 Linux/Ebury . Version 1.2.1 . libkeyutils.so
58f185c3fe9ce0fb7cac9e433fb881effad31421 Linux/Ebury . Version 1.3.1 . libkeyutils.so
09c8af3be4327c83d4a7124a678bbc81e12a1de4 Linux/Ebury . Version 1.3.2 . libkeyutils.so
2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b Linux/Ebury . Version 1.3.2 . libkeyutils.so
39ec9e03edb25f1c316822605fe4df7a7b1ad94a Linux/Ebury . Version 1.3.2 . libkeyutils.so
3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf Linux/Ebury . Version 1.3.2 . libkeyutils.so
74aa801c89d07fa5a9692f8b41cb8dd07e77e407 Linux/Ebury . Version 1.3.2 . libkeyutils.so
7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 Linux/Ebury . Version 1.3.2 . libkeyutils.so
899b860ef9d23095edb6b941866ea841d64d1b26 Linux/Ebury . Version 1.3.2 . libkeyutils.so
8daad0a043237c5e3c760133754528b97efad459 Linux/Ebury . Version 1.3.2a . libkeyutils.so
8f75993437c7983ac35759fe9c5245295d411d35 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9bb6a2157c6a3df16c8d2ad107f957153cba4236 Linux/Ebury . Version 1.3.2 . libkeyutils.so
a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 Linux/Ebury . Version 1.3.2 . libkeyutils.so
adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f Linux/Ebury . Version 1.3.2 . libkeyutils.so
b8508fc2090ddee19a19659ea794f60f0c2c23ff Linux/Ebury . Version 1.3.2 . libkeyutils.so
bbce62fb1fc8bbed9b40cfb998822c266b95d148 Linux/Ebury . Version 1.3.2 . libkeyutils.so
bf1466936e3bd882b47210c12bf06cb63f7624c0 Linux/Ebury . Version 1.3.2 . libkeyutils.so
e14da493d70ea4dd43e772117a61f9dbcff2c41c Linux/Ebury . Version 1.3.2 . libkeyutils.so
f1ada064941f77929c49c8d773cbad9c15eba322 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9e2af0910676ec2d92a1cad1ab89029bc036f599 Linux/Ebury . Version 1.3.3b . libkeyutils.so
5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 Linux/Ebury . Version 1.3.3 . libkeyutils.so
d552cbadee27423772a37c59cb830703b757f35e Linux/Ebury . Version 1.3.3 . libkeyutils.so
1a9aff1c382a3b139b33eeccae954c2d65b64b90 Linux/Ebury . Version 1.3.4b1 . libkeyutils.so
2e571993e30742ee04500fbe4a40ee1b14fa64d7 Linux/Ebury . Version 1.3.4b2 . libkeyutils.so
e2a204636bda486c43d7929880eba6cb8e9de068 Linux/Ebury . Version 1.3.5 . libkeyutils.so

Sample Output:

Ebury

Never Ever rely on what provider says, believe in what you are doing and if you are 100% sure than just reply logically and up-to the mark,

Our Reply:

our

Provider’s Reply:

provider

Trust me on this ,we didn’t reload OS and we just end up killing shared segment with perm 666 not governed by any process using ipcrm command.

I end up creating this signature from snort to iptables,

-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j LOG –log-prefix “Ebury SSH Rootkit:” –log-level 7
-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j DROP

Final note for “SSH -G” option, picture is worth a thousand words

ssh

 

Ref: http://roumenpetrov.info/openssh/

Cheers!!!

IPtables is not an ordinary firewall unless and until not used properly.

Their are cases where security analysts do not have an option to play with hardware firewall or sometime no access to DC, mainly cpanel servers.

In this case, the only option is customized solution, one of the kind is shown below,

iptables

 

with iptables string matching, you can achieve the highest security possible with log scanning if anything bypasses firewall.

This is mainly IPS/IDS dependent upon the signature matching.

  1. Create a chain, say “woot”
  2. After all the input rules, goto woot chain for additional checks.
  3. then specify malicious signatures to detect different types of attacks.
  4. If matches then first log the packet and then drop.

your iptables file should be something like below,

:INPUT ACCEPT [2404:336622]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2359:257349]
:LOGGING – [0:0]
:w00t – [0:0]

-A INPUT -i lo -j ACCEPT
-A INPUT -s 127.0.0.1 -j ACCEPT
-A INPUT -s x.x.x.x -p tcp -m tcp –dport 22 -j ACCEPT
-A INPUT -s x.x.x.x -p tcp -m tcp –dport 22 -j ACCEPT

-A INPUT -p tcp -j w00t

-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j LOG –log-prefix “Ebury SSH Rootkit:” –log-level 7
-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j DROP

-A w00t -p tcp -m string –string “w00tw00t.at.ISC.SANS” –algo bm –to 65535 -j LOG –log-prefix “w00tw00t detected:” –log-level 7
-A w00t -p tcp -m string –string “w00tw00t.at.ISC.SANS” –algo bm –to 65535 -j DROP

The above will first log packets (chain woot), and then drop it.

Log:

Apr 14 20:19:29 darkwizz kernel: w00tw00t detected:IN=eth0 OUT= MAC=00:30:48:8f:49:c9:01:24:00:08:00 SRC=x.x.x.x DST=96.31.85.162 LEN=86 TOS=0x00 PREC=0x00 TTL=115 ID=10415 DF PROTO=TCP SPT=59420 DPT=80 WINDOW=256 RES=0x00 ACK PSH FIN URGP=0

Test Scenarios:

1. open a command line and type in,

nc -l 5501

2. open another session of that server

telnet server-ip 5501

from telnet command line,

test

3. first session where you typed in nc command, you’ll see “test” appearing on that session

4. from second terminal (point 2), type in  “w00tw00t”

5. string won’t appear on first session and then consequent packets will be dropped from that server.

6. check the logs and you will see.

Where to find the log location for iptables?

cat /etc/syslog.cong

kern.*                        /var/log/firewall.log

Back to our firewall rules,

their is outbound traffic from server getting dropped if it matches a hex-string, “120b01000001” which is a string for Ebury Trojan.

you can easily convert any snort rules into IP table rules.

Some Rules according to types of attacks,

1. SQL Injection Attacks: 

The “%27or%271%27%3D%271” was an encoding. When decoding the URL, it would result as a message ‘ or ‘1’=’1.

-m string –string  “%27+or+%271%27%3d%271”

“%27+or+1%3d1”

“%27+or+%271%27%3d%271”

“%27)+or+%27%27%3d%27”

“%27)+or+1%3d1”

“%27)+or+%271%27%3d%271”

“%27)+or+(%27%27%3d%27”

“%27)+or+(1%3d1”

“%27)+or+(%271%27%3d%271”

2. Buffer Overflow Exploits

-m string –string “AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”

Recently I posted “http://ustechnica.com/category/security/wordpress-arbitrary-file-download-and-file-deletion-exploit/”

and was able to download /etc/passwd file, you can even match this string and drop the request.

3. Arbitrary File download

-m string –string “/etc/passwd”

4. Cross Site Scripting

-m string –string %3C%73%63%72%69%70%74%3E”

-m string –string “<script>”

you just need to convert all snort rules to iptables, below is the good reference can be used,

Ref:

ftp://ftp.itb.ac.id/pub/ISO-IMAGES/linux/filenya-putu-shinoda/bukulinux/LinuxFirewall-Attack%20Detection.pdf

http://www.bandwidthco.com/sf_whitepapers/firewalls/IPTables%20Linux%20Firewall%20with%20Packet%20String-matching%20support.pdf