Tag Archive: SSH


Off lately their was buzz all around Internet regarding SSH Ebury trojan, lots of misconceptions!!

We have around 400 linux servers with over 8000 websites hosted on it, still remember the day when there was a bombard of abuse reports from providers.

I was like WTF!!! all servers with two factor authentication as 2nd layer of security and it got rooted?

We started with reading through Ebury with couple of commands already out on Internet with all the checks and memory dumps, we were all clean.

Why the hell we are receiving abuse alerts? That too indirectly from CBL?

Okay, by far I have read about Ebury and what I have understood is that their are only two ways to get this installed,

  1. server rooted
  2. yum repos via automate updates

Later part was what I worried about, I started with the commands already known to 90% of security freaks but all the theories ended up with two commands which according to CBL and providers and most of the people out there is enough to assume that server is infected by Ebury, commands are shown below,

  • ipcs -m (looking for shared memory segments with permission 666)
  • size of libkeyutils file greater than 25KB, for some, greater than 20KB and so on.

Let’s starts with “ipcs -m”,

Sample Output:

root@bt [/proc/8890]# ipcs -mp

—— Shared Memory Creator/Last-op ——–
shmid owner cpid lpid
3997696 root 1675 1675
4358145 root 2038 2057
3964930 root 1668 1668
4390915 root 2038 2057
4423684 root 2038 2057
5898245 root 2866 19322
5931014 root 2866 19322

root@bt

Now if I run ps aux | grep lpid or cpid as shown below,

root@bt [/proc/2866]# ps aux | grep 1668 | grep -v grep
root 2648 0.0 0.0 21668 900 ? Ss Apr10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root@bt [/proc/2866]#

so next we, cd /proc/2866 you’ll see cmdline file,

root@bt [/proc/2866]# cat cmdline
/usr/local/apache/bin/httpd-kstart-DSSL

this will show the actual command and exe linked to,

lrwxrwxrwx   1 root root 0 Apr 16 17:40 exe -> /usr/local/apache/bin/httpd

Their will be another file called as “smaps”

open the file and you will see to which bin/lib it’s linked to,

3449406000-3449606000 —p 00006000 08:02 2860848                        /usr/lib64/libltdl.so.3.1.4

344a000000-344a031000 r-xp 00000000 08:02 2868115                        /usr/lib64/libidn.so.11.5.19

from this you can grep the actual linking of libraries, this is only applicable if it’s 666 permission but this holds true only if it’s governed by SSHD process.

Having 666 permission is common and shouldn’t be concluded as hacked or effected.

we had these on servers not governed by any process, doing nothing.

  • Let’s check for “size of libkeyutils”,

This varies on linux flavors to flavors and from repos to repos, on cpanel servers yum update and /scripts/upcp update will always differ in size.

Same applies for amazon, “yum reinstall –enable remi libkeyutils” and “yum reinstall libkeyutils” which will take amazon repos will differ in size compared to remis.

In-case of any abuse reports, run through a series of commands and do not reload OS, that’s not the option.

By far the only reliable I found was starce command,

strace-sshd.sh  (Execute this script first)
 
STRACE=`ps aux | grep ssh[d] | awk ‘{print “-p ” $2}’`
echo $STRACE
strace -o out -f -s 5000 $STRACE
 
syscallparse.sh (from second window after executing ctrl-c on first terminal)
 
DIR=/root/syscalls
 
if [ ! -d “$DIR” ];
then
        mkdir $DIR
fi
 
for x in `awk ‘{print $2}’ /root/out | grep [a-z] | sed s/\(.*//g | sort | uniq`
do
grep -n “^[[:digit:]]\+[[:space:]]\+$x(” /root/out >> /root/syscalls/$x
done

Under /root/syscalls directory, you’ll find a file called as “connect”, this file will say to which outbound port it is trying to connect.

Sample:

479:32560 connect(3, {sa_family=AF_INET6, sin6_port=htons(22), inet_pton(AF_INET6, “::”, &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
483:32560 connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr(“0.0.0.0”)}, 16) = 0
488:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
492:32560 connect(3, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
589:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
593:32560 connect(6, {sa_family=AF_FILE, path=”/var/run/nscd/socket”…}, 110) = -1 ENOENT (No such file or directory)
633:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
652:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0
671:32560 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“xx.xx.xx.xx”)}, 28) = 0

So what happens here?

the connection attempts to /var/run/nscd/socket, this means ssh first tried to connect to NSCD – the Name Service Cache Daemon — which is fine but the IP it attempts to connect should be one of the IPs from /etc/resolv.conf which is name server IP for name lookups.

IP other than name-server ip’s  then you should be  with WTF!!! face 😮

DNS is port 53 so “sin_port=htons(53)”, it doesn’t have “sendto” in above sample so nothing was being sent to port 53.

Never the less, I had prepared the script which checks almost everything apart from,

  • libkeyutils size
  • signatures of openssh rpms

which I will add this week, execute it from /root/, you’ll need hashes.txt file to which is at the bottom.

###Script Starts here###

#!/bin/bash

LIB64=/lib64/libkeyutils.so.1.9
LIB64_1=/lib64/libkeyutils-1.2.so.2
LIB32=/lib/libkeyutils.so.1.9
LIB32_1=/lib/libkeyutils-1.2.so.2
red=’\e[0;31m’
NC=’\e[0m’
green=’\e[0;32m’

if [ -f $LIB64 ]; then
echo The server is compromised, $LIB64 found
exit 0
fi

if [ -f $LIB64_1 ]; then
echo The server is compromised, $LIB64_1 found
exit 0
fi
if [ -f $LIB32 ]; then
echo The server is compromised, $LIB32 found
exit 0
fi

if [ -f $LIB32_1 ]; then
echo The server is compromised, $LIB32_1 found
exit 0
fi

echo -e “\t\t\t$HOSTNAME”
echo ” “
echo -e “${green}Cannot find compromised library: Clean${NC}”
#exit 1
SHMEM=`ipcs -m | grep 666`

if [ -z “$SHMEM” ];
then
echo -e “${green}Checking Shared Memory Segments with 666: Clean${NC}”
else
echo -e “${red}Checking Shared Memory Segments with 666: Infected${NC}”
ipcs -m
fi

ILLOPTION=`ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo “System clean” || echo “System infected”`
CHK=`echo $ILLOPTION | grep -ie infected`

if [ -z “$CHK” ];
then
echo -e “${green}Checking for SSH illegal option: $ILLOPTION ${NC}”
else
echo -e “${red}Checking for SSH illegal option: $ILLOPTION ${NC}”
fi
echo ” “
echo -e “\t\t\tChecking for Trojanized sshd, ssh, ssh-add:”
echo ” “
SHASUM=`which sha1sum`
SSH=`which ssh`
SSHD=`which sshd`
SSHADD=`which ssh-add`
SHASSH=`$SHASUM $SSH | awk ‘{ print $1 }’`
SHASSHD=`$SHASUM $SSHD | awk ‘{ print $1 }’`
SHASSHADD=`$SHASUM $SSHADD | awk ‘{ print $1 }’`
HASHSSH=`grep $SHASSH -F /root/hashes.txt`
HASHSSHD=`grep $SHASSHD -F /root/hashes.txt`
HASHSSHADD=`grep $SHASSHADD -F /root/hashes.txt`

if [ -z “$HASHSSH” ];
then
echo -e “${green}Hashes for $SSH not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSH matching with Linux/Ebury hashes: $HASHSSH,${NC}”
fi

if [ -z “$HASHSSHD” ];
then
echo -e “${green}Hashes for $SSHD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHD matching with Linux/Ebury hashes: $HASHSSHD,${NC}”
fi

if [ -z “$HASHSSHADD” ];
then
echo -e “${green}Hashes for $SSHADD not matching with Linux/Ebury hashes${NC}”
else
echo -e “${red}Hashes for $SSHADD matching with Linux/Ebury hashes: $HASHSSHADD,${NC}”
fi

echo ” “
echo -e “\t\t\tChecking for libkeyutils:”
echo ” “
for UTILS in `locate libkey | grep -v home`;
do
CHKHASH=`which sha1sum`
RES=`$CHKHASH $UTILS | grep -ie 98cdbf1e0d202f5948552cebaa9f0315b7a3731d -ie 4d12f98fd49e58e0635c6adce292cc56a31da2a2 -ie 0daa51519797cefedd52864be0da7fa1a93ca30b -ie 7314eadbdf18da424c4d8510afcc9fe5fcb56b39 -ie 575bb6e681b5f1e1b774fee0fa5c4fe538308814 -ie fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b -ie c4c28d0372aee7001c44a1659097c948df91985d -ie 267d010201c9ff53f8dc3fb0a48145dc49f9de1e -ie 471ee431030332dd636b8af24a428556ee72df37 -ie 58f185c3fe9ce0fb7cac9e433fb881effad31421 -ie 09c8af3be4327c83d4a7124a678bbc81e12a1de4 -ie 2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b -ie 39ec9e03edb25f1c316822605fe4df7a7b1ad94a -ie 3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf -ie 74aa801c89d07fa5a9692f8b41cb8dd07e77e407 -ie 7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 -ie 899b860ef9d23095edb6b941866ea841d64d1b26 -ie 8daad0a043237c5e3c760133754528b97efad459 -ie 8f75993437c7983ac35759fe9c5245295d411d35 -ie 9bb6a2157c6a3df16c8d2ad107f957153cba4236 -ie a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 -ie adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f -ie b8508fc2090ddee19a19659ea794f60f0c2c23ff -ie bbce62fb1fc8bbed9b40cfb998822c266b95d148 -ie bf1466936e3bd882b47210c12bf06cb63f7624c0 -ie e14da493d70ea4dd43e772117a61f9dbcff2c41c -ie f1ada064941f77929c49c8d773cbad9c15eba322 -ie 9e2af0910676ec2d92a1cad1ab89029bc036f599 -ie 5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 -ie d552cbadee27423772a37c59cb830703b757f35e -ie 1a9aff1c382a3b139b33eeccae954c2d65b64b90 -ie 2e571993e30742ee04500fbe4a40ee1b14fa64d7 -ie e2a204636bda486c43d7929880eba6cb8e9de068`

if [ -z “$CHKHASH” ];
then
echo -e “${red}Hashes matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
else
echo -e “${green}Hashes not matching with Ebury${NC}”
ls -lh $UTILS
echo ” “
fi

done

###hashes.txt###

98cdbf1e0d202f5948552cebaa9f0315b7a3731d Linux/Ebury . Version 0.4.4 . sshd
4d12f98fd49e58e0635c6adce292cc56a31da2a2 Linux/Ebury . Version 0.4.4 . sshd
0daa51519797cefedd52864be0da7fa1a93ca30b Linux/Ebury . Version 0.8.0 . sshd
7314eadbdf18da424c4d8510afcc9fe5fcb56b39 Linux/Ebury . Version 0.8.0 . sshd
575bb6e681b5f1e1b774fee0fa5c4fe538308814 Linux/Ebury . Version 0.8.0 . ssh-add
fa6707c7ef12ce9b0f7152ca300ebb2bc026ce0b Linux/Ebury . Version 0.8.0 . ssh
c4c28d0372aee7001c44a1659097c948df91985d Linux/Ebury . Version 0.8.0 . ssh
267d010201c9ff53f8dc3fb0a48145dc49f9de1e Linux/Ebury . Version 1.1.0 . libkeyutils.so
471ee431030332dd636b8af24a428556ee72df37 Linux/Ebury . Version 1.2.1 . libkeyutils.so
58f185c3fe9ce0fb7cac9e433fb881effad31421 Linux/Ebury . Version 1.3.1 . libkeyutils.so
09c8af3be4327c83d4a7124a678bbc81e12a1de4 Linux/Ebury . Version 1.3.2 . libkeyutils.so
2fc132440bafdbc72f4d4e8dcb2563cc0a6e096b Linux/Ebury . Version 1.3.2 . libkeyutils.so
39ec9e03edb25f1c316822605fe4df7a7b1ad94a Linux/Ebury . Version 1.3.2 . libkeyutils.so
3c5ec2ab2c34ab57cba69bb2dee70c980f26b1bf Linux/Ebury . Version 1.3.2 . libkeyutils.so
74aa801c89d07fa5a9692f8b41cb8dd07e77e407 Linux/Ebury . Version 1.3.2 . libkeyutils.so
7adb38bf14e6bf0d5b24fa3f3c9abed78c061ad1 Linux/Ebury . Version 1.3.2 . libkeyutils.so
899b860ef9d23095edb6b941866ea841d64d1b26 Linux/Ebury . Version 1.3.2 . libkeyutils.so
8daad0a043237c5e3c760133754528b97efad459 Linux/Ebury . Version 1.3.2a . libkeyutils.so
8f75993437c7983ac35759fe9c5245295d411d35 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9bb6a2157c6a3df16c8d2ad107f957153cba4236 Linux/Ebury . Version 1.3.2 . libkeyutils.so
a7b8d06e2c0124e6a0f9021c911b36166a8b62c5 Linux/Ebury . Version 1.3.2 . libkeyutils.so
adfcd3e591330b8d84ab2ab1f7814d36e7b7e89f Linux/Ebury . Version 1.3.2 . libkeyutils.so
b8508fc2090ddee19a19659ea794f60f0c2c23ff Linux/Ebury . Version 1.3.2 . libkeyutils.so
bbce62fb1fc8bbed9b40cfb998822c266b95d148 Linux/Ebury . Version 1.3.2 . libkeyutils.so
bf1466936e3bd882b47210c12bf06cb63f7624c0 Linux/Ebury . Version 1.3.2 . libkeyutils.so
e14da493d70ea4dd43e772117a61f9dbcff2c41c Linux/Ebury . Version 1.3.2 . libkeyutils.so
f1ada064941f77929c49c8d773cbad9c15eba322 Linux/Ebury . Version 1.3.2 . libkeyutils.so
9e2af0910676ec2d92a1cad1ab89029bc036f599 Linux/Ebury . Version 1.3.3b . libkeyutils.so
5d3ec6c11c6b5e241df1cc19aa16d50652d6fac0 Linux/Ebury . Version 1.3.3 . libkeyutils.so
d552cbadee27423772a37c59cb830703b757f35e Linux/Ebury . Version 1.3.3 . libkeyutils.so
1a9aff1c382a3b139b33eeccae954c2d65b64b90 Linux/Ebury . Version 1.3.4b1 . libkeyutils.so
2e571993e30742ee04500fbe4a40ee1b14fa64d7 Linux/Ebury . Version 1.3.4b2 . libkeyutils.so
e2a204636bda486c43d7929880eba6cb8e9de068 Linux/Ebury . Version 1.3.5 . libkeyutils.so

Sample Output:

Ebury

Never Ever rely on what provider says, believe in what you are doing and if you are 100% sure than just reply logically and up-to the mark,

Our Reply:

our

Provider’s Reply:

provider

Trust me on this ,we didn’t reload OS and we just end up killing shared segment with perm 666 not governed by any process using ipcrm command.

I end up creating this signature from snort to iptables,

-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j LOG –log-prefix “Ebury SSH Rootkit:” –log-level 7
-A OUTPUT -p tcp -m tcp –dport 53 -m string –hex-string “|120b01000001|” –algo bm –to 65535 -j DROP

Final note for “SSH -G” option, picture is worth a thousand words

ssh

 

Ref: http://roumenpetrov.info/openssh/

Cheers!!!

Sometimes we need to allow user to access SSH in a specific time interval.

For Example : Allow Mike to access SSH service between 1:00 AM to 5:00 AM.

We can achieve this using pam_time module.The pam_time module is an account module type. No arguments are passed directly to the module, but instead all configuration takes place within /etc/security/time.conf.

The time.conf operates based on rules, and each rule uses the following syntax:

services;ttys;users;times

Steps to be followed,

  • SSH to the server and vi /etc/pam.d/sshd

cat /etc/pam.d/sshd
#%PAM-1.0
auth include system-auth
account required pam_time.so
account required pam_nologin.so
account include system-auth
password include system-auth
session optional pam_keyinit.so force revoke
session include system-auth

auth required pam_shells.so

The one in BOLD must be added.

  • vi /etc/pam.d/system-auth

add “account     required      pam_time.so” 

  • vi /etc/security/time.conf

sshd;*;mike;Wk0100-0500

  • Restart sshd service

**Available abbreviates for the days of the week

Mo : Monday
Tu : Tuesday
We : Wednesday
Th : Thursday
Fr : Friday
Sa : Saturday
Su : Sunday
Wd : Sa/Su
wk : Mo/Tu/We/Th/Fr
Al : All Days