1. Look for suspicious processes
Processes that are named to look like the common or legit process, unusual outbound connections, deleted processes are some of the characteristics of suspicious processes.
Linux commands to view process:
$ ps: ps command is used to list all the running processes and their PID. ps is designed for non-interactive use. It reads the process information from the virtual files in /proc file-system.
Although there are many options that can be used with ps command, but few useful options to remember would be:
-e: Will show processes for all the users
-f : Will display detailed information like username, pid, parent pid, process start time, related command.
-H: Will print process tree
$ top: List the processes and the resource usage like cpu and memory usage. Top command orders or sorts the processes by the amount of processor power they use. top allows you display of process statistics continuously until stopped vs. ps which gives you a single snapshot.
$ netstat: List out all network connections on a system.
Some common options that are used with netstat:
-p : PID/Program Name
-l: Listening sockets
-a: all sockets (listening & non-listening)
-n: numeric numbers for host and port
Easy to remember: netstat -plan
2. Look for suspicious files and directories
Directories that look like common system directories, hidden directories, directories with weird permissions, attributes, creation dates, tampered or missing audit log files, files that are not of the type they claim to be, binaries modified or in strange locations are common characteristics of suspicious directories and files.
Commonly targeted locations are:
/tmp
/var/tmp
/usr/bin
/dev/shm
Use the command, ls -al /var/tmp, to list the directories in /var/tmp
/tmp directory is present in every linux box and often used as scratch pad by attackers. Look at the directories in /tmp and other directories above to see if something looks weird.
For example, look for directories with immutable flags. Immutable flags are a persistence mechanism to retain the file even after reboot. Dir/files with immutable flags cannot be deleted wih normal system commands. It is not typical of linux to set such flags.
Use the command: ls -attr, to see file attributes
----i------ : immutable flag
High value and frequently targeted log files are:
/var/log/wtmp - all valid past logins
/var/log/lastlog - last login of each user
/var/log/btmp - all bad logins
/var/run/utmp - all current logins
/var/log/* - various other logs
Look for suspicious file attributes such as 0 byte size file, date/time identical for multiple files, system binaries that are different from
Use command, debsums -c (not installed by default), to see changed files.
To find files modified/created within last day, use command: find / -mtime -1
3. Identify persistence mechanisms
Check for cron jobs
Linux cron service enables you to schedule tasks to run at a specified time in the future. The cron service checks for files in the /var/spool/cron and /etc/cron.d directories. The individual user cron files are located in /var/spool/cron, and system services and applications generally add cron job files in the /etc/cron.d directory.
Use the following command to view cron files
$ ls -al /var/spool/cron
$ ls -al /etc/cron.*
* - Replace * with d, daily, hourly, weekly, monthly
The cron utility runs based on commands specified in a cron table (crontab). Each user, including root, can have a cron file. These files don't exist by default, but can be created in the /var/spool/cron directory using the crontab -e command that's also used to edit a cron file (see the script below).
To list the contents of cron table, run the command: crontab -l
Inspect /etc/init.d, /etc/rc*.d, /etc/rc.local directories
In Linux, init is the first process that starts after the machine boots and kernel loads. It decides how a system service should load, in what order, and whether it should start automatically. Init uses runlevel to find out what start services it needs to run. Runlevel is a state of system that defines what system services should run. Runlevels are defined by numbers.
0 – System halt i.e the system can be safely powered off with no activity.
1 – Single user mode.
2 – Multiple user mode with no NFS(network file system).
3 – Multiple user mode under the command line interface and not under the graphical user interface.
4 – User-definable.
5 – Multiple user mode under GUI (graphical user interface) and this is the standard runlevel for most of the LINUX based systems.
6 – Reboot which is used to restart the system.
When init enters a runlevel, it calls rc script with a numeric argument specifying the runlevel the system should go to. Rc then starts and stops services on system as required for that runlevel.
To avoid script duplication, the files in the /etc/rc*.d/ directories are symbolic links to script files located in the /etc/init.d/ directory. Usually, these directories should show common linux services like ssh, HTTP, apache that need to started when system boots. But custom scripts to carry out tasks can also be created in etc/init.d directory and then symbolic links to those scripts can be created in /etc/rc*.d directories.
For some tasks, however, we may not like to undergo the whole process of setting up complex init scripts. For example, some tasks need to be executed once the system boots and do not need to be run every time the system changes run levels. In such cases, /etc/rc.local file comes in handy. Hence, when looking for persistence, along with /etc/init.d and /etc/rc*.d, you should also inspect /etc/rc.local file.
4. Review user activity
SSH authentication logs - Check for suspicious success and failed ssh logins. SSH authentication logs are available in /var/log directory. In Redhat/CentOS, authentication messages are logged in /var/log/secure file. In Debian/Ubuntu, authentication messages are logged in /var/log/auth file.
wtmp, utmp, btmp logs - These log files are present in /var/log directory and show user login events.
utmp will give you complete picture of users logins at which terminals, logouts, system events and current status of the system, system boot time (used by uptime) etc.
wtmp gives historical data of utmp.
btmp records only failed login attempts.
But these are binary files and are not readable when cat, less & more linux commands are used on them. Last command is used to read their contents.
$ last -f /var/log/wtmp ### To open wtmp file and view its content
$ last -f /var/run/utmp ### To see still logged in users
$ last -f /var/log/btmp ### To view btmp file
Alternative commands to view above files:
$ who - who commands pull information about who’s logged in and what they’re doing from the /var/run/utmp file. If you want to see the list of users who are currently logged in, use who.
$ last - The last command provides how they logged in, when they logged in and when they logged out etc info on the screen. It pulls the logs from /var/log/wtmp file.
$ last b - command show history of logged sessions contained within /var/run/btmp.
.bash_history - Review the contents of .bash_history file to view the previously executed commands. .bash_history file is present in user home folder.
$ cat /home/username/.bash_history - To view the contents of bash history file
Since there may be more than 1 user logging to system, use below command to list history files for all users
$ find / -name .*history
Administrative privileges - Review user and group id to see if any user account has elevated or administrative privileges other than root.
Use command, $ id, to list uid and gid of a specific user
To view all the user accounts with either uid or gid sett to 0, check /etc/passwd file
$ grep ":0:" /etc/passwd
Note: /etc/passwd file does not actually contain the passwords for the user accounts. It holds the value: x in the field for password. But it contains some other useful information for every username like uid, gid, user’s home directory and user’s shell account. Passwords for each user are available in a separate file called shadow file, found in /etc but passwords are stored in encrypted form in /etc/shadow.
SSH authorized_keys - The authorized_keys file in .ssh directory specifies the SSH keys that can be used for logging into the user account for which the file is configured. Reviewing this file to identify and suspicious entry in the file.
To Find all ssh authorized_keys files, use the command:
$ find / -name authorized_keys
Quick Recap
Below is the list of linux commands that we discussed all above but are summarized here for a quick recap.
$ ps -efH ## To show the currently running process for all users with detailed options in a hierarchal order
$ top ## To show To list the processes and the resource usage like cpu and memory usage
$ netstat -plan ## To list out network connections for all sockets with PIDs and numeric numbers for host and port
$ ls -al /var/tmp ## To view the contents of tmp directory
$ find / -mtime -1 ## To find files modified/created within last day
$ ls -al /var/spool/cron ## To view the contents of cron directory
$ ls -al /etc/cron.* ## To view the contents of /etc/cron.d, cron.daily, and other cron directories
$ crontab -l ## To list the contents of cron table
$ ls -al /etc/rc.*d ## To list the contents of etc/rc[0-6].d directories
$ ls -al /etc/rc.local ## To list the contents of rc.local directory
$ who ## To see the currently logged in users
$ last ## To see all valid past logins
$ lastb ## To see all failed logins
$ history ## To view the previously executed commands
$ find / -name .bash_history ## To find .bash_history file for all users
$ find / -name authorized_keys ## To find all shh authorized_keys files
$ grep “:0:” /etc/passwd ## To view all the user accounts with either uid or gid set to 0
References