Incident Handling with Splunk (TryHackMe)
Incident Handling refers to the process of managing and responding to cybersecurity incidents, such as data breaches, malware infections, or unauthorized access attempts. The goal is to minimize the impact of the incident, recover normal operations, and prevent future occurrences.
Before we begin we need to understand the difference between a security event and a security incident.
Security event — This is any observable occurrence in a system or network. Some security events can be false positives or true positives.
Security incident — This is any event that has a negative consequence on the system/network. For example, system crashes, unauthorized use of system privileges, execution of malware, and unauthorized access to sensitive data.
Learning Objective
- Learn how to leverage OSINT sites during an investigation
- How to map Attacker’s activities to Cyber Kill Chain Phases
- How to utilize effective Splunk searches to investigate logs
- Understand the importance of host-centric and network-centric log sources
Task 2: Incident Handling Life Cycle
The NIST Incident Response Life Cycle, as outlined in the NIST Special Publication 800–61 Revision 2, provides a structured framework for managing and responding to cybersecurity incidents. Incident response aims to limit the attack's impact, assess the damage caused, and implement recovery procedures. The Incident Handling process has 4 phases:
- Preparation — This phase covers the readiness of an organization against an attack. That means documenting the requirements, defining the policies, and incorporating the security controls to monitor like EDR / SIEM / IDS / IPS, etc. It also includes creating and training an incident response team.
- Detection and Analysis — This phase covers getting alerts from security controls like SIEM/EDR and investigating the alert to find the root cause. This phase also covers hunting for the unknown threat within the organization. NIST provides several recommendations for making incident analysis easier and more effective, for example, profile networks and systems, understand normal behavior, run packet sniffers to collect additional data, keep all host clocks synchronized, etc.
- Containment, Eradication, and Recovery — This phase covers the actions needed to prevent the incident from spreading and secure the network. It involves steps to prevent an attack from spreading into the network, isolating the infected host, clearing the network from the infection traces, and gaining control from the attack to restore normal operations.
- Post-incident Activity/ Lesson Learnt — This phase includes identifying the loopholes in the organization’s security posture that led to an intrusion and improving them so that the attack does not recur. The steps involve identifying weaknesses that led to the attack, adding detection rules so that a similar breach does not happen again, and most importantly, training the staff if required.
Task 3: Incident Handling Scenario
A Big corporate organization Wayne Enterprises has recently faced a cyber-attack where the attackers broke into their network, found their way to their web server, and have successfully defaced their website http://www.imreallynotbatman.com. Their website is now showing the trademark of the attackers with the message YOUR SITE HAS BEEN DEFACED.
The good thing is that they have Splunk already in place, so we have all the event logs related to the attacker’s activities captured. We need to explore the records and find how the attack got into their network and what actions they performed. This investigation comes under the Detection and Analysis phase.
Interesting log Sources
Some of the interesting log sources that will help us in our Investigation are:
Note: All the event logs that we are going to investigate are present in index=botsv1
Our task as Security Analysts will be to investigate this cyber attack and map the attacker’s activities into all 7 phases of the Cyber Kill Chain. Connect to the room machine so we can start our investigation.
Task 4: Reconnaissance Phase
Reconnaissance is simply discovering and collecting information about a target. It could be knowledge about the system in use, the web application, employees or location, etc.
We will start our analysis by examining any reconnaissance attempt against the webserver imreallynotbatman.com. If we look at the available log sources, we will find some log sources covering the network traffic, which means all the inbound communication towards our web server will be logged into the log source that contains the web traffic.
We begin by searching for the domain in the search head and see which log source includes the traces of our domain.
Search Query: index=botsv1 imreallynotbatman.com — This query means that we are looking for event logs in the index “botsv1” which contains the term imreallynotbatman.com
In the sourcetype field, we have 4 log sources that contain the traces of the term we searched.
Our first task is to identify the IP address attempting to perform reconnaissance activity on our web server. It would be obvious to look at the web traffic coming into the network. Let us begin looking at the log source stream:http, which contains the http traffic logs, and examine the src_ip field from the left panel. Src_ip field contains the source IP address it finds in the logs.
Search Query: index=botsv1 imreallynotbatman.com sourcetype=stream:http
Explanation: This query will only look for the term imreallynotbatman.com in the stream:http log source.
We have found two IPs in the src_ip field 40.80.148.42 and 23.22.63.114. The first IP seems to contain a high percentage of the logs as compared to the other IP, which could be the answer. If you want to confirm further, click on each IP one by one, it will be added to the search query, and look at the logs, and you will find the answer.
To further confirm our suspicion about the IP address 40.80.148.42, click on the IP and examine the logs. We can look at interesting fields like User-Agent, Post request, URIs, etc., to see what kind of traffic is coming from this particular IP.
We have narrowed down the results to only show the logs from the source IP 40.80.148.42, looked at the fields of interest, and found traces of the domain being probed. A user-agent string that appears repeatedly in logs with a high frequency could indicate an automated tool scanning for vulnerabilities.
Validate the IP that is scanning
We will then narrow down the result, look into the suricata logs, and see if any rule is triggered on this communication.
Search Query: index=botsv1 imreallynotbatman.com src=40.80.148.42 sourcetype=suricata
Explanation: This query will show the logs from the suricata log source that are detected/generated from the source IP 40.80.248.42
QUESTIONS
1. One suricata alert highlighted the CVE value associated with the attack attempt. What is the CVE value?
On the search query, I will include the term CVE and then use the alert.signature field.
We can see that the alert above was triggered 18 times for the CVE-2014–6271
2. What is the CMS our web server is using?
Joomla
Here I will click on the http.url field. Joomla is the CMS our web server is using. A CMS(Content Management System) simplifies the process of building and managing web content. It offers a wide range of features, including easy content management, template-based design, and extendable functionality through plugins.
3. What is the web scanner, the attacker used to perform the scanning attempts?
Acunetix
I included the term “scan” in the search query.
4. What is the IP address of the server imreallynotbatman.com?
192.168.250.70
We can find this from the dest_ip field. IP 192.168.250.70 received high traffic indicating it was under attack.
Task 5: Exploitation Phase
In this task, we will look at the potential exploitation attempt from the attacker against our web server and see if the attacker was successful in exploiting it or not.
To begin our investigation, let’s note the information we have so far:
- We found two IP addresses sending requests to our server from the reconnaissance phase.
- One of the IPs 40.80.148.42 was seen attempting to scan the server with IP 192.168.250.70.
- The attacker was using the web scanner Acunetix for the scanning attempt.
Search Query: index=botsv1 imreallynotbatman.com sourcetype=stream* | stats count(src_ip) as Requests by src_ip | sort — Requests
Explanation: This query uses the stats function to display the count of requests for each source IP address.
Now we will narrow down the result to show requests sent to our web server, which has the IP 192.168.250.70
Search Query: index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70"
Query Explanation: This query will look for all the inbound traffic towards IP 192.168.250.70.
The result in the src_ip field shows three IP addresses (1 local IP and two remote IPs) that originated the HTTP traffic towards our webserver.
Another interesting field, http_method will give us information about the HTTP Methods observed during these HTTP communications. We observed most of the requests coming to our server through the POST request.
To see what kind of traffic is coming through the POST requests, we will narrow down on the field http_method=POST.
Search Query: index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70" http_method=POST
The result in the src_ip field shows two IP addresses sending all the POST requests to our server.
src_ip, form_data, http_user_agent, and uri are some interesting fields containing valuable information.
- src_ip — This field logs the IP address from which the request originated. It shows the client’s IP address that initiated the request to the server.
- form_data — This field contains the data submitted through web forms in POST requests, such as login credentials, search queries, or any user-input fields.
- http_user_agent — This field stores the user agent string, which identifies the client software, browser, operating system, or bot making the request.
- uri — This field captures the specific page or resource being requested by the client, essentially the path portion of the URL
Our web server is using Joomla CMS. It is seen in a couple of fields like uri, uri_path, http_referrer, etc. It is important because the uri contains the login page, /joomla/administrator/index.php, to access the web portal therefore we will be examining the traffic coming in for a potential brute-force attack.
We can narrow down our search to see the requests sent to the login portal.
Search query: index=botsv1 imreallynotbatman.com sourcetype=stream:http dest_ip=”192.168.250.70" uri=”/joomla/administrator/index.php”
Explanation: We are going to add uri=”/joomla/administrator/index.php” in the search query to show the traffic coming into this URI.
We suspect the attacker may have tried multiple credentials in an attempt to gain access to the admin panel. To confirm, we will dig deep into the values contained within the form_data field.
Search Query: index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70" http_method=POST uri=”/joomla/administrator/index.php” | table _time uri src_ip dest_ip form_data
Explanation: We will add this -> | table _time uri src_ip dest_ip form_data to create a table containing important fields.
We find two interesting fields, username and passwd, with 23.22.63.114 as the source IP. This indicates a potential brute-force attack.
We will then use regex to extract only these two fields and their values from the logs and display them by adding form_data=*username*passwd* in the search head.
Search Query:index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data “passwd=(?<creds>\w+)” | table src_ip creds
Explanation: This query uses regex to extract all the password values found against the field passwd in the logs.
We have two values against the http_user_agent. The first value came from a Mozilla browser and the majority came from a Python script used to automate the brute-force attack.
This clearly shows a continuous brute-force attack attempt from an IP 23.22.63.114 and 1 password attempt batman from IP 40.80.148.42 using the Mozilla browser.
QUESTIONS
1. What was the URI which got multiple brute force attempts?
/joomla/administrator/index.php
From the URI field, we see that /joomla/administrator/index.php got the brute-force attack attempts.
2. Against which username was the brute force attempt made?
admin
3. What was the correct password for admin access to the content management system running imreallynotbatman.com?
batman
4. How many unique passwords were attempted in the brute force attempt?
412
From the task, we see the http_user_agent that performed the brute force sent 412 requests.
5. What IP address is likely attempting a brute-force password attack against imreallynotbatman.com?
We have two source IPs and 23.22.63.114 sends the majority of the requests.
6. After finding the correct password, which IP did the attacker use to log in to the admin panel?
40.80.148.42
Task 6: Installation Phase
In this phase, we will investigate any payload/malicious program uploaded to the server from any attacker’s IPs and installed into the compromised server.
To begin the investigation, we first will narrow down the traffic coming into our server containing the term “.exe”. The search query does not lead to finding but it's a good place to start from.
Search Query: index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70" *.exe
With the search query in place, we are looking for the fields that could have some values of interest. Observing the interesting fields and values, we can see the field part_filename{} contains the two file names, an executable file 3791.exe and a PHP file agent.php.
Next, we need to find if any of these files came from the IP addresses that were found to be associated with the attack earlier.
Search Query:index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70" “part_filename{}”=”3791.exe”
We have found that file 3791.exe was uploaded to the server. Now we need to find out whether the file was executed on the server. We will narrow down our search query to show the logs from the host-centric log sources to answer this question.
Search Query: index=botsv1 “3791.exe”
Following the host-centric log, i.e. sysmon, wineventlog, and fortigate_utm, traces of the executable 3791.exe were found in the sources. We can leverage sysmon and look at the EventCode=1 for program execution.
Search Query: index=botsv1 “3791.exe” sourcetype=”XmlWinEventLog” EventCode=1
Explanation: This query will look for the process Creation logs containing the term “3791.exe” in the logs.
We can see that the file was executed on the compromised server.
QUESTIONS
1. Sysmon also collects the Hash value of the processes being created. What is the MD5 HASH of the program 3791.exe?
AAE3F5A29935E6ABCC2C2754D12A9AF0
We can find this in the event log.
2. Looking at the logs, which user executed the program 3791.exe on the server?
NT AUTHORITY\IUSR
3. Search hash on the virustotal. What other name is associated with this file 3791.exe?
ab.exe
Task 7: Action on Objective
This is the final phase of the cyber kill chain and it involves the attacker taking actions that achieve their goals after successfully gaining access to the target system.
We need to find out what the attacker’s objective was. Our first step could be to figure out the traffic flow.
Search Query: index=botsv1 dest=192.168.250.70 sourcetype=suricata
By examining the Suricata log source, we see that no external IP is communicating with the server. Now let us change the flow direction to see if any communication originates from the server.
Search Query: index=botsv1 src=192.168.250.70 sourcetype=suricata
Usually, web traffic does not originate from the web server, but in this case, we have a large chunk of outbound traffic going to three external IPs. This is worth checking for each of these IPs.
Search Query: index=botsv1 src=192.168.250.70 sourcetype=suricata dest_ip=23.22.63.114
The URL field shows 2 PHP files and one jpeg file. This jpeg file looks interesting. Let us change the search query and see where this jpeg file came from.
Search Query: index=botsv1 url=”/poisonivy-is-coming-for-you-batman.jpeg” dest_ip=”192.168.250.70" | table _time src dest_ip http.hostname url
The result shows the suspicious jpeg file was downloaded from the attacker’s host. The attacker’s objective in this case might have been to intimidate Wayne Enterprises or demonstrate the capability of breaching the web server.
QUESTIONS
1. What is the name of the file that defaced the imreallynotbatman.com website ?
poisonivy-is-coming-for-you-batman.jpeg
2. Fortigate Firewall ‘fortigate_utm’ detected SQL attempt from the attacker’s IP 40.80.148.42. What is the name of the rule that was triggered during the SQL Injection attempt?
HTTP.URI.SQL.INJECTION
We will include fortigate_utm as the sourcetype, and the term SQL in the search head.
Task 8: Command and Control Phase
The attacker uploaded the file to the server before defacing it. While doing so, the attacker used a Dynamic DNS to resolve a malicious IP. Our objective would be to find the IP that the attacker decided the DNS. We will use the network-centric log sources to investigate communication to and from the adversary’s IP address.
Let us start with fortigate_utm to review the firewall logs.
Search Query: index=botsv1 sourcetype=fortigate_utm”poisonivy-is-coming-for-you-batman.jpeg”
Looking into the Fortinet firewall logs, we can see the src IP, destination IP, and URL. The URL field contains the Fully Qualified Domain Name.
Lets us look at another log source, stream:http.
Search Query: index=botsv1 sourcetype=stream:http dest_ip=23.22.63.114 “poisonivy-is-coming-for-you-batman.jpeg” src_ip=192.168.250.70
We have identified the suspicious domain as a Command and Control Server, which the attacker contacted after gaining control of the server.
QUESTIONS
1. This attack used dynamic DNS to resolve to the malicious IP. What fully qualified domain name (FQDN) is associated with this attack?
prankglassinebracket.jumpingcrab.com
Task 9: Weaponization Phase
In this phase of the kill chain, adversaries would create malware for initial access, establish domains similar to the target domain to trick users and create a command and control server for the post-exploitation activity.
This task will mainly look into OSINT sites to see what more information we can get about the adversary. We have found a domain prankglassinebracket.jumpingcrab.com associated with this attack. Our first task would be to find the IP address tied to the domains that may potentially be pre-staged to attack Wayne Enterprise.
First, let's use Robtex threat intel site. This site provides information about IP addresses, domain names, etc.
We find some of the IP addresses associated with the domain. We also find domains and subdomains under the shared tab.
Next, we will search for IP address 23.22.63.114 on robtex. The IP is associated with some domains that look similar to the Wayne Enterprise site.
Next, we’ll use Virustotal. This is also a threat intel site site used to analyze suspicious files, domins, IPs, URLs, and hashes. Search the IP 23.22.63.114.
On the RELATIONS tab, we can see that all the domains associated with this IP look similar to the Wayne Enterprise Company.
In the domain list, we saw the domain that is associated with the attacker www.po1s0n1vy.com. Let us search for this domain on Virustotal.
QUESTIONS
1. What IP address has P01s0n1vy tied to domains that are pre-staged to attack Wayne Enterprises?
23.22.63.114
2. Based on the data gathered from this attack and common open-source intelligence sources for domain names, what is the email address that is most likely associated with the P01s0n1vy APT group?
Task 10: Delivery Phase
Attackers create malware and infect devices to gain initial access or evade defenses and find ways to deliver it through different means. We have identified various IP addresses, domains, and Email addresses associated with this adversary. Our task for this lesson would be to use the information we have about the adversary and use various Threat Hunting platforms and OSINT sites to find any malware linked with the adversary. Our objective would be to understand more about the attacker and their methodology and correlate the information found in the logs with various threat Intel sources.
Let us start our investigation by looking for the IP 23.22.63.114 on the Threat Intel site ThreatMiner
We found three files associated with this IP, from which one file with the hash value c99131e0169171935c5ac32615ed6261 seems to be malicious and something of interest.
Next, search for the hash on Virustotal. Here, we can get information about the metadata about this Malware in the Details tab.
Hybrid Analysis is a beneficial site that shows the behavior Analysis of any malware. Here you can look at all the activities performed by this Malware after being executed. Some of the information that Hybrid-Analysis provides are:
- Network Communication.
- DNS Requests
- Contacted Hosts with Country Mapping
- Strings
- MITRE ATT&CK Mapping
- Malicious Indicators.
- DLLs Imports / Exports
- Mutex Information if created
- File Metadata
- Screenshots
QUESTIONS
1. What is the HASH of the Malware associated with the APT group?
c99131e0169171935c5ac32615ed6261
2. What is the name of the Malware associated with the Poison Ivy Infrastructure?
MirandaTateScreensaver.scr.exe
We can find this using virus total on the Details tab.
Conclusion
In conclusion, our investigation into the cyber-attack on Wayne Enterprise’s website, “imreallynotbatman.com,” shows a methodical attack that followed the stages of the Cyber Kill Chain. The attacker used reconnaissance tools like Acunetix, an automated web scanner, to identify vulnerabilities and successfully executed a brute-force attack from IP 23.22.63.114. IP 40.80.148.42 was used to gain access. After compromising the server, the attacker installed a malicious executable file, 3791.exe, and ultimately defaced the website. Further analysis uncovered details of the attacker’s infrastructure, including associated IPs, multiple masquerading domains, and malware used in the attack. Throughout this investigation, we leveraged host-centric logs to track changes on the compromised server, such as malicious file upload and hash information, while network-centric logs helped us trace external IPs and domains. This approach ensures a comprehensive investigation into what happened.
