Parsing ESXi Logs for Incident Response with QELP

February 10, 2025 7 mins

Parsing ESXi Logs for Incident Response with QELP

Threat actors frequently target ESXi servers to disrupt business environments and deploy ransomware to encrypt datastores. Stroz Friedberg’s proprietary ESXi log parsing tool Quick ESXi Log Parser (“QELP”) can assist analysts with gaining insights about interesting events unfolding on ESXi servers.

Introduction

Stroz Friedberg has responded to several incidents over the last two years that involved encryption of data at the hypervisor or datastore level. Our team has identified common tactics and techniques that threat actors have used to disrupt services on ESXi servers. Some of these TTPs include:

Enabling SSH to access ESXi servers remotely.
Changing root user passwords to disrupt access to ESXi servers.
Deploying ransomware binaries to encrypt ESXi servers and datastores.

Detecting this malicious activity can be time consuming and challenging, especially when dealing with a significant number of logs from multiple ESXi servers or partial encryption of the logs during ransomware incidents.

To better overcome these analysis challenges, Stroz Friedberg has developed an open-source CLI tool named Quick ESXi Log Parser (“QELP”) that has enhanced investigations by swiftly parsing ESXi logs at scale and producing CSV reports in timeline format, consisting of events such as:

Enabling SSH and shell access.
User logon.
User password change.
New user creation.
File transfer.
File and directory traversal.

Threat actor actions on ESXi servers

VMware ESXi servers are type-1 or bare-metal hypervisors¹ used for creating and managing virtual resources, making them high-value assets in a network. Ransomware threat actors have identified the value in compromising ESXi servers and have transitioned from encrypting data at the endpoint and virtual machine level to encrypting data at the hypervisor level. In addition, a recent increase in ESXi vulnerabilities² coupled with relatively few security controls at the hypervisor level enables data tampering on a larger scale and with lower effort.

Stroz Friedberg has identified common tactics and techniques utilized by threat actors to continue encrypting data at the hypervisor level. Some of the common methods include but are not limited to:

1. Enabling SSH Access: Once threat actors gain access to a network, they may enable SSH access on ESXi hosts using vCenter or Vsphere web clients, to remotely access ESXi hosts.

2. Changing Root Passwords: Threat actors may extract and edit host-profiles from the vCenter web client, edit user settings in the vSphere web client, or utilize the ESXi shell to change root passwords of ESXi servers. Changing these passwords limits access only to threat actors and hinders containment and recovery during incident response.

3. Hypervisor Encryption: Threat actors often retrieve ESXi-compatible ransomware binaries from remote servers or from local compromised systems using SSH. Final steps include deployment of the binary to encrypt data at hypervisor level, including any datastores that contain virtual machines.

Detection of threat actor activities

Threat actor activities like those listed above can be investigated by analyzing ESXi logs. ESXi logs are stored at one of the following locations:³

1. /var/log/

2. /var/run/log/⁴

3. /scratch/log/

4. /tmp (if persistent logging is not enabled)

While ESXi logs can be collected manually from these locations, an easier approach is generating an ESXi support bundle. ESXi has built-in functionality to create and bundle support files for troubleshooting ESXi issues, which includes logs that are valuable for forensic analysis. An ESXi support bundle can be generated by executing the command vm-support from the ESXi shell or by following this guide.

The following logs contain information about common activities conducted by threat actors on ESXI servers.⁵

Log	Forensic Value	Default Log Retention
Hostd.log	Includes details about the ESXi host agent, including: SSH & Console (physical KVM) logons. Web console logons. Files and directories accessed from the web console. Password changes events.	Log files before rotation: 10. Maximum log size: 10 MB.
Syslog.log	Contains general logging including: Logon activity. File and directories accessed via SFTP/SSH, ESXi shell, and vSphere web client.⁶ Password change events.	Log files before rotation: 8. Maximum log size: 1 MB.
Auth.log	Consists of authentication information including: Successful logons. Failed logon attempts. Password change events.	Log files before rotation: 8. Maximum log size: 1 MB.
Vobd.log	Holds details about VMkernal Observations (VOBs) including: SSH Logons. Shell & SSH enable/disable events. Reboot events. Password change events.	Log files before rotation: 8. Maximum log size: 1 MB.
Rhttpproxy.log	Comprises details regarding the ESXi HTTP Reverse Proxy service, including: Source IP addresses that attempt to connect to ESXi server.	Log files before rotation: 8. Maximum log size: 1 MB.
Shell.log	Contains commands executed in the ESXi Shell, such as: Commands related to download and execution of ransomware binary.	Log files before rotation: 8. Maximum log size: 1 MB.
Vmkernel.log	Includes details about ESXi kernel, including: Warnings about the files being executed but are not part of a Vsphere Installation Bundle, if execInstalledOnly is enabled. Source IP addresses that attempt to connect to ESXi server.	Log files before rotation: 8. Maximum log size: 10 MB.
Vmauthd.log	Consists of details about ESXi authorization events, including: Source IP addresses that attempt to connect to ESXi server.	Log files before rotation: 8. Maximum log size: 1 MB.

Log

Forensic Value

Default Log Retention

Hostd.log

Includes details about the ESXi host agent, including:

SSH & Console (physical KVM) logons.
Web console logons.
Files and directories accessed from the web console.
Password changes events.

Log files before rotation: 10.

Maximum log size: 10 MB.

Syslog.log

Contains general logging including:

Logon activity.
File and directories accessed via SFTP/SSH, ESXi shell, and vSphere web client.⁶
Password change events.

Log files before rotation: 8.

Maximum log size: 1 MB.

Auth.log

Consists of authentication information including:

Successful logons.
Failed logon attempts.
Password change events.

Log files before rotation: 8.

Maximum log size: 1 MB.

Vobd.log

Holds details about VMkernal Observations (VOBs) including:

SSH Logons.
Shell & SSH enable/disable events.
Reboot events.
Password change events.

Log files before rotation: 8.

Maximum log size: 1 MB.

Rhttpproxy.log

Comprises details regarding the ESXi HTTP Reverse Proxy service, including:

Source IP addresses that attempt to connect to ESXi server.

Log files before rotation: 8.

Maximum log size: 1 MB.

Shell.log

Contains commands executed in the ESXi Shell, such as:

Commands related to download and execution of ransomware binary.

Log files before rotation: 8.

Maximum log size: 1 MB.

Vmkernel.log

Includes details about ESXi kernel, including:

Warnings about the files being executed but are not part of a Vsphere Installation Bundle, if execInstalledOnly is enabled.
Source IP addresses that attempt to connect to ESXi server.

Log files before rotation: 8.

Maximum log size: 10 MB.

Vmauthd.log

Consists of details about ESXi authorization events, including:

Source IP addresses that attempt to connect to ESXi server.

Log files before rotation: 8.

Maximum log size: 1 MB.

Automating ESXi log parsing

When analyzing ESXi servers, rapid identification of the source system(s) used to access ESXi servers and of the activities conducted by threat actors in ESXi is crucial. However, this can be time consuming when handling many logs from multiple ESXi servers. Additionally, ESXi logs are often partially encrypted by threat actors, resulting in data loss, but still might contain unencrypted information relevant to the incident. Manually identifying the ESXi logs containing the most relevant information and creating a timeline of events can cost valuable analysis time. To overcome these challenges, Stroz Friedberg has developed an open-source Python script, called QELP, to automate ESXi log processing and timelining. The capabilities of QELP include:

Parsing ESXi logs from multiple ESXi servers in a timely and efficient manner.
Parsing partially encrypted ESXi logs.
Output to CSV reports for easy ingestion and analysis.
Timeline generation of the most relevant events to form an overall picture of the events that occurred on ESXi server.

QELP usage and output example

QELP requires input directory containing ESXi support or log archives having only zip, tar, gz, or tgz extensions and an output directory for storing results.
To run QELP, execute:

rye run qelp <esxi_input_dir> <esxi_output_dir>

QELP outputs CVSs containing results from respective ESXi logs as well as a timeline CSV consisting of events that are important from an incident response perspective. Following is a sample timeline generated by QELP:

Figure 1: Sample timeline generated by QELP.

1. https://www.vmware.com/topics/bare-metal-hypervisor

2. According to CVEDetails and MITRE CVE data, in 2024, 8 VMware ESXi vulnerabilities were disclosed as compared to 2 in 2023.

3. Locations can be changed by the system administrator.

4. Can be a symbolic link for the /scratch/log/ directory.

5. https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-832A2618-6B11-4A28-9672-93296DA931D0.html

6. During Stroz Friedberg’s testing, not every file/directory accessed via vSphere web client was captured in Syslog.log.

Building — Support

Contact our cyber response teams for urgent breach assistance.

Contact Us

Author

Phalgun Kulkarni

DFIR

Special Thanks

Kimberly Stone

Solutions Development
Mitch Green

DFIR

Cyber security services are offered by Stroz Friedberg Inc., its subsidiaries and affiliates. Stroz Friedberg is part of Aon’s Cyber Solutions which offers holistic cyber risk management, unsurpassed investigative skills, and proprietary technologies to help clients uncover and quantify cyber risks, protect critical assets, and recover from cyber incidents.

While care has been taken in the preparation of this material and some of the information contained within it has been obtained from sources that Stroz Friedberg believes to be reliable (including third-party sources), Stroz Friedberg does not warrant, represent, or guarantee the accuracy, adequacy, completeness or fitness for any purpose of the article and accepts no liability for any loss incurred in any way whatsoever by any person or organization who may rely upon it. It is for informational purposes only. You should consult with your own professional advisors or IT specialists before implementing any recommendation or following the guidance provided herein. Further, we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. Further, this article has been compiled using information available to us up to 02/10/24.

The contents herein may not be reproduced, reused, reprinted or redistributed without the expressed written consent of Aon, unless otherwise authorized by Aon. To use information contained herein, please write to our team.