RCA : Client Raised possibility of DDOS Attack

Project : Promise Engine

Issue Raised :

Client informed over evening of 15th November that they saw unusual traffic in one of their Service and unusual tail latencies

Investigation :

Checking following Metrics for App Engine :

Requests
Latency

Traffic
Utilization
Memory Usage
Summary of all

OBSERVATIONS

Requests

There are very Short impulsive Peaks in requests that have been there since long, are very short lived and occur at near same time of day
indicating it is a common traffic pattern and not artificially Induced

Also the Peaks are just twice of the average traffic

Latency

These Increase in Latenices co-relates with the impulsive peaks, Normal and expected behaviour

Traffic

Correlates with 1st point

Utilization

Correlates with 1st point

Memory Usage

Correlates with 1st point

Summary
Conclusion

The impulsive Spikes are very short lived and seems to occur 2-3 times a day every day and the peak is only in order of 2-3X that of
average.

Why this doesn’t indicate DDOS Attack?

1. Consistency in Traffic Volume: The graph shows consistency in the volume of traffic over the past four days. In a typical DDoS attack,
there would be a noticeable surge in traffic, far above the regular levels, as the attacker tries to flood the network with excessive
requests.
2. Lack of Sustained Peaks: Although there are spikes in traffic, they are not sustained over a long period. DDoS attacks generally involve
sustained high traffic, not just short bursts, as attackers aim to overwhelm the service for an extended duration.
3. Symmetry in Sent and Received Traffic: The graph shows a rough symmetry between sent and received traffic. In the case of a DDoS
attack, received traffic would typically dwarf sent traffic due to a large number of incoming requests.
4. Server Performance: If the servers were under a DDoS attack, we would likely observe additional indicators such as increased error
rates, server timeouts, or degraded performance.
5. Network Performance Metrics: If there were a DDoS attack, network performance metrics would likely show anomalies. This includes
increased packet loss, higher latency, and possibly the exhaustion of network resources like bandwidth. which are not there in this Case

What can be done to smoothen out these Peaks ?

Increase the Instance Size( Memory, CPU, Network Bandwidth) to next available ladder and monitor the traffic and keep increasing until the
traffic smoothens out

To Ensure protection of Google Cloud deployments from multiple types of threats, including distributed denial-of-service (DDoS) attacks and
application attacks like cross-site scripting (XSS) and SQL injection (SQLi) Google Cloud Armour can be used.
link : Google Cloud Armor overview

For future reference

How to identify DDOS Attack?

1. Unusual Increase in Traffic

Sudden Spike: A sudden, unexplained, and significant increase in network traffic is a strong indicator of a DDoS attack.
Traffic Patterns: Look for traffic that doesn't follow your normal user or usage patterns. For instance, a high volume of traffic coming
from a particular geographic region that doesn’t usually visit your site.

2. Unusual Requests in Server Logs

Repeated Requests: A high number of requests over a short period, especially if they are similar or identical, can be a sign of an attack.
Suspicious IP Addresses: Multiple requests from the same IP address or range of IP addresses in a short time frame.
3. Resource Exhaustion
Bandwidth/CPU Overload: An unexpected maxing out of bandwidth or CPU resources can be a symptom of a DDoS attack.
Unusual System Reports: System reports showing unexplained overuse of system resources.

4. Service Disruptions
Inability to Access the Service: Users reporting that they cannot access the service, especially if the issues are widespread and not
localized.
API or Service Failures: If you notice your APIs or other backend services failing inexplicably, it could be due to being overwhelmed by
requests.

5. Invalid Traffic Patterns

Non-human Traffic: A large amount of traffic that doesn’t appear to be human (e.g., no user-agent, odd access times, high speed, and
frequency of requests).
Traffic from Unusual Locations: An unexpected amount of traffic from countries or regions that do not align with your typical user
base.

Structured Method to Handle it

1. Initial Assessment
Identify the Symptoms: Confirm that the issue you experienced was indeed a DDoS attack. Common indicators include a sudden
surge in traffic, unusually slow network performance, or complete unavailability of your website or application.
Gather Basic Information: Note the time and duration of the attack, any error messages, and which parts of your application were
affected.

2. Analyze Logs and Metrics

Access Logs: Review the logs in Google Cloud's Logging (formerly Stackdriver) to identify patterns indicative of a DDoS attack, such as
a high number of requests from particular IP addresses or geographic locations, or an abnormal increase in traffic.

Metrics Analysis: Use Google Cloud Monitoring to analyze traffic patterns and server responses during the time of the attack.

3. Engage with Google Cloud Support

Contact Support: If you suspect a DDoS attack, contact Google Cloud Support. They can provide insights and assistance specific to
Google Cloud services.

4. DDoS Attack Characteristics to Investigate

Source IP Addresses: Look for clusters of IP addresses generating a large number of requests. However, sophisticated attacks might
use a distributed network, making this challenging.

Request Patterns: Look for patterns in the requests that are abnormal, like identical query parameters, user agents, or referrers.
Geographic Origins: Analyze if the traffic is coming disproportionately from specific countries or regions.
Traffic Volume: Compare the traffic volume during the attack to normal levels.

5. Implement Mitigation Strategies

Google Cloud Armor: Consider using Google Cloud Armor, which provides DDoS protection and web application firewall (WAF)
capabilities.
Rate Limiting: Implement rate limiting to cap the number of requests a user can make within a certain timeframe.
IP Blacklisting: Temporarily block IP addresses or ranges that are identified as sources of malicious traffic.

Geographic Blocking: If the attack is from specific regions and those regions are not significant for your business, consider blocking or
rate-limiting traffic from those areas.