[AWS] - How to get hardware resources and send event on cloudwatch

Three way to get resources

• profiling local storage information (cpu usage)
  • Q. is it possible when you use ecs with fargate
    • we need average cpu utils
• occur event and send event using lambda to api gateway
  • we need to implement additional router
• profiling ecs cpu usage using aws sdk
  • often check cpu usage

Profiling local cpu information

import process from 'process'

class CpuUtil {
  static getCpuUsage() {
    return process.cpuUsage()
  }
}

class Throttler {
  constructor() {
    
  }
  
  private async delay(callback: Function,delayTime: number) {
    return new Promise((resolve, reject) => {
      setTimeout(resolve(callback()), delayTime)
    })
  }
  
  private consume(key) {
    return () => TokenBucket.consume(key)
  }
  
  private isConsumed(result) {
    return result.error
  }
  
  public async throttlingRequest(key, next) {
    const result: RateLimiterRes = TokenBucket.consume(key)
    
    if(isConsumed(result)) {
      return next()
    } 
    
    const cpuUsage = CpuUtil.getCpuUsage()
    const backoffTime = BackoffUtil.getBackoffTime(cpuUsage);
    
    delay(this.consume(key), backoffTime)
     
  }
}

class TokenBucket {
  private rateLimiter: RateLimiter
  
  public consume(key): Promise<RateLimiterRes> {
    const result = this.rateLimiter.consume(key).catch((res) => ({
      rateLimiterRes: payload
      error: true
    }));
    
    if(result.error === undefined) return {
      rateLimiterRes: payload,
      error: false
    }
    return error;
  }
}

Amazon ECS CloudWatch Metrics

Description

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html

• monitor ecs resources using cloudwatch near real-time
• statistics are recorded for a period of two weeks
• send ecs metric data send to cloudwatch every minute

Using CloudWatch metrics

Any Amazon ECS service using the Fargate launch type has CloudWatch CPU and memory utilization metrics automatically

• any ecs service using the fargate launch type has cloudWatch
  • cpu utilization (automaically)
    • average cpu utilization
  • memory utilization (automatically)
  • don`t need to take any manual steps

CpuReservation

• only support for ecs with ec2

CpuUtilization

• ClusterUtilization
  • only support for ecs with ec2
• ServiceUtilization
  • both support for ecs with fargate and ecs with ec2
  • metrics that are filtered by ClusterName and ServiceName
  • unit: percentage

Cluster reservation

• metrics are measured as percentage of CPU, memory, and GPU
• all reserved by all Amazon ECS tasks on a cluster
• aggregate CPU
• It's not supported on clusters with tasks using the Fargate launch type.

 

Cluster utilization

• metrics are measured as the percentage of CPU and memory
• It's not supported on clusters with tasks using the Fargate launch type.
•  

Include metrics

• CPUReservation
  • The percentage of CPU units that are reserved by running tasks in the cluster.
• CPUUtilization
  • The percentage of CPU units that are used in the cluster or service.
• MemoryReservation
  • The percentage of memory that is reserved by running tasks in the cluster.
• MemoryUtilization
  • The percentage of memory that is used in the cluster or service.
• others

Amazon ECS events

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html

ECS sends the following types of events to EventBridge

• container instance state change (EC2)
  • generated to account for the change
    • available resources
      • cpu
      • memory
      • available ports
• task state change (Fargate) our case

return type: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Task.html

• service action
• service deployment state change events

container state changed and task state changed events contain two version fields

one is the main body of the event

one in the detail object of the event

Version Field

In main body

• set to 0 on all events

In detail

• each time a resource changes state, this version is incremented

Container instance state change events

• call the StartTask, RunTask, or StopTask API operations, either directly or with the AWS Management Console or SDKs

Handling Events

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwet_handling.html

• Amazon ECS sends events on an at least once basis.
• receive multiple copies of a given event
• To order of events properly
• each time a resource changes state the version is increment

CloudWatch Container Insights to view Amazon ECS lifecycle events

You can view Amazon ECS task and service lifecycle events within the CloudWatch Container Insights console

• Container instance state change events
• Service action events
• Task state change events

---

Get ECS Cpu Usage

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html

To retrieve CloudWatch metric values

Need to know

• Calls to the GetMetricData API have a different pricing structure than calls to GetMetricStatistics. For more information about pricing
•  
• assume if we request per minute a price for one month is (60 * 24 * 30) request = 0.432$
  • if task is only one
• assume if we request per 30 sec a price for one month is 0.864$
  • if task is only one

GetMetricStatistic

• The maximum number of data points returned from a single call is 1,440
  • which means max 1,440
  • cloudWatch will save data each minute
  • if request more than 1440 data point occur error
  • can narrow the specified time range and make multiple requests across adjacent time ranges

Request Parameters

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_Dimension.html

Dimension

A dimension is a name/value pair that is part of the identity of a metric.

InstanceId=asdf-12341-afasd

MetricName

The name of the metric, with or without spaces

Statistics.member.N

SampleCount | Average | Sum | Minimum | Maximum

Example

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-cloudwatch/classes/getmetricstatisticscommand.html

const client = new CloudWatchClient(config);
const command = new GetMetricStatisticsCommand(input);
const response = await client.send(command);

{
  "Metrics" : [
    ...
    {
        "Namespace": "AWS/EC2",
        "Dimensions": [
            {
                "Name": "InstanceId",
                "Value": "i-1234567890abcdef0"
            }
        ],
        "MetricName": "NetworkOut"
    },
    {
        "Namespace": "AWS/EC2",
        "Dimensions": [
            {
                "Name": "InstanceId",
                "Value": "i-1234567890abcdef0"
            }
        ],
        "MetricName": "CPUUtilization"
    },
    {
        "Namespace": "AWS/EC2",
        "Dimensions": [
            {
                "Name": "InstanceId",
                "Value": "i-1234567890abcdef0"
            }
        ],
        "MetricName": "NetworkIn"
    },
    ...
  ]
}

Our c

const MetricStatisticsCommand: GetMetricStatisticsCommandInput = {
  MetricName: 'CPUUtilization',
  Namespace: 'AWS/ECS',
  Dimension: [{
    Name: 'ServiceName', // beacues we use fargate not ec2
    Value: 'sample-gateway' // i don`t know our ecs service name yet
  }],
  Statistics: ["Average", "Maximum", "Minimum"],
  StartTime = new Date() // determines the first data point to return
}

const command = new GetMetricStatisticsCommand(input);
const response = await client.send(command);

case1: use local cpu usage: it is not accurate value because only get current value

case2: i think it can`t be because ecs event doesn`t include cpu util

case3: this would be work but price can be expensive

Reference

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_AlarmAtThresholdEC2.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-Monitoring-CloudWatch-Metrics.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/AddEventsPutEvents.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Rule.html

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwet_handling.html

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/lifecycle-metrics.html

https://www.reddit.com/r/aws/comments/ctcpu1/am_i_misunderstanding_fargate_cpu_utilization/

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html