ECS Container Task Stats

This isn’t edited because who has the time.

I needed to get stats from task instances and put them into cloudwatch

I used dockerstats formated it and stuck it into cloudwatch. I have some more improvements to make and clean up. It’s going to happen someday. I think. But this is quick and I didn’t find a well formed version in my searching.

Get some docker stats by getting into your EC2 ECS container

https://d.pr/i/DF9YNi

Run

docker stats

This will format some stuff

docker stats — no-stream — format “{{ .Container }},{{ .MemPerc }},{{ .CPUPerc }},{{ .NetIO }},{{ .Name }}”

It’ll spit out some stats

I set a file location

#set a file for stats
statsfile = “/opt/dataplatform/metrics/stats.txt”
#statsfile = “./test.txt”

I put that stats crap in that file

## get docker stats
dockerStatsToText = “docker stats — no-stream — format \”{{ .Container }},{{ .MemPerc }},{{ .CPUPerc }},{{ .NetIO }},{{ .Name }}\” >>” + statsfile

I pulled out the ecs-agent stuff becasue I didn’t care about that container

# remove ecs-agent lines
for line in fileinput.input(statsfile, inplace =1):
    line = line.strip()
    if not 'ecs-agent' in line:
        print line
    metrics = line.split()
    name = metrics[-1]

I parsed that file… poorly.

#parse metrics and send to ecs
#this could be cleaned up a lot
with open(statsfile) as metricfile:
    for line in metricfile:
        contentline = line.strip()
        name = contentline[-20:]
        print name
        contentline = str(contentline + "," + name)
        contentline = str(contentline + "," + name)
        metrics = contentline.split(",")
        print(metrics)
        containerID = metrics[0]
        memPercent = metrics[1][:-1]
        cpuPercent = metrics[2][:-1]
        print metrics[3]
        IOreceivedRaw = metrics[3].split("/")[0]
        print IOreceivedRaw
        IOsentRaw = metrics[3].split("/")[1]
        contName = metrics[4]
        taskID = metrics[5]
        ##convert to MB
        if IOreceivedRaw[-3:] == "kB ":
            IOreceivedMB = float(IOreceivedRaw[:-3])*.001
        if IOreceivedRaw[-3:] == "MB ":
            IOreceivedMB = float(IOreceivedRaw[:-3])
        if IOreceivedRaw[-3:] == "GB ":
            IOreceivedMB = float(IOreceivedRaw[:-3])*1000
        print IOreceivedMB
        ##sent
        print IOsentRaw
        if IOsentRaw[-2:] == "kB":
            IOsentMB = float(IOsentRaw[:-2])*.001
        if IOsentRaw[-2:] == "MB":
            IOsentMB = float(IOsentRaw[:-2])
        if IOsentRaw[-2:] == "GB":
            IOsentMB = float(IOsentRaw[:-2])*1000
        print IOreceivedMB
        print IOsentMB
        print taskID

I sent it to cloudwatch — and also printed out stuff so I could see what was going on… I’m sure there is a way better way to do it.

print "Put CPU Command: aws cloudwatch put-metric-data --metric-name CPUPercentage --namespace DataplatformECS --unit Percent --value " + cpuPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name CPUPercentage --namespace DataplatformECS --unit Percent --value " + cpuPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)

print "Put Mem Command: aws cloudwatch put-metric-data --metric-name memoryPercentage --namespace DataplatformECS --unit Percent --value " + memPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name memoryPercentage --namespace DataplatformECS --unit Percent --value " + memPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)


print "Put IO Rec Command: aws cloudwatch put-metric-data --metric-name IOreceived --namespace DataplatformECS --unit Megabytes --value " + str(IOreceivedMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name IOreceived --namespace DataplatformECS --unit MB --value " + str(IOreceivedMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)



print "Put IO sent Command: aws cloudwatch put-metric-data --metric-name IOsent --namespace DataplatformECS --unit Megabytes --value " + str(IOsentMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name IOsent --namespace DataplatformECS --unit Megabytes --value " + str(IOsentMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)

I stuck some stuff in the user data in cloudformation to make it run on startup…

"yum install aws-cli -y\n",
"yum install git -y\n",
"mkdir file dir to put it/\n",
"touch file text thing like ./test.txt \n",
"chmod 766 set the permsiions on the file\n",
"aws s3api get-object --bucket bucketname --key scriptname.py /opt/dataplatform/metrics/cloudwatchPutECStaskMetrics.py\n",
"crontab -l | { cat; echo '*/5 * * * * python /opt/dataplatform/thescript'; } | crontab -\n",

Thats about it… Here is the thing. If someday someone sees this and cleans it up let me know. Send an email or something if you have questions. also this looks more put together I think… and would probably be waht I update if it gets better. https://gist.github.com/pyralis/938c2aa55ad7515ead15b2b5652cd9b2

#python
import fileinput
import subprocess
#set a file for stats
statsfile = “/opt/dataplatform/metrics/stats.txt”
#statsfile = “./test.txt”
# remove ecs-agent lines
for line in fileinput.input(statsfile, inplace =1):
    line = line.strip()
    if not 'ecs-agent' in line:
        print line
    metrics = line.split()
    name = metrics[-1]
#parse metrics and send to ecs
#this could be cleaned up a lot
with open(statsfile) as metricfile:
    for line in metricfile:
        contentline = line.strip()
        name = contentline[-20:]
        print name
        contentline = str(contentline + "," + name)
        contentline = str(contentline + "," + name)
        metrics = contentline.split(",")
        print(metrics)
        containerID = metrics[0]
        memPercent = metrics[1][:-1]
        cpuPercent = metrics[2][:-1]
        print metrics[3]
        IOreceivedRaw = metrics[3].split("/")[0]
        print IOreceivedRaw
        IOsentRaw = metrics[3].split("/")[1]
        contName = metrics[4]
        taskID = metrics[5]
        ##convert to MB
        if IOreceivedRaw[-3:] == "kB ":
            IOreceivedMB = float(IOreceivedRaw[:-3])*.001
        if IOreceivedRaw[-3:] == "MB ":
            IOreceivedMB = float(IOreceivedRaw[:-3])
        if IOreceivedRaw[-3:] == "GB ":
            IOreceivedMB = float(IOreceivedRaw[:-3])*1000
        print IOreceivedMB
        ##sent
        print IOsentRaw
        if IOsentRaw[-2:] == "kB":
            IOsentMB = float(IOsentRaw[:-2])*.001
        if IOsentRaw[-2:] == "MB":
            IOsentMB = float(IOsentRaw[:-2])
        if IOsentRaw[-2:] == "GB":
            IOsentMB = float(IOsentRaw[:-2])*1000
        print IOreceivedMB
        print IOsentMB
        print taskID
print "Put CPU Command: aws cloudwatch put-metric-data --metric-name CPUPercentage --namespace DataplatformECS --unit Percent --value " + cpuPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name CPUPercentage --namespace DataplatformECS --unit Percent --value " + cpuPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)

print "Put Mem Command: aws cloudwatch put-metric-data --metric-name memoryPercentage --namespace DataplatformECS --unit Percent --value " + memPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name memoryPercentage --namespace DataplatformECS --unit Percent --value " + memPercent + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)


print "Put IO Rec Command: aws cloudwatch put-metric-data --metric-name IOreceived --namespace DataplatformECS --unit Megabytes --value " + str(IOreceivedMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name IOreceived --namespace DataplatformECS --unit MB --value " + str(IOreceivedMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)



print "Put IO sent Command: aws cloudwatch put-metric-data --metric-name IOsent --namespace DataplatformECS --unit Megabytes --value " + str(IOsentMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1"
subprocess.Popen("aws cloudwatch put-metric-data --metric-name IOsent --namespace DataplatformECS --unit Megabytes --value " + str(IOsentMB) + " --dimensions ContainerID="+ containerID +",ContainerName="+ contName +",taskID=" + taskID +" --region us-east-1", shell=True)