AWS Cloudwatch to S3


I've been researching on how to move Cloudwatch logs to S3 Bucket for archiving purposes. Cloudwatch logs can be manually exported as one of its Action menu. Maybe in the future it can be one of its feature to automate but right now I need to create a solution to automate it.

Good thing there is an AWS CLI we can use to automate it. First you need to install the python AWS plugin in able to use it.

https://pypi.python.org/pypi/awscli-cwlogs/1.4.0

Once installed, in may case, I use a linux box for my automation tool (e.g. Puppet, git, ansible, etc.). I just created a cron job that will export the logs to S3 bucket.

0 0 * * * /usr/local/bin/aws logs create-export-task --task-name "LogExport1" --log-group-name "Windows" --destination "prod-os-logs1" --destination-prefix "WindowsLogs/$(date)" --from "$(($(date +\%s\%3N) - 86400000))"  --to "$(date +\%s\%3N)"

Where "Windows" is Cloudwatch log group and "prod-os-logs1" is S3 Bucket