In this guide we'll connect it to our own domain.

Services we use

• Route53 for DNS management

• AWS Certificate Manager for SSL certificates

• CloudFront

This guide assumes you have DNS management in AWS. I'll lightly mention the other 2 options we might have if you don't want that within AWS.

Step by step

This guide assumes you went through with the steps of creating an S3 static website, and therefore will build on that architecture.

In a nutshell these are the steps we'll take:

1. create a hosted zone under subdomain (optional)

2. create 4 NS records in Cloudflare for aws.krisfeher.com (optional)

3. edit CloudFront to add Alternate Domain Name

4. request a certificate in N.Virginia (all CloudFront certificates requires a cert in this region)

5. add the DNS validation check entry to Route53

6. Select the now existing cert in ACM

7. Wait for deploy

8. Create alias record in Route53 to direct to the CloudFront distribution

Create a hosted zone

As mentioned, this step is optional, you may already have one and in that case just skip this step.

Go to Route 53 => Hosted zones => Create hosted zone

Add the domain you want to control and make it public.

Direct NS queries from your provider to AWS

You may be in the same shoes as me, and have your DNS managed in CloudFlare (or elsewhere). If that's not the case, you can safely ignore this step as well.

Otherwise, you can create an NS record for a subdomain within Cloudflare and direct it to AWS's name servers:

You can of course just redirect the entire root domain if you want to handle it all in AWS.

You can find out your NS name servers in Route53 once you created a public Hosted Zone for your domain/subdomain:

The advantages of this approach is that SSL termination will be handled entirely in AWS, and this setup is very simple.

(alternatively) If you want to avoid a Hosted Zone in AWS, you'd need to set up SSL ideally in both Cloudflare (or wherever you manage your DNS), and AWS.

It's because you want to make sure SSL is enabled from your browser to => CloudFlare and from CloudFlare to => AWS.

This process is a little more complicated and the SSL setup entirely depends on your provider.

Add custom domain to CloudFront

Go to CloudFront => Distributions => EXXXXXXXXT => Edit settings

Here you can add your Alternate Domain name.

Then select the SSL certificate which will prompt you to create a new one.

Request a new CloudFront SSL certificate

The previous step should take you to N. Virginia region's create certificate console.

If it hasn't, the you can navigate to:

AWS Certificate Manager => Certificates => Request certificate

Request a public certificate there, and add your fully qualified domain name or a wildcard if you want to have the certificate cover your entire domain.

Add the DNS validation check entry to Route53

This is required if you selected DNS validation.

AWS will generate a CNAME name and a CNAME value on the certificate you need to add to Route53, which looks like this:

You can add these manually to wherever you control DNS (either to Route53 or to Cloudflare/etc.)

For convenience AWS added a "Create records in Route53" button on top which just does this for you.

Continue in Cloudfront to add the certificate

Once you did the above step, wait a couple of minutes until your SSL certificate is validated. You can see a "Success" message on the certificate instead of "Pending..."

If that's done, refresh the certificate list on Cloudfront and select the newly created cert.

Click "save chances" on the bottom of the page and wait until the distribution re-deploys (took me about 5 minutes).

Create an alias in Route53

The last step is to direct calls to aws.krisfeher.com to the CloudFront distribution

Create a new A record in Route53 and tick the "alias", then select CloudFront and your distribution name.

Done!

Once this is completed, you can access your static site from the new domain name. In my case, the example Vite project:

This is going to be a quick article, as I thought a lot of people could benefit from it.

AI coding is not the future. It's the present! If you haven't tried it yet, then what are you waiting for? 🙂🤖

There're various paid tools available to get started, but my favourite stack I'll describe below so far has a lot of advantages over others. Mainly price and privacy.

2. Why not chatGPT?

For 3 reasons:

1. from my experience ChatGPT recently has been "dumbed down", which you may heard elsewhere. It generally became lazier at writing longer text (cost saving measure I'd guess), which is quite important for long pieces of code

2. chatGPT has no context of your code

3. chatGPT isn't private and they may use your input to train their model.

Of course you can use chatGPT API, and issue #3 and #1 would be resolved, but that's still a pretty dang expensive tool for coding.

3. So what's the alternative?

Well, your own PC!

If you have a half-decent CPU / GPU, it's worth giving it a go and see how much "speed" you'll get.

You'll VERY LIKELY NOT be able to run bigger models (like a 70 Billion parameter), as those require a lot of GPU vRAM, which your laptop won't have.

Anyway, to get started, go ahead and download https://jan.ai/
This is a tool that allows you to run all those juicy LLMs locally!

The dudes maintaining the application are amazing! They have their own discord channel where all development work is OPEN TO PUBLIC! (along with the source code). You can ask them questions, help them test or contribute to the project. They're very responsive.

Additionally, as far as I know jan.ai the ONLY solution that works with AMD powered GPU laptops. (for desktop PC there're other options)

4. What if I have a crappy PC?

Don't worry, you still have plenty of options.

There're a LOT of serverless infrastructure providers that gives you option to an API you can use.

Places like mystic.ai , predibase and my favourite together.ai . But there's more!

They all give you free $20-25 credit, so there's no hurt in trying them.

Do remember the word "serverless", as those are the ones charging you per token or per second of inference (the other option is "dedicated").

The best part of it?

Much better pricing!

Yes, you read that right, it's 1M tokens, not 1k!! That makes this solution about 500-1000 times cheaper than openAI's GPT-4 turbo!

5. what about opensource model performance?

Good question.

Let me show you a little screenshot:

(source: https://deepseekcoder.github.io/)

It isn't so bad at all!

DeepSeek-Coder Instruct 33B is about as good as GPT4!

Now you probably won't be able to run a 33B model on your local PC (unless you have a beefy machine at home), that's why we can use together.ai !

• "Ok, so these are the models. What about the IDE?"

6. IDE and extensions

Here, you have more than just one options. VSCode and Jetbrains has their own AI assistants, but they're paid. There're probably others too.
Then you have more niche IDEs like "Cursor", which is an AI driven IDE from ground up (using VSCode as basis). The gui of this is excellent!

This means it's smoother and more integrated, and it's easier to do AI-related tasks within the IDE. I've used this for a while, but as you may guessed, it's paid as well (unless you use your own openai key)

Then recently I discovered an extension called "Continue". And that's where I stopped!

With continue, you can use either VScode or Jetbrains IDEs :

So to get going, install the extension and modify the config file.

Here's 2 examples you can add in the config file for local (jan.ai) and remote (together.ai):

    {
      "title": "deepseek-coder-6.7b-instruct.Q2_K",
      "model": "deepseek-coder-6.7b-instruct.Q2_K",
      "apiBase": "http://127.0.0.1:1337/v1",
      "completionOptions": {},
      "provider": "openai"
    },
    {
      "title": "deepseek-ai/deepseek-coder-33b-instruct",
      "model": "deepseek-ai/deepseek-coder-33b-instruct",
      "apiKey": "your secret API key for together.ai",
      "completionOptions": {},
      "provider": "together"
    },

As you see the first item has the provider "openai" set. The reason behind this is that jan.ai creates a web server that has openAI compatible API. So any other models can be accessed the same way as openAi. (switching between them is easy)

And honestly? That's it! That's all you need to set this up.

If you really want, you can download additional models from Huggingface and just copy it into the following folder : C:\Users\youruser\jan\models\deepseek-coder-6.7b-instruct.Q2_K

Make sure the model and the containing folder name are identical.

So to summarize, the tools we used:

• jan.ai

• continue.dev

• vscode (or vscodium as I did above)

• an LLM model

• together.ai

Let me know if you're stuck or you have a question and good luck AI-coding!

In this guide I'll show you how to create a static website on S3 and serve it via Cloudfront.

You'll end up with something like this:
- duhttestg7mi2.cloudfront.net

Which is a generated domain name. In the next article I'll show you how to do a custom domain name.

So why is it free?

Well... technically not free, but VERY VERY cheap.

The only 2 services you'll use is S3, where storing a small website costs virtually nothing, and CloudFront which has a free tier resetting every month forever.

And what is a static website?

A static website consists of pre-built pages with fixed content, displaying the same information to every visitor. Created using HTML, CSS, and JavaScript files, these sites don't require server-side processing or database management. This simplicity leads to faster load times, improved performance, and easy maintenance. Static websites can be hosted on various platforms, including CDNs, enhancing speed and reliability. In short, static websites are efficient and reliable, offering benefits like faster load times and easier upkeep, making them popular among web developers and content creators.

What examples can you do with a static website?

• Personal blogs or portfolios showcasing an your work

• Small business websites providing information about products, services, and contact details

• Landing pages for marketing campaigns, product launches, or events

• Online documentation or knowledgebases for software products or services

What are the limitations and things you CANNOT accomplish with a static website?

• Inability to handle complex, dynamic content like user-generated content or real-time data feeds

• Limited support for e-commerce functionality, such as shopping carts and payment processing

• Challenges in implementing advanced search capabilities or personalized content based on user preferences

• Difficulty in managing large-scale websites with frequently updated content, as each update requires regenerating the entire site

Okay, so this is out of the way, let's get to work!

2. Services we will use

S3 => to host the website
Cloudfront => to cache the content and give it HTTPS

What this guide will not cover?

• a custom domain name. You'll end up with an autogenerated site-name (you'll find custom website name in the next article)

3. Create a bucket

There's nothing special in here.

Go to Amazon => S3 Buckets and create a bucket.

Contrary to what you heard you don't need to do any of the following:

• Enable static website hosting (don't do it)

• Enable public access to the bucket (don't do it)

The reason for the above is, that we'll block public access to S3 and only have our static site available from CloudFront via OAC (Origin Access Control).

However what we need later on is:

• A bucket policy to allow access from CloudFront

The only step you do here is the following:

• Upload your static website to S3. I you need an easy way to upload your website, you can use my previous article about accessing S3 from your PC (or just use AWS Console)

I will just use one of the example Vite project I created a short while ago.

4. Cloudfront

We can have Cloudfront in front of our S3 website to provide secure access (S3 doesn't provide us with an SSL certificate)

• Go to Cloudfront and create a new distribution.

• Make sure you enter the S3 bucket name and NOT the bucket endpoint

• Add Origin access control settings and create a new OAC with default settings. Make sure you select "Sign Requests (recommended)" here, otherwise you'll get an Access Denied from CloudFront

  

• Select : "Redirect HTTP to HTTPS" at the Viewer section.

• Add index.html do the "Default root object - optional" section

You can leave the rest as default or modify it to your liking.
Click Create.

After a few minutes of deploying you'll be provided a domain distribution name like https://xxxxxxx.cloudfront.net that you can use to access your website.

While you wait you can do the bucket policy update.
When you created the CF distribution, you probably had a yellow alert on top of the page:

You can either click on the provided "Copy policy" button or have this added to your bucket

{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "AllowCloudFrontServicePrincipal",
            "Effect": "Allow",
            "Principal": {
                "Service": "cloudfront.amazonaws.com"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::staticwebsiteexample123456/*",
            "Condition": {
                "StringEquals": {
                    "AWS:SourceArn": "arn:aws:cloudfront::123456767:distribution/EOXIN7AR0ZZO2"
                }
            }
        }
    ]
}

Of course use your own CloudFront and s3 bucket ARN.

Once you have all these you should have your static site deployed.

In the next article I'll show you how to have a custom domain name!

After setting this up once, you don't need to even open a browser!

1. The tools

We'll use the following tools:

• S3

• IAM for permissions

• Cyberduck (or any other file explorer client with S3 support)

That's all!

2. S3 setup

Go to S3 here, and create a bucket Amazon S3 => Buckets => Create bucket

Give it a name, and leave everything else default (for now).

3. IAM setup

As Cyberduck (and many 3rd party tools) only accepts access key and secret key, we'd need to generate one with appropriate permissions.

If you're doing this as an admin, you don't want to have all permissions associated with your account provided to Cyberduck.

In security, it's called Principle of least privilege, which is:

... a security concept that ensures users or entities only have access to the necessary data, resources, and applications for completing a task, and no more.

Let's create a new user for this purpose at IAM => Users => Create user . Give it a name.

Then create a new policy at IAM => Policies => Create policy

And add a JSON for the policy with your own bucket name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::cyberduck-bucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:InitiateMultipartUpload",
                "s3:UploadPart",
                "s3:CompleteMultipartUpload"
            ],
            "Resource": "arn:aws:s3:::cyberduck-bucket/*"
        }
    ]
}

Save this policy under a name, then go back to your user and attach the policy to it.

Once done, create an access key for the user IAM Users => cyberduck-user => Create access key

Once you generated one, you can go to the next section and enter it in Cyberduck.

4. File explorer

Download Cyberduck (or your tool of choice) and install it.

Create a new "bookmark" and fill in these details.

Add the Access Key ID of the user we created in the previous section.

Make sure you add the bucket name to the "path" variable. As you see this is also appended to the URL above. If you're using a different client, it may ask for the full URL instead.

Once done, you can drag and drop files across S3 without resorting to S3's own console or using the terminal.

That's it!

Hope this quick tutorial helped you being more comfortable using S3.

Well, it's Friday afternoon, and I had way too much coffee, so this article will be a little light hearted ! If you're not up for it, and you want serious content.... well, I guess I have some AWS-related content you can have a look instead 🙂

My dream is to have a Jarvis-like AI available in my home and help me out when I'm building my iron man su........when I'm doing aws stuff.

Today I'll show you how to remind yourself of important things that we can't live without. Things that makes to world go by, things that provide people satisfaction and a reason to live.

Yes, I'm talking about taking out the bins! Yep, that bin. Not the virtual one on your pc, but real-life one that you forgot to take out and now looks like a tower built from trash.

Here's what we'll do today:

"We'll have an AI notify us every time the garbage collector comes to take our bins away!"

Tools we'll use

We'll use the following tools for this purpose:

N8N

This is similar to Zapier. An automation framework that'll greatly simplify building this all. Why not Zapier? Because N8N can be self-hosted on my own homelab and have access to secure private information (and my home assistant setup). Also, N8N self-hosting is free, and always has been. Big thanks to the guys at N8N for making this possible!

I'll not cover installing N8N, but if you have docker installed, it takes like 2 minutes. If you don't use Docker yet....well then what are you waiting for?

https://docs.n8n.io/hosting/installation/docker/

ChatGPT / OpenAI API

• Yep, we'll use the big evil here.

• Yes, I don't like it either.

• Yes it'd be better locally.

• Do I have disposable £2000 to build a local LLM machine running at decent speed? Maybe.

• Can I justify the spending for my wife? No. 🙂

As a silver lining our AI masters has promised us not to use data on the API as training data, so we can be relatively safe:

Once you have openAI api access, you can generate an API key for it.

There's also a possibility to use other Huggingface models with inference (which may be actually free), but I'm not sure which model would perform well.

Telegram/Pushover/Email, whatever

So here we'll set up a Lambda function via Eventbridge and use SNS to send a text.......nah, just kidding, not on a FRIDAY AFTERNOON!

Basically just pick any notification service. Pick your preferred one. Check N8N integrations, there's around 700 different integrations.

I'll not detail this part, as it'll be different for everyone, and N8N makes it incredibly easy to set it up with whatever preference you have.

You can even have it create a reminder for you in your google calendar if you wish.

I personally use a Telegram chatbot, because I find it more flexible than anything else. An easy option as well is Pushover, which is a service for like a decade that does extremely simple phone notification. Both of them are just an API call, so it's not rocket science 🙂
Oh, and both are free.

A bin collection website

For our particular area, there's a website that provides me with the bin collection dates. You type in the postcode, and it gives you a list of when it's being collected and what date.

If you don't find it immediately check the browser dev tools for any sort of API calls. If you're lucky, you'll be able to call it from your own machine. Here's how mine looked like:

Which means I can take a note of the request URL and retrieve the bin collection dates anytime I want specific to my address. Keep a note of your URL, as you'll need it later on.

Step 1: HTTP request

Not much to say here, grab the HTTP request node and have the URL pasted in with a GET request

Extract content

Get an HTML node and extract the specific content. Here I had a block of text with an ID assigned to them (I got lucky), but you can pick any CSS selectors, try to make sure it's somewhat unique and it's a dynamic element that may move around the page.

you can go on further extracting more stuff from this using even more HTML nodes and go as specific as you want. I wanted to remove my address from the response, so I made some gymnastics using a few HTML nodes and merges to do that.

At the end, this is how I made my response look:

Which resulted in something like this:

[{"response": "Friday 9th February - ResidualBin"}]

This is 2 lists, on containing the dates, other the bin types. Then with a code block, I returned the first hit, which is the next collection date and bin type.

Checking if date is tomorrow

For that I wrote a disgusting piece of code that determines if that piece of text is tomorrow. I had chatGPT generate me code. I'm not crazy enough to do this myself.

There's probably easier ways to do this, but how could we call ourselves engineers if we didn't pick the most complicated way of doing something? 😊

function isDateTomorrow(dateString) {
    // Extract date components from the string
    const dateComponents = dateString.match(/(\w+)\s+(\d+)\w+\s+(\w+)/);
    if (!dateComponents) {
        return false; // Format not matched
    }

    // Map month names to month numbers
    const monthNames = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"];
    const monthNumber = monthNames.indexOf(dateComponents[3]);

    if (monthNumber === -1) {
        return false; // Invalid month name
    }

    // Create a date object from the extracted components
    const year = new Date().getFullYear(); // Assuming the year is the current year
    const parsedDate = new Date(year, monthNumber, parseInt(dateComponents[2], 10));

    // Get today's date for comparison
    const today = new Date();
    today.setHours(0, 0, 0, 0); // Reset time to 00:00:00 for date-only comparison
    today.setDate(today.getDate() + 1)

    // Compare the parsed date with today's date
    return parsedDate.getTime() === today.getTime();
}

const isTomorrow = isDateTomorrow($input.last().json.response);
const resp = { isTomorrow: isTomorrow };

if (isTomorrow) {
    resp.when = $input.last().json.response;
}

return [{response: resp }]

After a tiny bit of modification it worked fine.

For every return statement please make sure it conforms to the following structure:

return [{some object}] or in our case it's: [ { "response": { "isTomorrow": true, "when": "Friday 9th February - ResidualBin" } } ]

i.e. an object (or multiple) returned in a list. If you don't do it like that N8N will complain 🙂

Then once you have this, just return the value:

So for now our flow does the following:

• On being called, it returns the date of next collection, along with the fact if it's tomorrow or not.

Here's how the whole flow looks:

Step 2: AI Agent

Wow that sounds fancy isn't it? 🙂

This will be your agent you can ask to do anything (i.e. you can re-use this later on for other tasks)

First of all, add your OpenAI API key to the credentials menu on the left. It'll then apply to all further OpenAI related nodes.

A few nodes to mention that you can see on the screenshot

• Execute Workflow Trigger => it's basically a node that lets other workflows trigger this workflow.

• On a new manual Chat message => this will be added automatically when you add a new agent. It just means you can test the agent from this screen via a chat interface

• Model: here's where you add the model you want to use. Nothing else required here

• Agent: Again, nothing else here besides the value to use from the previous node, which is : {{ $json.chat_input }}

• Window Buffer memory : this is to keep your Agent's conversation somewhere, so it remembers your previous message. For this guide it's irrelevant.

• Tools: you can see a few tools attached. These are basically other workflows you created in N8N that the agent can use.

  Here's how the tool looks like that we've created in the previous step:

  

  Take a note of your workflow ID. That's the URL of the previous workflow you worked on.

The agent will make an educated guess on which tool to use.
Unfortunately I haven't had too much luck on complex tool usage, especially if the agent doesn't get the expected output. It just keeps on calling the tool until it gets bored. Literally.

Working with an AI Agent is like working with a kid. At least that's how it feels!

Step 3: Scheduling

This is a short step, and can be done in other ways.

I set up a schedule workflow separately that calls the Agent with a specific input:

The Schedule has a daily schedule every day 8pm. The edit fields contains the following:

with a value of: "Please check if there's bin collection tomorrow. Once you did that, send me a mobile message if it's tomorrow with the details. If it's NOT tomorrow, don't send me any message."

You can see from the above, that it kept sending me messages even if it wasn't tomorrow 😅 Yes, we got to the point where an AI agent is harassing me.

Then the last node is just to call the workflow with the specific ID.

So the full flow so far:

1. Schedule triggers and calls Agent

2. Agent gets instruction and uses bin collection tool

3. Bin collection tool scrapes website and returns bin date

4. Agent decides to send message or not based on response

5. (not done yet) Agent uses messaging tool to send message

As you see it's fairly simple and looking at it now I can see that the AI agent is a completely redundant part of the flow (you could just send a message straight away after "istomorrow" is true.)

But I wrote all these up already, I won't delete the entire article, sorry 😅
At least we learned how to use an agent! And N8N!

One thing is left tho, which is sending the notification to your phone

Step 4: Notification

Again, this will be different for every person, but here's my flow:

In the middle is an API call to Telegram, while you can see an error or a success node at the end to lead the Agent on the right path.

Once you have this, save it and add it as a tool for your AI Agent.
Now your AI agent has 2 tools!

That's it!

As you can see it's fairly easy to add more tools to your agent and do other stuff for you.

Let me know if you have any questions.

N8N and LLMs are something I'm actively playing with, so you can expect more of these articles in the future (besides AWS)

to continue.... :

1. Setting up EC2

cluster

Let's set up an ECS cluster to host our application. If you have this already available you can skip this step and jump to the next one.

In ECS click on create cluster, and in the infrastructure section you can set what EC2 would you like your docker host to be.

The network configuration will be already filled in with all your subnets and VPC, but you can customize it if you want.

Select the defaulted public subnets.

What this will do is create an EC2 along with an autoscaling group that scales your EC2 (with default dynamic scaling) and a launch template which is just a blueprint for your EC2.

Your next step should be to modify the auto-scaling group, as it's totally wrong 🙂

A few things we need to create a new launch template version
EC2 => Launch templates => Modify template (Create new version)

• pick an AMI that's ECS optimized. Search for "ECS" within the AWS Marketplace AMIs, and pick the operating system you wish. Here's the one I used for Amazon linux in us-east : ami-04d4dd7b34e293332

• change the user_data: Open up the "Advanced" section and dunk this script in, where "test-cluster" if the name of your cluster. If it's already there, happy days.

    #!/bin/bash 
    echo ECS_CLUSTER=test-cluster >> /etc/ecs/ecs.config;
  

There're a few things that need to happen in order to have an EC2 attach to an ECS cluster, namely:

• user data with the cluster name as above

• ECS agent installed, which is included in the AMI

• linked to capacity provider, which is done by AWS when you created the cluster

• IAM instance profile for ECS agent, done as well by AWS

• Outgoing security groups to provide communication, done as well.

I'll provide a terraform template later on that includes all these.

2. Setting up ECS

task definition

This is a blueprint for our that defines containers configurations(Docker image to use, CPU and memory requirements, environment variables, and networking settings).

To create one, within ECS on the left hand menu click on "Task definition", then give it a name. You can fill in each section as below:

Infrastructure requirements

You can leave most of this default.
Only a few to change:
- Launch type is EC2
- network mode is bridge (this is important, as awsvpc doesn't support dynamic port mapping)
- (optional) Task size: 512 MB (we'll not require much memory for this)

Container

Some of the changes I've made:

• Add a name and the ECR image URL.

• host port: 0 (this is to indicate dynamic port mapping)

Once done, click create and it'll create revision 1 of your task definition.

Again, I'll provide a TF template for this.

Service

The next up is the service. Why not a task you may ask?

• A Task Definition outlines container requirements, such as Docker image, ports, resources, and environment variables; a Task runs the defined containers, suitable for short-lived jobs, and is not replaced automatically if stopped.

• A Service ensures a set number of Tasks are constantly running, replaces failed Tasks, can balance them across resources and zones, and can be configured with a load balancer, unlike standalone Tasks.

For our purpose, a service is better.

Here're the sections one-by-one:

• environment

Choose the default capacity provider strategy we created earlier.

• deployment configuration

Give a name to your service, add the task definition we created earlier and select "Replica" as the service type.
You can then select how many tasks (clones) you want to run on this service. For the sake of simplicity I selected one.

• load balancing

This is an important one. Make sure you select your current load balancer, and the target group you defined above. If you're unable to do so "it's greyed out, or not available", then make sure you have the following:

1. your target group exists, does not have targets, and is an instance type target group

2. your target group is assigned to the load balancer, within the correct VPC

3. your task definition includes "bridge" networking

4. your container definition includes host port of 0

The rest of the options you can leave it as default. There's a lot more to pick here, but we'll not delve into those options.

Once this is done, you can see the tasks provisioning for the service (depending on how many task you picked earlier):

3. Terraform templates

To provide you with a terraform templates:

EC2.tf

resource "aws_launch_template" "EC2LaunchTemplate" {
  name_prefix   = "ECSLaunchTemplate-"
  image_id      = "ami-04d4dd7b34e293332"  #ECS-OPTIMIZED AMI
  instance_type = "t3.medium"
  key_name      = aws_key_pair.sshkey.key_name

  iam_instance_profile {
    arn = aws_iam_instance_profile.ecsInstanceRole.arn
  }

  vpc_security_group_ids = [aws_security_group.ecs_sg.id]

  user_data = base64encode("#!/bin/bash\necho ECS_CLUSTER=${var.cluster_name} >> /etc/ecs/ecs.config")
}

resource "aws_key_pair" "sshkey" {
  key_name   = "ssh-key"
  public_key = "add your own SSH public key here you generated on your PC"
}

resource "aws_autoscaling_group" "AutoScalingGroup" {
  name_prefix      = "ecs-asg-"
  min_size         = 1
  max_size         = 2
  desired_capacity = 1
  launch_template {
    id      = aws_launch_template.EC2LaunchTemplate.id
    version = "$Latest"
  }

  vpc_zone_identifier = [aws_subnet.public_subnet_1.id, aws_subnet.public_subnet_2.id]
  health_check_type   = "EC2"
}

resource "aws_iam_role" "ecsInstanceRole" {
  name_prefix = "ecsInstanceRole-"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
  })
}

resource "aws_security_group" "ecs_sg" {
  name_prefix = "ecs-sg-"
  vpc_id      = aws_vpc.my_vpc.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_iam_instance_profile" "ecsInstanceRole" {
  name = aws_iam_role.ecsInstanceRole.name
  role = aws_iam_role.ecsInstanceRole.name
}

resource "aws_iam_role_policy_attachment" "ecs_role_policy" {
  role       = aws_iam_role.ecsInstanceRole.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

ECS.tf

resource "aws_ecr_repository" "ECRRepository" {
  name = "cicd-example"
}

resource "aws_ecs_cluster" "ECSCluster" {
  name = var.cluster_name

  capacity_providers = [aws_ecs_capacity_provider.ecs_capacity_provider.name]

  default_capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.ecs_capacity_provider.name
    weight            = 1
    base              = 1
  }
}

resource "aws_iam_policy" "ecs_logs_policy" {
  name        = "ecsLogsPolicy"
  description = "Allow ECS tasks to interact with CloudWatch Logs"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "logs:CreateLogGroup",
          "logs:DescribeLogStreams"
        ],
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

resource "aws_iam_policy_attachment" "ecs_logs_policy_attachment" {
  name       = "ecs-logs-policy-attachment"
  roles      = [aws_iam_role.ecsTaskExecutionRole.name]
  policy_arn = aws_iam_policy.ecs_logs_policy.arn
}


resource "aws_iam_role" "ecsTaskExecutionRole" {
  name = "ecsTaskExecutionRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Service = "ecs-tasks.amazonaws.com"
        },
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ecsTaskExecutionRole_policy" {
  role       = aws_iam_role.ecsTaskExecutionRole.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_ecs_task_definition" "ECSTaskDefinition" {
  family                   = "example-react-project"
  execution_role_arn       = aws_iam_role.ecsTaskExecutionRole.arn
  network_mode             = "bridge"
  requires_compatibilities = ["EC2"]
  cpu                      = "1024"
  memory                   = "512"
  container_definitions = templatefile("container_definitions.json.tpl", {
    account_id = data.aws_caller_identity.current.account_id
  })
}

resource "aws_ecs_service" "ECSService" {
  name                               = "react-service"
  cluster                            = aws_ecs_cluster.ECSCluster.arn
  task_definition                    = aws_ecs_task_definition.ECSTaskDefinition.arn
  desired_count                      = 2
  deployment_maximum_percent         = 200
  deployment_minimum_healthy_percent = 100
  scheduling_strategy                = "REPLICA"

  load_balancer {
    target_group_arn = aws_lb_target_group.exampleTG.arn
    container_name   = "react-container"
    container_port   = 80
  }
}

resource "aws_ecs_capacity_provider" "ecs_capacity_provider" {
  name = "EC2CapacityProvider"

  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.AutoScalingGroup.arn

    managed_scaling {
      maximum_scaling_step_size = 1
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 100
    }
  }
}

And the container definition:

container_definition.json.tpl

[
  {
    "name": "react-container",
    "image": "${account_id}.dkr.ecr.us-east-1.amazonaws.com/cicd-example:main",
    "cpu": 0,
    "portMappings": [
      {
        "containerPort": 80,
        "hostPort": 0,
        "protocol": "tcp"
      }
    ],
    "essential": true,
    "environment": [],
    "mountPoints": [],
    "volumesFrom": [],
    "logConfiguration": {   
      "logDriver": "awslogs",
      "options": {
        "awslogs-create-group": "true",
        "awslogs-group": "/ecs/example-react-project",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "ecs"
      }
    }
  }
]

The above terraform templates represent the manual steps you've done before.

Because AWS does a lot of wiring automatically in the console, it's not required to do there, however for TF templates it isn't the case, hence the lengthy configuration.

One more thing. Before you deploy the ECS service, make sure you run your bitbucket pipeline, otherwise it won't deploy.

4. Next steps

So far you should have an architecture that deploys your code fine to ECR, however it doesn't yet re-trigger the deploy step in ECS.

An easy way to get around this issue is to trigger a re-deploy via this bitbucket pipe:

https://bitbucket.org/product/features/pipelines/integrations?search=ecs&p=atlassian/aws-ecs-deploy

you can simple add a few new lines to your bitbucket-pipelines.yaml file:

- pipe: atlassian/aws-ecs-deploy:1.12.1
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                AWS_DEFAULT_REGION: "us-east-1"
                CLUSTER_NAME: "cicd-cluster"
                SERVICE_NAME: "react-service"
                FORCE_NEW_DEPLOYMENT: "true"

on every push now, the service will redeploy:

There you go! You now have a very basic pipeline that deploys to ECS.

Needless to say, please don't use this in production, as this isn't meant for that. This is meant to show how you could do the same, and provides a baseline you can improve later on.

• A bitbucket repository with existing SSH keys to connect to and 2FA enabled

• An existing repository with some code to build

• An ALB (Application Load balancer)

• Your application dockerized

The simplified flow

1. Developer pushes up code to bitbucket on developer branch, then creates pull request to main branch

2. Somebody approves the pull request after reviewing code changes

3. Bitbucket triggers build on their server and tests are run

4. Docker image is is built and pushed up to ECR

5. Based on new image, ECS service is re-deployed

Things this guide will not cover

1. Changes to database schema on push

Sometimes a change is required in the database schema. This is generally more complex, depending on how the database is set up, if any ORM is in place, how the database differs between environments, if there're data fixtures, etc. This requires a lot more thought, and the goal of this article isn't something else.

2. Backend layer

This largely depends on the architecture, but in a simple 3-tier, the backend would be just another similar ECS service beside the front-end.

3. Complex code

With simple code and architecture it's easier to understand how the flow works, so adding complex code (or testing code) is not necessary. For the same reason we'll not have a separate web server container or worry about database.

4. Branching strategy

While it's an important part of the pipeline, for this article I'll have only have 1 branch (main).

In an ideal scenario you would want to have multiple branches associated with multiple environments, say "feature branches", "staging", "preprod/UAT" and "production" as an example. Then each developer would create pull requests to one of the environment branches, get the code approved and merge the changes.

At release-time, you would tag a commit as a release (with the release number), and deploy that commit. We'll omit all these today, but will come back to it later.

1. The setup

AWS

Terraform template

In this section we need to have an example architecture, which in our case is:

• 1x ECR repository to store our docker images

• 1x ECS machine set up as a docker host for ECS (yes, in this guide we'll not use Fargate)

• + other parts within ECS: cluster, service, task (+ task definition)

• + AWS wiring, like permissions, auto-scaling, networking, etc.

This is boring to set up, so I show you with a Terraform template to speed this up. It'll create the necessary bits.

If you don't know how to run terraform, first you need to install AWS CLI. Here's a quick tutorial I found for that: https://medium.com/@simonazhangzy/installing-and-configuring-the-aws-cli-7d33796e4a7c

Then you can use Terraform : https://developer.hashicorp.com/terraform/tutorials/aws-get-started/infrastructure-as-code

networking.tf

resource "aws_vpc" "my_vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
}

resource "aws_subnet" "public_subnet_1" {
  vpc_id                  = aws_vpc.my_vpc.id
  cidr_block              = "10.0.1.0/24"
  map_public_ip_on_launch = true
  availability_zone       = "us-east-1a"
}

resource "aws_subnet" "public_subnet_2" {
  vpc_id                  = aws_vpc.my_vpc.id
  cidr_block              = "10.0.4.0/24"
  map_public_ip_on_launch = true
  availability_zone       = "us-east-1b"
}

resource "aws_internet_gateway" "internet_gw" {
  vpc_id = aws_vpc.my_vpc.id
}

resource "aws_route_table" "public_route_table" {
  vpc_id = aws_vpc.my_vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.internet_gw.id
  }
}

resource "aws_route_table_association" "public_subnet_association" {
  subnet_id      = aws_subnet.public_subnet_1.id
  route_table_id = aws_route_table.public_route_table.id
}

resource "aws_route_table_association" "public_subnet_2_association" {
  subnet_id      = aws_subnet.public_subnet_2.id
  route_table_id = aws_route_table.public_route_table.id
}

resource "aws_security_group" "allow_web" {
  vpc_id = aws_vpc.my_vpc.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

This will create a brand new VPC, along with 2 subnets, route tables, internet gateway.

variables.tf

data "aws_caller_identity" "current" {}

variable "ami_id" {
  description = "The AMI ID to use for EC2"
  type        = string
  default     = "ami-04d4dd7b34e293332"
}

variable "instance_type" {
  description = "The instance type of the EC2 instance"
  type        = string
  default     = "t3.medium"
}

variable "cluster_name" {
  description = "The name of the ECS cluster"
  type        = string
  default     = "cicd-cluster"
}

Here you can add your specific requirements for your EC2 host (for ECS), and the ECS cluster name. (we'll have a TF template for the cluster later on)

load_balancing.tf

resource "aws_lb" "alb1" {
  name               = "alb1"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.allow_web.id]
  subnets            = [aws_subnet.public_subnet_1.id, aws_subnet.public_subnet_2.id]
}

resource "aws_lb_target_group" "exampleTG" {
  name        = "exampleTG"
  port        = 80
  protocol    = "HTTP"
  target_type = "instance"
  vpc_id      = aws_vpc.my_vpc.id

  health_check {
    enabled = true
    path    = "/"
  }
}

resource "aws_lb_listener" "http_listener" {
  load_balancer_arn = aws_lb.alb1.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.exampleTG.arn
  }
}

Above will create you a load balancer and a target group.

Target Groups

As you see the above template includes a target group. You can do it manually as well as using terraform.
Once you have a load balancer, you would need to have a target group to direct traffic.

You can create the Target group on the console here:
EC2 => Target groups => Create target group

A few things to watch out for :

• target-type: instance (IP does not work for dynamic port mapping)

• VPC: select your VPC, it'll default to......the default VPC 🙂

• DO NOT assign any targets to the target group. Leave it empty and save it.

Bitbucket

• Enable deployments for your repository in the "Deployments" section. For this you'll require 2FA to be set up.

• Enable pipelines as well in the repository settings

• At the same place go to Repository Variables and add your AWS access keys:

  

2. Setting up the pipeline to run

dockerfile

In order to create a docker image on the pipeline, we need a docker file.

A very simple example for a react vite application could be this:

FROM node:20 as build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

FROM nginx:stable-alpine
COPY --from=build /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

bitbucket-pipelines.yml

Here's the example I've used.

image: node:20

options:
  docker: true

pipelines:
  branches:
    main:
      - step:
          name: Build and Push Docker Image
          script:
            # Install AWS CLI v2
            - echo " --> Installing AWS CLI..."
            - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" --create-dirs -o "/tmp/awscli/awscliv2.zip"
            - unzip -qq /tmp/awscli/awscliv2.zip -d /tmp/
            - /tmp/aws/install

            # Login to Amazon ECR
            - echo " --> Logging into Amazon ECR..."
            - aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 529768619555.dkr.ecr.us-east-1.amazonaws.com

            # Build Docker image with cache-from option and build arguments
            - echo " --> Building Docker image..."
            - docker build . --progress=plain --tag=cicd-example
            - docker images

            # Push the Docker image to Amazon ECR
            - echo " --> Pushing Docker image to Amazon ECR..."
            - pipe: atlassian/aws-ecr-push-image:1.5.0
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                AWS_DEFAULT_REGION: "us-east-1"
                IMAGE_NAME: "cicd-example"
                TAGS: "${BITBUCKET_BRANCH} ${BITBUCKET_COMMIT}"

Once you push this file up, a build will trigger.

You can see the description in the comments. The "pipe" section is something specific to bitbucket (bitbucket pipes), which are on their own little docker functions to do a specific task.

You can read about this particular here:
https://bitbucket.org/bitbucket/product/features/pipelines/integrations?search=ecr

Once this pipeline finishes, you should see something like this in ECR:

For now, we'll leave it at this.

In the next part we'll continue with ECS and EC2 setup. If you have any question feel free to contact me, happy to help!

Public AWS Lambda functions are powerful, but without proper safeguards, they can be misused. Here’s what can go wrong:

1. Open Access: If anyone can trigger your Lambda function, it might be used for the wrong reasons, leading to overuse or sensitive data exposure.

2. Cost Issues: Lambda charges are based on usage. If someone repeatedly triggers your function, your bills could skyrocket.

3. Data Theft: A Lambda function dealing with sensitive data can be a target for data leaks if not secured properly.

4. Service Disruption: Excessive traffic, whether intentional or not, can overload your Lambda functions, disrupting the service.

In this guide I'll focus on #2.

The steps

Here're the steps in order we'll do.

1. Create a Lambda function + test

2. Add an API gateway endpoint to it

3. Create an API key with limited usage on it + test

The Lambda function

First we need to create an example Lambda function:

Let's fill in the lambda function with code that adds a unix timestamp to the request JSON:

import json
import time

def lambda_handler(event, context):
    # Parsing the JSON body from the event
    data = json.loads(event['body'])

    # Append the current Unix timestamp
    data['timestamp'] = int(time.time())

    # Return the modified data as JSON
    return {
        'statusCode': 200,
        'body': json.dumps(data)
    }

Click Deploy.

Once done, you can add a new test event and test with this example that imitates an API gateway JSON:

{
    "resource": "/formsubmit",
    "path": "/formsubmit",
    "httpMethod": "POST",
    "headers": {
        "Content-Type": "application/json",
        "Accept": "application/json"
    },
    "queryStringParameters": null,
    "multiValueQueryStringParameters": null,
    "pathParameters": null,
    "stageVariables": null,
    "requestContext": {
        "requestTime": "30/Jan/2024:12:31:45 +0000",
        "path": "/prod/formsubmit",
        "protocol": "HTTP/1.1",
        "stage": "prod",
        "domainName": "api.example.com",
        "requestId": "123456789",
        "requestTimeEpoch": 1580389905401,
        "accountId": "123456789012",
        "apiId": "abcdefghij"
    },
    "body": "{\"name\": \"John Doe\", \"email\": \"johndoe@example.com\"}",
    "isBase64Encoded": false
}

In the execution results you should see something like this, having the timestamp added:

Response
{
  "statusCode": 200,
  "body": "{\"name\": \"John Doe\", \"email\": \"johndoe@example.com\", \"timestamp\": 1706616281}"
}

API Gateway

At this point you have a lambda function that can be used within AWS.

In order to call it from outside your architecture, you can have an API Gateway endpoint triggering your Lambda function.

Something like this:

API gateway will give us the power to restrict the incoming calls using API keys.

So as a next step...create a trigger in your lambda function and select API gateway.

Create a REST Api:

Then visit your newly created API.

Deploy this to any stage, I used default, then take a note of the invoke URL that's something similar to this, and append your function name to it:

https://123456asdfg.execute-api.us-east-1.amazonaws.com/default/timestamper

You can use this URL to test your endpoint.

Here's the fun part, let's try it!

You can curl the endpoint from any terminal:

curl -X POST "https://12345asdfg.execute-api.us-east-1.amazonaws.com/default/timestamper" \
     -H "Content-Type: application/json" \
     -d "{\"name\": \"John Doe\", \"email\": \"johndoe@example.com\"}"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100    77  100    52    180    122 --:--:-- --:--:-- --:--:--   304{"name": "John Doe", "email": "johndoe@example.com", "timestamp": 1706626702}

As you see the response has come back and now our gateway endpoint is public 😱

What we need to do now is to limit the access of it via API key.

Gateway API keys

Within API Gateway => APIs => API keys => Create API key section, create an API key. You only need to add a name to it and autogenerate it.

Similarly, just above that, you can add a API Gateway => APIs => Usage plan => Create usage plan

And this is where the magic happens.

Right here you can restrict the API in various ways, as an example:

This will provide

• 10 calls a day with this API key

• max. 1 call a second

• max. 1 call at a time

Of course it is fairly restrictive, but this will prevent anyone calling your 100s of times a second racking up a nice Lambda and API Gateway bill for you.

Once done, go back to your API key and add it to the usage plan. You can find this option under the "Actions" button.

You can now go back to your API and modify the resource with the EDIT button:

Once clicked, you can turn on "Api key required", then save.

Before you re-try your curl, make sure you "Deploy API" and wait a few minutes.

After that, your response should be something like this:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    75  100    23  100    52     59    133 --:--:-- --:--:-- --:--:--   193{"message":"Forbidden"}

Great! Now our endpoint requires an API key.

That's great, but how do I send the API key? 🤔

Well, here's how:

curl -X POST "https://12345asdfg.execute-api.us-east-1.amazonaws.com/default/timestamper2" \
      -H "Content-Type: application/json" \
      -H "x-api-key: 1234567890asdfghjkl" \
      -d "{\"name\": \"John Doe\", \"email\": \"johndoe@example.com\"}"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100    77  100    52    138     93 --:--:-- --:--:-- --:--:--   232{"name": "John Doe", "email": "johndoe@example.com", "timestamp": 1706632463}

If we send request too quickly we'll get this response:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    83  100    31  100    52     71    120 --:--:-- --:--:-- --:--:--   193{"message":"Too Many Requests"}

And if we run out of the daily limit we'll get this:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    80  100    28  100    52     81    151 --:--:-- --:--:-- --:--:--   234{"message":"Limit Exceeded"}

Closing thoughts

There's of course plenty of other ways to secure your lambda function, to name a few:

• AWS WAF for an extra security layer. It’s like having a guard to block unwanted traffic. With this, you can selectively deny traffic.

• Control Access with IAM who can use your Lambda functions.

• Use CloudWatch for detailed logging and alerts.

• Set limits on how many instances of your Lambda function can run at once via Lambda concurrency. This prevents your system from getting overwhelmed by too much traffic.

This guide will provide a simple way to set up an EC2 and give secure access to a user without opening any ports on the EC2.

What we'll end up with is:

• an EC2 machine in our current VPC with appropriate permissions

• additional permissions to a chosen user that allows sessions to above EC2

Prerequisites:

• A VPC

• A user with access key and ID set up on your local machine

• AWS cli on your local machine to access

The advantages of this setup:

• As it'll be CloudFormation, it's easy to tear it down once not used

• No inbound ports need open for the EC2 => no chance to hack in via open port 22 or any other port

• still full access from your own local terminal application

The CloudFormation template

AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 with Session access to a user

Parameters:
  InstanceType:
    Type: String
    Description: EC2 instance type
    Default: t2.micro
  VPCId:
    Type: AWS::EC2::VPC::Id
    Description: VPC ID where the instance will be launched
  AMIId:
    Type: String
    Description: AMI ID for the EC2 instance
    Default: ami-0a3c3a20c09d6f377
  IAMUser:
    Type: String
    Description: The name of the existing IAM user

Resources:
  MyInstance:
    Type: 'AWS::EC2::Instance'
    Properties:
      ImageId: !Ref AMIId
      InstanceType: !Ref InstanceType
      IamInstanceProfile: !Ref SSMInstanceProfile

  SSMInstanceProfile:
    Type: 'AWS::IAM::InstanceProfile'
    Properties:
      Path: "/"
      Roles:
        - !Ref SSMRole

  SSMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: "/"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

  IAMUserPolicy:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: 'StartSessionPolicy'
      Users:
        - !Ref IAMUser
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: "Allow"
            Action: "ssm:StartSession"
            Resource: !Sub "arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${MyInstance}"
          - Effect: "Allow"
            Action:
              - "ssm:DescribeSessions"
              - "ssm:GetConnectionStatus"
              - "ssm:DescribeInstanceProperties"
              - "ec2:DescribeInstances"
            Resource: "*"
          - Effect: "Allow"
            Action:
              - "ssm:TerminateSession"
              - "ssm:ResumeSession"
            Resource: !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:session/${IAMUser}-*"

Outputs:
  InstanceId:
    Description: The Instance ID
    Value: !Ref MyInstance

To run this, go to CloudFormation and upload a new template.

You'll be requested to enter a few details:

Stack name: doesn't matter, let your imagination free 🙂
IAMUser: the user you want to assign permissions to access the new EC2
AMIId : it's the ID of the amazon machine image that you should use. You can get this by manually trying to create an EC2 machine. This will be unique to your operating system requirements and region, but feel free to copy mine (us-east-1).
InstanceType: it's the type and size of instance you want to deploy.
VPCId: it's going to be a dropdown with all your VPCs. Pick one.

Once you filled this in, click next a few times and accept the checkbox about creating IAM resources.

Wait a few minutes until the stack completes and you'll be presented with your EC2 machine.

This machine will have the appropriate security groups, instance profile and your user will receive the necessary permissions to access this machine.

Accessing the machine

Port 22 SSH access will not work, as no port is open, however you can access your machine the following way from your terminal:

1. Make sure your and credentials file contains your access key and ID and your config file contains the correct region.

2. make a note of the instance ID. You can see that in the "outputs" tab of your CloudFormation stack.

3. in your console aws ssm start-session --target i-1234567890abcdef

4. this will give you the default shell. If you want say bash, then run /bin/bash

You should now have an EC2 with secure access.

Tearing everything down

Go back to your CloudFomation and click the "Delete" button on the stack.
This will remove all resources that was created beforehand.

If:

1. You have a significant amount, possibly millions of images with various usage patterns that you serve to website guests

2. If these images are not static images, but dynamic, generated on the user request

3. If you care about load times/SEO impact and cost

You should not use this guide:

1. If you have a few static images that can be easily pre-sized to appropriate size

2. If you're unable to make front-end code changes in image fetching.

Introduction

When transferring images in web and mobile applications, it's common to face challenges with image size optimization, as images are often sent without proper resizing, causing long transfer times, especially when handling multiple images.

This is typically not a problem with static images that can be pre-sized to various resolutions. However, if your web app doesn't yet know which images it needs to render, this issue can become more significant.

This solution involves enabling any front-end to request images in various sizes. This would be determined by the visitor’s device resolution, allowing for the delivery of an image that is optimally sized for each specific device.

Let's do first 4 different images sizes, however it's up to you how many you pick:

• Small

• Medium

• Large

• Extra Large

Users can request the size most suitable for their needs. For instance, a mobile device might require a 'small' image, whereas a desktop could be better served with an 'extra large' one.

Each size category scales the original image as follows, while maintaining the aspect ratio for height:

• Small: 150px width

• Medium: 300px width

• Large: 600px width

• Extra Large: 1200px width

Again, these sizes can be different.

Avoiding exact measurements for each image, such as 500x300, is strategic. It prevents the creation of an excessive number of image sizes (we'll store them later on in S3), which would otherwise increase storage costs and potentially slow down network transfers.

AWS tech we'll use

• S3 (static site, redirection rules, generic S3 knowledge)

• Lambda function (node js, lambda layers)

• API gateway (generic API gateway knowledge)

• Npm, build node projects, etc.

• Cloudfront (for caching images)

High level overview

There’re 3 players in this game.

1. S3 bucket (this stores the original images, and the resized images)
2. API gateway (this is used as an entry point for non-resized images)
3. Lambda function (this is used to resize images and redirect to the new image)

1. User requests image via Cloudfront URL. Something like https://somethingsomething.cloudfront.net/small/img/catimage.png

2. Cloudfront checks its own cache if it has the image. If yes, it serves it back to the user. In this case, the process stops.

3. If Cloudfront doesn't have it in its cache, it goes to S3 to retrieve the resized file. If S3 has the resized file, it returns it to Cloudfront, which caches it and returns it to the user.

4. If S3 also doesn't have the resized image, it results in a 404, and hence forwards the call to API gateway.

5. API gateway in return forwards the call to the Lambda function, which resizes the image

6. Lambda then places it in the S3 bucket and returns a new URL to the user.

As you see this would only need to execute once for each image. Any further calls would then get the resized image from Cloudfront.

The setup

Below you’ll find the steps to set up the solution.

I’ll not detail a generic S3, Lambda or API Gateway setup or permissions. There’re plenty of tutorials online. I’ll mention all the peculiarities though.

S3 initial setup

1. Set up an S3 bucket and dump some images on it

2. Set up public access to it

3. Set the permissions to allow public access (later on you can limit this to Cloudfront)

4. Enable static website hosting

At this point, you should be able to access all files in the bucket from anywhere.

Lambda + API gateway setup

1. create a function and attach a trigger to API gateway.

2. Leave the gateway open to public.

3. Set up Lambda permissions to S3 bucket

Here’s the policy I used (for logging and S3 PutObject)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::img.yourbucket/*"
        }
    ]
}

4. Create a node project on your local PC and add an index.js file with these details. This is the code that does the resizing. Feel free to modify it as you wish. We'll upload this to Lambda.

'use strict';

const AWS = require('aws-sdk');
const S3 = new AWS.S3({
  signatureVersion: 'v4',
});
const Sharp = require('sharp');

const BUCKET = process.env.BUCKET;
const URL = process.env.URL;

exports.handler = function(event, context, callback) {
  const key = event.queryStringParameters.key;
  const match = key.match(/(small|medium|large|xlarge)\/(.*)\.(jpg|jpeg|png)/);

  const scale = match[1];
  const imageName = match[2];
  const imageExtension = match[3];
  const imagePath = imageName + '.' + imageExtension;

  const sizes = new Map();
  sizes.set('small', 150);
  sizes.set('medium', 300);
  sizes.set('large', 600);
  sizes.set('xlarge', 1200);

  const newSize = sizes.get(scale);
  let contentType;
  if  (["png"].includes(imageExtension.toLowerCase())) contentType = 'image/png';
  if  (["jpg", "jpeg"].includes(imageExtension.toLowerCase())) contentType = 'image/jpeg';


  S3.getObject({Bucket: BUCKET, Key: imagePath}).promise()
    .then(data => Sharp(data.Body)
      .resize({
        fit: Sharp.fit.contain,
        width: newSize
      })
      .toBuffer()
    )
    .then(buffer => S3.putObject({
        Body: buffer,
        Bucket: BUCKET,
        ContentType: contentType,
        Key: key,
      }).promise()
    )
    .then(() => callback(null, {
        statusCode: '301',
        headers: {'location': `${URL}/${key}`},
        body: '',
      })
    )
    .catch(err => callback(err))
}

Here’s the package.json.

{
  "name": "image-resize",
  "version": "1.0.0",
  "description": "Serverless image resizing",
  "readme": "Serverless image resizing",
  "main": "index.js",
  "scripts": {
    "build-copy": "npm install && mkdir -p nodejs && cp -r node_modules nodejs/ && zip -r  {file-name}.zip nodejs"
  },
  "devDependencies": {
    "aws-sdk": "^2.1046.0",
    "sharp": "^0.29.3"
  }
}

5. npm install --arch=x64 --platform=linux This will create a node_modules folder and a package lock file.
Because Sharp package has a different package for arch linux (which is used by lambda), you’d need to install this type. Please see here.

6. Create a nodejs folder and copy node_modules folder into it. Zip the whole nodejs folder
7. Create a new layer in lambda and upload this zip as your layer. This is needed, so you can separate your dependencies from your actual code.

8. Zip your package.json, your json lock and index.js. Upload this file as your code to Lambda

9. Attach the previously created layer to this code. This can be version one.

10. Add new environment variables. One for the bucket and one for the url path. See example below

11. At this point your lambda function should work and you can test it by calling your API gateway EP. You can see your error logs for lambda in Cloudwatch.
The URL will be something like this: https://123456789.execute-api.us-west-1.amazonaws.com/default/image-resizer

Where /default/ is your stage you set and /image-resizer is your lambda function name.

Add redirection to S3 bucket

To have this all working together, you need to tell the S3 bucket to redirect calls that fail to retrieve any object.

Add a redirection rule to your static site settings:

[
    {
        "Condition": {
            "HttpErrorCodeReturnedEquals": "404"
        },
        "Redirect": {
            "HostName": "123456789.execute-api.eu-west-1.amazonaws.com",
            "HttpRedirectCode": "307",
            "Protocol": "https",
            "ReplaceKeyPrefixWith": "default/image-resizer?key="
        }
    }
]

Where HttpErrorCodeReturnedEquals is a condition that triggers when a 404 (not found) is returned from the bucket on a particular path.

HostName is your API gateway hostname

HttpRedirectCode is what you tell the user’s browser on the response (307 is temp. redirect)

Protocol is the secure http protocol

ReplaceKeyPrefixWith is used to replace incoming paths with other paths (along with KeyPrefixEquals tag in the condition block, which isn’t there now). In our case it simply adds the default/image-resizer?key= just after the domain name, making the rest of the path a query parameter instead, which we can parse in the Lambda function.

Add Cloudfront + HTTPS

Add a CloudFront distribution in front of your S3 bucket that contains the images. When you add your S3 bucket, make sure you don’t pre-select the bucket AWS offers you.
If you do so regardless, AWS will not use the static site endpoint (that has the redirection set up). The issue you’ll face will be a tricky one, as your available images will actually serve perfectly fine via Cloudfront, only the new (not yet resized) images will give “Access Denied” response. This response is actually quite misleading because when an object doesn’t exist on S3, you get this error from CloudFront.

So to avoid that, make sure you use the static site EP URL.

Additionally, change your lambda functions URL environment variable to point to Cloudfront instead of the S3 bucket. This will help your browser serve HTTPS images at all times (even on first call/cache)

This will have the following advantages:

• it'll save your money using Cloudfront instead of S3

• speed up network requests for users

Additional improvements to consider

Add Lifecycle rules for resized images only

This would be quite handy, as some images are only accessed once, and never again. Still, we’ll end up paying for the storage costs of those images. Deleting those images when not needed anymore would “free up space” on the bucket.

Reason why it’s recommended:
- not using an S3 bucket for storing an image that isn't used means saving money.

Add WEBP conversion (recommended, but later)

WebP is a modern image format designed for the web. It is both smaller than jpg and better quality.
(see: WebP Compression Study | Google for Developers )

Sharp gives you the option to convert images to WebP on the fly, further reducing the image size.

Reason why it’s recommended:
- smaller image size means smaller load times for users and faster pages
- extremely easy to implement and switch to webP at any time on our lambda function
- if you replace all your jpg and png with WebP, you'll save money on storage costs in S3

A secure password

At account creation make sure you pick a strong password. Understandably "strong password" is a vague explanation, but you type in something similar to a service like this:
https://bitwarden.com/password-strength/

Please note that even though Bitwarden is a trusted site, I would by no means enter any real password of mine. You can type in something similar in a similar format and length to give you a rough estimate how secure of a password you picked.

Here's an example I tried. It's deceptively easy to remember:

Once you've gone through account creation, you can log into your root account.

After logging in with root account

Setting up MFA

Root accounts for AWS are something that generally isn't meant to be used for day-to-day operations.
This account has all the power associated with your account. It can do literally anything.

To avoid abuse of this, the 2nd thing you should do is enable 2FA on your root account.

• To do this, navigate to your security credentials:

• At the Multi-factor authentication (MFA) section click on "Add MFA Device"

• Add a name above to associate it with the device and click on Authenticator APP

• Follow the on-screen instructions to add your device:

  

• It's asking for 2x MFA code, to get the 2nd code, just wait 30 seconds in your MFA application.

• Once done, you can log out of your account and test your MFA login

Setting up a budget alert

To alert you if your spending goes beyond a certain threshold, you can set up budget alerts.

To do this, navigate to Billing and Cost Management => Budgets => Create budget

You can then either set a monthly amount you're comfortable with along with an alert email.

This will alert you any time you're close to reaching your budget, however it'll do nothing else!

Update Billing Preferences

Navigate to Billing and Cost Management => Billing Preferences

Select both these options:

The first option will provide you alerts if you exceed Free Tier limits
The second option enables Cloudwatch alerts to be created around billing

Create a cloudwatch billing alarm

Navigate to CloudWatch => Alarms => Create alarm

For metric, select All => Billing => Total Estimated Charge

Specify metric to maximum => 6h . As noted above, anything quicker than 6h will cause the below warning, as billing updates aren't more frequent than 6h.

Choose a condition to trigger when it's greater than your set amount. As an example:

Click next, and on the next page create a target notification for the alarm.

Create a new topic and set your email address there as below:

Create this topic.

On the next page, give it a name and then review your alarm.

Once you did that, you'll receive an email that you need to confirm.

After all this, your alarm is active!

Do NOTs

• do not use the root account for day-to-day operations

• do not create an access key for the root account

• do not share your credentials with anyone

Creating a new day-to-day user

To avoid using your AWS root account, create a new user for logging in and performing daily activities.

Go to IAM => Users => Create user
Create a new user with AWS Management Console access, and select
I want to create an IAM user

Set them a password, then on the permissions page assign AdministratorAccess to them.

Click on create user and then log out and log back in with your newly created user.

On the newly created user, enable MFA just like you did with the root account.

Once you've done the above, your IAM => Dashboard should look like this:

And that's done!

Of course, there's much more you can do to enhance security, but the steps outlined above provide a solid starting point for those just beginning their journey.

A good practice to do backups is to follow the 3-2-1 rule. That's having

• 3 backups of the same data

• 2 of them on separate media

• 1 of them off-site

Translating this to the cloud we can do:

• 3 backups of the same data

• 2 of them are on separate clouds

• 1 of them offline

In this guide we'll deal with the "separate cloud" approach.

Options you may have

There are various options out there and generally, you have a category of these options:

1. Self-build scripts

2. Native Cloud products (AWS, GCP, Azure, etc.)

3. Off-the-shelf 3rd party products

While all of the above could be a solution to your problem, this guide will focus on #2, and specifically AWS => GCP

Initial setup

You'll need the following:

• An S3 bucket on AWS with something to back up

• A GCP bucket you'll clone the files to

The tools we'll use:

• AWS S3

• AWS IAM

• GCP Storage tranfer service

• GCP Cloud Storage

• GCP Pub/Sub

• GCP Cloud Functions

• (GCP Event Arc)

How it will work?

1. Item is uploaded to AWS S3

2. GCP Storage Transfer Service fires weekly/daily/etc. to copy over files

3. On fail/success, Storage service publishes a message on GCP Pub/Sub

4. On a Pub/Sub message, EventArc is triggered, which calls GCP Cloud Functions

5. GCP Cloud function takes the event, extracts the status and sends it to Slack

Step-by-step tutorial on how to do the above

Create a GCP bucket

This is a fairly simple task, you can pick a name, storage class, region, etc. for your bucket.

Make sure the bucket isn't public, and its name is unique.

Create an IAM user and access key in AWS

This will be used by your GCP service.

Here’s an example permission:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-db-backups",
                "arn:aws:s3:::your-db-backups/*"
            ]
        }
    ]
}

Then create an Access Key and Secret Key for this user and take (temporary) note of it.

The job is done with AWS for this tutorial. To GCP!

Create a Storage Transfer service job

1. Choose source (S3) and destination (GCP)

2. Set up the source details with the Access key you generated in the previous step

3. Choose destination GCP bucket (you may need to enable permissions here for the service account)

4. Choose how often to run the job (or just run it once)

5. Choose further misc. options (like deletion, overwrite, etc.) and make sure "Get transfer operation status updates via Cloud Pub/Sub notifications" is clicked.
   Create a new Pub/Sub topic here and select that.

6. Done. At this point you can already test the job.

Create a Cloud function

1. Go to your Pub/Sub topic that was created on storage service creation

2. Click on “Trigger Cloud Function”, which will prompt you to create a new function there

3. Add anything there (for now). This will create a basic function along with EventArc trigger, and hook them up together

4. Go to your function and write the code to call slack. Here’s the one I used:

const axios = require('axios');

const functions = require('@google-cloud/functions-framework');
const url = `to be filled in`;

functions.cloudEvent('notifySlack', cloudEvent => {
  let payload;

  try {
    console.log('CloudEvent:', JSON.stringify(cloudEvent, null, 2));

    payload = {
      status: cloudEvent?.data?.message?.attributes?.eventType,
      description: `JOB: ${cloudEvent?.data?.message?.attributes?.transferJobName}`
    }
  }
  catch (error) {
    payload = {
      status: "GCP db backup status",
      description: "An exception has happened in GCP CloudFunctions, backups has failed."
    }
  }


  axios.post(url, payload)
    .then(response => {
      callback(null, response.data);
    })
    .catch(error => {
      callback(error);
    });
});

1. You can test it via CloudShell and see the console log outputting some text.

Create your Slack Webhook

You can of course create any other notification method, or call and endpoint, but here we'll use Slack's built in webhooks.

1. Find the "Workflow builder" in Slack Automations and create a new "Workflow"

2. As trigger create a Webhook and take a note of it

3. As an action send a message to your slack channel

4. Go back to your Cloud function and add the webhook URL

5. Done!

At this point you should be able to test your implementation.

A good way to test it without much effect is to disable the Access keys in S3.
This will fail the Storage Transfer job and will send a message to Slack.

This guide covers automated, regular SQL dumps to S3.

The AWS services to use for this are:

• Systems manager “runcommand” feature to start the backup

• S3 to have the backup stored

• A Lambda function that does the runcommand operation

• Eventbridge to schedule the lambda execution

Prerequisites:

- AWS Session manager (within Systems manager)

Advantages

• You're not required to have a separate machine doing backups or use the DB host EC2 for this purpose.

• No need to open any inbound ports (beyond outbound port 443 to systems manager client EP)

• You can add logging and alerts to Cloudwatch to complement your existing logging

• All backup would still be at a central place within Lambda and Eventbridge

• Data traffic goes straight to database to S3 (no intermediary)

Disadvantages

• A little long to set this up

• Costs a tiny bit of money
  No charge for => RunCommand, Session manager, EventBridge(free tier), Lambda (free tier),
  Small charge for => S3, depending on DB size

Set up

Session manager

Make sure you have Session manager enabled and working with the EC2 that hosts the database

Lambda setup

Create a lambda with the following code:

import boto3
from botocore.exceptions import ClientError

def lambda_handler(event, context):
    ssm = boto3.client('ssm')
    instance_id = 'i-123456789' 
    database_user = 'db_user' 
    database_password = 'db_password' 
    database_name = 'db_name'
    bucket_name = 'db-backups-bucket'
    backup_file = 'backup_filename' 

    # The command to backup the database and upload to S3
    backup_command = f"mysqldump -h 127.0.0.1 -u {database_user} -p{database_password} {database_name} | aws s3 cp - s3://{bucket_name}/{backup_file}_$(date +%Y%m%d).sql"

    response = ''
    try:
        response = ssm.send_command(
            InstanceIds=[
                instance_id,
            ],
            DocumentName='AWS-RunShellScript',  # runs commands in the EC2 instance
            Parameters={
                'commands': [backup_command]
            },
        )
        print(f"Command sent to instance {instance_id}. Response: {response}")
    except ClientError as e:
        print(f"Unexpected error when sending command to EC2 instance: {e}")

    return {
        'statusCode': 200,
        'body': f"Command sent to instance {instance_id}. Command ID: {response['Command']['CommandId']}"
    }

This will create an IAM ROLE for this Lambda function. Add this extra permission to the role (beyond the Log creation one that was added automatically on creation)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:SendCommand",
            "Resource": [
                "arn:aws:ec2:region:account:instance/i-123456789",
                "arn:aws:ssm:region::document/AWS-RunShellScript"
            ]
        }
    ]
}

This will give permission to your lambda function to run a shell script on a particular EC2 machine.

You can test this code by clicking "Test" and run it with no test input.

You'll either see an error => in this case it's probably a permission issue

You see no error => your shell script run, however it may have failed.

You can look at the execution logs in Run Command within Systems Manager and see something like this:

Clicking on the Failed item, you can see some details about the error:

As at this point this isn't set up fully, you'll probably see an error corresponding to not having access to S3. Let's do that next!

S3 setup

Make sure you have a bucket you want to place the backups in. Please don't have this bucket public 🙂

Add an IAM role to the EC2 machine to allow upload to S3 (this way Lambda isn't required to handle large amounts of data or wait until execution finishes)

This policy worked for me to access the bucket. It omits delete permission for safety.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::db-backups-bucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::db-backups-bucket",
                "arn:aws:s3:::db-backups-bucket/*"
            ]
        }
    ]
}

Once this is done, re-test your lambda and it should work.

You can verify that the SQL backup file is in S3

Eventbridge

Go to Eventbridge Scheduler (it's a new feature).

Create a daily schedule to run a Lambda function:

Eventbridge then will create an IAM role for itself to give itself permissions to execute the Lambda function (or you can use your own)

Once this is done, you have a functioning daily backup.

Some things to improve

• As currently S3 will keep the backup forever, there's an option here to delete the latest one regularly within S3 lifecycle roles.

• Create an alarm via cloudwatch to notify of any issues with the backup

• Regularly check the integrity of the backups

You may occasionally come across services that needs access to your AWS infrastructure, or an EC2 machine.

One example of this is allowing Bitbucket build machines to SSH into EC2 within AWS.

A quick guide

1. create a file called bitbucket_whitelist.txt

   For convenience, here's the full list:
   https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/

This will give you one IP address / line.

1. create an .sh file with the following:

#!/bin/bash

SECURITY_GROUP_ID=sg-123456789 # This is your existing security group ID

while read -r CIDR
do
    aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol tcp --port 22 --cidr $CIDR
done < bitbucket_whitelist.txt

Make sure you save this with linux line endings. In visual studio you can find this option on the bottom right corner:

3. Add yourself permission in IAM:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:AuthorizeSecurityGroupIngress",
            "Resource": "arn:aws:ec2:region:accountnumber:security-group/sg-123456789"
        }
    ]
}

Done.

Once you run this, it should one-by-one add the CIDR ranges to the inbound allow list of your given security group.

Enabling session manager and accessing EC2 terminal is a safer way than doing it via port 22 and SSH-ing into the machine. For the below, you don't need to enable any incoming ports, however you need (at least) port 443 to be open for outbound traffic.

1. Install session manager agent to the EC2 machine. Amazon Linux 2 machines have this default installed, however the version may be old, so it’s still worth updating it.

sudo systemctl status amazon-ssm-agent # check if it's already enabled 
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb sudo dpkg -i amazon-ssm-agent.deb # install 
sudo systemctl enable amazon-ssm-agent # enable the agent 
sudo systemctl status amazon-ssm-agent # check if it's working OK

Alternatively if the above doesn't work, here's the snap installation:

Install SSM Agent on Ubuntu Server 22.04 LTS, 20.10 STR & 20.04, 18.04, and 16.04 LTS 64-bit (Snap) - AWS Systems Manager

1. Add the IAM role to the EC2 machine that’d allow access to session manager:

AmazonSSMManagedInstanceCore

Make sure you wait a few minutes until the permissions changes.
You may need to restart the session manager agent for this to pick it up instantly.

sudo systemctl restart amazon-ssm-agent
# or 
sudo systemctl restart snap.amazon-ssm-agent.amazon-ssm-agent.service

After this, you should see the instance coming up in
AWS Systems Manager ===> Fleet Manager

Enable the users to log into the EC2 terminal

Grant the following permission to the desired user. Remember to replace the placeholder with your EC2 instance ID.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ec2:REGION:ACCOUNT:instance/INSTANCEID"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeSessions",
                "ssm:GetConnectionStatus",
                "ssm:DescribeInstanceProperties",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:TerminateSession",
                "ssm:ResumeSession"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:session/${aws:userid}-*"
            ]
        }
    ]
}

At this point your users can now access the terminal of your EC2.

However, you'll notice that the default shell is used. To switch it to bash, go to the following:

AWS Systems Manager => Session Manager and add a linux shell profile: /bin/bash

You'll now be greeted with this: ssm-user@ip-172-31-4-102:/usr/bin$

Guide for users to access terminal

what you need

1. AWS CLI

2. AWS user credentials

For of all you need to install AWS CLI:
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions

run aws configure to bring up the interactive setup where you can add your AWS credentials and region.

Alternatively, you can find your credentials and config file and just modify it manually.
In Windows it's in c:\Users\username\.aws\ , for other operating systems, you can find it here: https://docs.aws.amazon.com/sdkref/latest/guide/file-location.html

You also need to install the session manager plugin for your PC:
https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html

Once you've done it, you can access the EC2 terminal by :

aws ssm start-session --target i-01fc8012335d9639b

This gets you into your EC2 machine.

In this guide I'll explain how to send data from a NON-AWS linux server to Cloudwatch to track disc space.

This guide though doesn't include creating alerts based on the metric.
This guide will also omit creating guide on basic IAM policy creation.

Configure CWAgent

Download the latest version:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

Install the agent

sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Create a file here. This will be your configuration file the agent will consume.

sudo vim /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Here's a simple example config file.

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "metrics_collected": {
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/"
        ]
      }
    }
  }
}

Permissions

If there's already an AWS user set up on your Linux machine, you can use that.

Your AWS credentials are in ~/.aws/credentials , and your config is in ~/.aws/config

The AWS credentials file should look something like this:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_key

[default] is the profile name. Feel free to - alternatively - duplicate the above and add another profile below with different access_key_id and secret_access_key, like so:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_key
[CWagent]
aws_access_key_id = your_CWagent_access_key_id
aws_secret_access_key = your_CWagent_secret_key

Your config file mentioned earlier should look something like this:

[default]
region = eu-west-1

Similarly, you can duplicate the profile and add a different region.

If you don't have these files, you're missing AWS CLI. You can follow the installation steps to download and install it: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

The above mentioned profiles with their respective access_keys need to correspond to an IAM user with appropriate permisson.

The permission you want to give to that user in order to create logs is :
https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchAgentServerPolicy.html

That's the only permission it requires.

Once you have the permissions and you've configured AWS CLI with the above, with an appropriate profile, you need to modify a file in the agent config:

sudo vim /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

You'll see something like this. Uncomment the [credentials] and add your own path and profile:

That's all the configuration!

Once done, you can start CWagent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

This will start CWagent with the appropriate config file you crafted above.
At some point CWagent "consumes" this file and creates a .toml file from it, so don't be surprised if it disappears.
You should also see some commands running in your terminal as it starts up.

You can then check the status after that with:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m onPremise -a status

which should have an output like this:

You may see "stopped" or "unconfigured", which then indicates that's something wrong with your config file. In that case, re-create the json file above and start CWAgent once again.

You can also check the logs:

cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Which usually tells you what's the issue.

Done!

Once this is done, you can go to Cloudwatch metrics => CWagent

AWS offers the option to generate permissions based on Cloudtrail logs.

That's great for management events, however it doesn't work well for data events. But why?

AWS' policy generation gives you nothing:

And Cloudtrail doesn't record data events in the Event History tab (only Management events).

So when would you even need this?
Here's the specific use case:

"I want to find out what S3 buckets a user accesses and what kind of actions it requires, so I can generate the minimum amount of IAM permissions for the user/users"

How-to

1. You need to have Cloudtrail log already set up. I'll not cover it in this guide (maybe later), but once you have it, let it run for a few days/weeks to allow some actions to be generated. These logs will be saved in an S3 bucket. Keep a note of it.

2. Go to Cloudtrail => Event history and click on Create Athena table

3. This will generate you a create table SQL command, which for me was wrong. Make sure you select a Storage location is set to the bucket that holds your data events and make sure on the bottom the LOCATION is correct. (it wasn't for me)

4. if it's correct, go ahead and create a table, then go to Athena

5. if not, then copy the create table SQL, go to Athena and try to create a table there pasting in your SQL query with the modified LOCATION property (unfortunately you can't edit the query from Cloudtrail 🤷‍♂️

6. once there, you can use this SQL to give you the buckets and the actions that are executed on it:

    SELECT 
        DISTINCT 
        split_part(resource.arn, '/', 1) AS bucket, 
        eventname AS action
    FROM 
        "default"."bucketxzy_cloudtrail_logs_s3_events" 
    CROSS JOIN 
        UNNEST(resources) AS t(resource)
    WHERE 
        useridentity.arn = 'arn:aws:iam::123456789012:user/john' 
    ORDER BY 
        bucket, action;
   

7. which will result in something like this:

   

8. Once you have this, you can generate an IAM policy from them, which will look something like this (you'd need to find the appropriate permission for each action above):

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:DeleteObject",
                    "s3:GetObject",
                    "s3:ListBucket",
                    "s3:PutObject"
                ],
                "Resource": [
                    "arn:aws:s3:::your_bucket/*"
                ]
            }
        ]
    }
   

Further considerations

Now the above gives you an overview of what buckets a user is accessing, but you can take this further. As an example:

• you can modify the SQL to bring back all users rather than just one

• you can request the full path to the object that's accessed, creating an even more granular permission set.

Additionally, please note, none of these services are free. Athena, S3 and Cloudtrail all cost money.