Connecting to private AWS resources, such as giving psql commands to a Postgres RDS database, is usually done with a bastion host. But bastion hosts can be expensive and difficult to manage. Here I’ll show you how to do it inexpensively, conveniently, and securely with Amazon EC2 Instance Connect.

AWS security best practices recommend that the database be accessible only on a private IP address range to ensure network isolation. In other words, it is recommended as best practice to host your database such as RDS in a private subnet.

One way to access private resources is via a bastion host.

However, bastion hosts also add cost, as they often run 24x7 just to give occasional access, and incur the overhead of managing a one-off instance, including periodic security patches, and exception rules in the network firewall. Poorly managed bastion hosts could increase your system’s attack surface. For example, a cybersecurity threat from a long-lived SSH key to access the bastion host that you pre-shared with your business partner.

The new feature of Amazon EC2 Instance Connect Endpoint announced on June 13, 2023, offers an additional mechanism to access the workloads in a private subnet without the need of maintaining your bastion host.

Table of Contents

· Table of Contents
· Overview
· Step-by-step
· Pre-requisite
· Create a Security Group for the EC2 Instance Connect Endpoint
· Create EC2 Instance Connect Endpoint
· Configure the IAM policy for the connection
· Container DevOps connecting to the private RDS
· Connect to the RDS via EC2 Instance Connect Endpoint
· Troubleshooting
· InvalidParameter from aws ec2-instance-connect open-tunnel
· Conclusion

Overview

The overall architecture of EC2 Instance Connect connecting to RDS in Private Subnet

1. Users connecting from the internet to the EC2 Instance Connect Endpoint Service using AWS CLI  aws ec2-instance-connect open-tunnel ;
2. The EIC connections will be subject to IAM permissions and optionally logs;
3. The connections from the EC2 Instance Connect Endpoint are required to allow inbound by the workload Security Group
4. The EC2 Instance Connect Endpoint will be connecting to the RDS via Private IPs

Step-by-step

Pre-requisite

1. To demonstrate the capability of connectivity without the Internet, make sure your RDS subnet route table does not have a route to an Internet Gateway.
2. Take note of the RDS Security Group, for example,  sg-012345 .
3. Take note of the RDS VPC, for example,  vpc-012345 .
   We will create the EC2 Instance Connect Endpoint in the same VPC as illustrated in the diagram above.
4. Take note of the private subnets that the RDS is hosted, for example,  subnet-0abc101  and  subnet-0abc102 .
   We will create the EC2 Instance Connect Endpoint in one of the subnets.
5. Take note of the private IPs of the RDS, for example,  10.0.128.61
   We will be connecting from the Internet to the EC2 Instance Connect using AWS CLI, and as of AWS CLI version 2.12.1, the  open-tunnel  accepts only IP addresses to open a tunnel.

Private IP of RDS instance can be found from from “Network interfaces” in the EC2 Dashboard, filter by “Description=RDSNetworkInterface”.

If the results return multiple entries, narrow down by the VPC.

Select the relevant Network Interface, the Private IPv4 address is in the panel of “IP addresses”

Search for the Private IP of RDSNetworkInterface

With the information of the private AWS resources that we want to connect with the EC2 Instance Connect Endpoint, we will first create a Security Group for the EC2 Instance Connect Endpoint, followed by the EC2 Instance Connect Endpoint itself.

Create a Security Group for the EC2 Instance Connect Endpoint

We will create a stand-alone Security Group for the EC2 Instance Connect Endpoint. The Security Group has no inbound rules but one outbound rule to connect to the RDS tagged with the RDS Security Group.

• Go to “Create security group” in the EC2 console using this link https://console.aws.amazon.com/ec2/v2/home#CreateSecurityGroup.
• Enter meaningful “Security group name” and “Description”.
• Select the VPC for the RDS. In this example, it will be  vpc-012345
• Leave the “Inbound rules” empty
• In the “Outbound rules”, configure an outbound rule with the following:
  a. Type: Customer TCP
  b. Port range: RDS connectivity port, such as 5432 for Postgresql
  c. Destination: Select the Security Group of the RDS. In this example, it is  sg-012345
  d. Description: Outbound connection from EC2 Instance Connect Endpoint to RDS
• Select “Create Security Group” at the bottom right of the page
• A stand-alone Security Group will be created, for example,  sg-0abcde

Create a new Security Group for the EC2 Instance Connect Endpoint

Create EC2 Instance Connect Endpoint

1. Go to the Create endpoint page in the VPC using this link https://console.aws.amazon.com/vpc/home#CreateVpcEndpoint
2. Enter a meaningful Endpoint name
3. In the Service Category, select “EC2 Instance Connect Endpoint”
4. In the VPC list, select the RDS VPC. In this example, select  vpc-012345
5. In the “Security groups”, select the Security Group created for the EC2 Instance Connect Endpoint. In this example, it is  sg-0abcde
6. Select one of the subnets that RDS hosted. In this example, it is either  subnet-0abc101 or  subnet-0abc102
7. Select “Create endpoint”

Create EC2 Instance Connect Endpoint and use the EC2 Instance Connect Endpoint Security Group

Configure the IAM policy for the connection

With the EC2 Instance, Connect Endpoint created and the connectivity in place with Security Group tagged, we will proceed to create the least privileged IAM user that can use the connection.

1. Go to Create Policy in the IAM console following this link https://console.aws.amazon.com/iamv2/home#/policies/create?step=addPermissions
2. Switch to JSON Policy Editor, and enter the policy below. Replace the EC2 Instance Connect Endpoint Resource with the relevant EC2 Instance Connect Endpoint ARN, and replace the variables in angle brackets to your environment:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2-instance-connect:OpenTunnel",
            "Resource": "arn:aws:ec2:<AWS Region>:<AWS Account>:instance-connect-endpoint/eice-<EICE ID>",
            "Condition": {
                "NumericEquals": {
                    "ec2-instance-connect:remotePort": "5432"
                },
                "IpAddress": {
                    "ec2-instance-connect:privateIpAddress": [
                        "<CIDR of subnet-0abc101>",
                        "<CIDR of subnet-0abc102>"
                    ]
                }
            }
        },
        {
            "Sid": "Describe",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceConnectEndpoints"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Container DevOps connecting to the private RDS

Now that we have the EC2 Instance Connect and the least privileges policy associated with the IAM user, we can test the connection.

We will connect using a Postgres client from a Postgres container image, following the Container DevOps approach.

A Container DevOps helps to achieve reliability, repeatability, and security:

• Reliability through container image packaged with the version of the tools, dependencies, and settings for consistency.
• Repeatability through the same container image used in the CICD pipeline and the same error can be reproduced in DevOps debugging environment.
• Security by conducting container image security scanning, and any automation scripts scanned with SCAT, before being used across the DevOps team and in CICD automation.

Connect to the RDS via EC2 Instance Connect Endpoint

1. Use the AWS profile for the IAM user that is attached to the policy created above
2. Run a Postgres Client from Docker to connect to the private RDS with the command docker run -it — rm — network=bridge postgres psql -h host.docker.internal -U <db user>
3. Enter the password for the  db user
4. If all goes well, you should have the  postgres  prompt

 

% docker run -it --rm --network=bridge postgres psql -h host.docker.internal -U postgres 
Password for user postgres: 
psql (15.3 (Debian 15.3-1.pgdg120+1), server 14.7)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, compression: off)
Type "help" for help.

postgres=>

Troubleshooting

InvalidParameter from aws ec2-instance-connect open-tunnel

I encountered the following error when running aws ec2-instance-connect open-tunnel

% aws ec2-instance-connect open-tunnel --instance-connect-endpoint-id <EIC ID> --private-ip-address <RDS Endpoint DNS> --local-port 5432 --remote-port 5432
Listening for connections on port 5432.


[1] Accepted new tcp connection, opening websocket tunnel.
2023-06-20 23:32:09,666 - awscli.customizations.ec2instanceconnect.websocket - ERROR - {"ErrorCode":"InvalidParameter","Message":"The specified PrivateIpAddress is not valid. Specify a valid IPv4 PrivateIpAddress and retry your request."}

AWS_ERROR_HTTP_WEBSOCKET_UPGRADE_FAILURE: Failed to upgrade HTTP connection to Websocket.

This error is due to the implementation of WebSocket used by AWS CLI requiring a private IP. Resolve the RDS Endpoint to the internal IP as the parameter of of --private-ip-addressfor aws ec2-instance-connect open-tunnel

Conclusion

EC2 Instance Connect is a much-improved way to connect to private resources such as RDS. As a managed service, it adds security, saves money, and reduces management overhead.

Most importantly, it is easy to use!