Data Pipes CloudFormation template description

Data Pipes deployment is done using a CloudFormation template (cf. our setup guide for more detail).

The CloudFormation template is creating the following component within your AWS account.

Many component are built based on the deployment configuration chosen.

For 2 AZs standard deployment:

1 VPC in /21

8 Subnets in /24 (and all the associated route tables):

• 2 for core datapipes subnet

• 2 for database subnet

• 2 for cache subnet

• 2 for public subnet

1 NLB

1 internet gateway

1 EKS cluster with:

• 1 control plan

• 3 nodes

• 11 Datapipes services deployed individually as Helm chart:

  • audit-service

  • catalog-service

  • core-service

  • frontend

  • gatekeeper

  • governor-service

  • ingestor-service

  • notification-service

  • rbac-service

  • sql-listener

• 4 infrastructure Data Pipes service deployed as Helm chart:

  • bottlerocket updater (if bottlerocket is chosen as EKS node OS)

  • certificat manager

  • cluster auto scaler

  • ingress controller

1 PostgreSQL Database

1 SQS topic

1 elasticache redis instance

3 S3 bucket

6 VPC endpoints:

• Athena

• S3

• SQS

• Glue

• Lakeformation

• Macie2

• Databrew

1 hosted zone with all the associated records

All the associated IAM policies and users

20 lambda function

21 Secret within AWS Secret Manager