Overview

Architecture

Data Pipes is a cloud-native data management platform to help analytics and IT teams control, trust, and democratize their data by automating data practices.

Data Pipes provides a rich Graphical User Interface for different personas such as Data Administrators, Domain Owners and Data Users to perform their roles as part of Data Management. These include integration, ingestion, transformation, curation, governance and distribution features which sit on a single, user intuitive platform.

Key Features

Data Ingestion

Data Pipes comes with an extensive list of pre-built data source connectors for on-premises and cloud databases, SaaS applications, documents, NoSQL sources, and more to quickly load data into your cloud data environment.

From a no-code interface, users can create data ingestion pipelines from different data sources into the cloud data lake and perform data preparation. Any sensitive data is automatically identified and data owners can select how to secure it before storing into the data lake. Data Pipes supports both column level and row level security.

Data Catalog

At the core of Data Pipes is the Data Catalog that provides users a platform for data discovery. It allows users to manage, organize and search for data to accelerate data-driven business decision. Data owners can create and edit the metadata of their datasets so consumers can better understand what they’re searching for. 

Users can consume the data using Amazon QuickSight and SageMaker, or any JDBC compliant tool of their choice.

Data Governance

Data Pipes provides a powerful suite of tools to control access to data for users. Authorized users can create rules that grant users access to data based on the role and department. These rules provides granular access control to specific tables, and individual columns and rows.

Individual columns in Data Pipes can be marked as classified, this allows rules to be easily created at scale to deny access for unauthorized users to classified data. Tables containing classified data can also be completely removed from the Catalog view. 

In addition, Data Pipes automatically keeps an audit trail of data access and permission changes by the users in real-time. These trails can be easily exported for any audit purposes.