Data Discovery

Data Pipes provides a rich catalog interface to search for tables and datasets. Each table ingested in the data lake is automatically registered in the catalog. Tables created by third party applications can be registered using the Data Pipes API or SDK. 

The Data catalog search screen works like google for your Data Lake: Data users enter keywords, and the Catalog will show any table relevant to that keyword. Under the hood, Data Pipes uses table and column descriptions, as well as the column names themselves to match the user entered keyword. 

The search is lightning fast and shows a description of the table and keyword highlights to help the user quickly assess which table they want to inspect. On a click on the results, users will be brought to the table details page, described in the next section.

On the Data Pipes Portal, click on the Catalog Icon (Search) on the left , you will land on the Data Catalog.

Starred Tables

Data users can also “star” tables from the Catalog. Starred tables will then always be shown on their Data Pipes home page, in this way their favorite tables are very accessible.

Tags

Domain owners can assign tags to individual tables. These can be used to group together tables used in the same project, or any other custom classification. Tags are searchable.