RDBMS based ingestion

Data Pipes allows users to ingest data from RDBMS sources. Please follow the steps as shown below.

1. Click on the pipeline icon on the left hand side of the Data Pipes portal. 

2. On the ingestion pipelines page, click on the plus icon to create a new pipeline.

   

3. Click on configure source. In the select source category, select RDBMS

4. You can click on the dropdown to view the existing sources or click on to set up a new source.

5. For the RDBMS you wish to connect, please provide the connection information as per the screenshot below. Please note that for Oracle DB, an additional option to use either an SID or Service Name for connectivity is provided. Once you have entered the connection details, select validate + add source.

   

6. Once the source is configured, click on the configure destination option and select the default destination of the data lake.

7. RDBMS ingestion supports 3 modes at present:

   1. FULL: In this mode, the pipeline will run at a specified frequency and refreshes the entire data in a batch

   2. INCREMENTAL: In this mode, the pipeline will run at a specified frequency, pick the incremental data and append it to the existing catalog table. For this the input source should support filtering results by timestamp or date.

   3. CDC (Change Data Capture): It is a process where changes in the source (inserts/updates/deletes) are captured instantly and propagated to the Data Lake with low latency.

8. Once the replication mode is selected, click on configure pipeline to provide details like - pipeline name, domain name (where the data is to be loaded), table name.

9. Click on the play button to configure verify the configuration. 

10. Enable the scheduler and set the frequency if you want to set up recurring ingestion. If the scheduler is not enabled the pipeline will ingest the data only once.

    

11. The next step is to select the schemas and tables that need to be ingested. Please select the list of tables and click on “PREPARE FOR REPLICATION”. This would begin the ingestion process.

12. Once you create the pipeline after verifying all the details, the “start replication” button will appear at the bottom. Click on it to begin the ingestion.