Demo - Netezza to Databricks Migration
This session demonstrates how Smart Data Frameworks (SDF) can be used to migrate and continuously replicate data between databases, using Netezza (source) and Databricks (target) as the example.The walkthrough begins with setting up the source (Netezza) and target (Databricks) connections, along with an AWS S3 bucket as the staging area. Using SDF’s migration wizard, the process includes:
- Schema Migration – Extracting table definitions from Netezza, generating the necessary DDL, and creating equivalent schemas and tables in Databricks.
- Initial Data Migration – Exporting data from Netezza, staging it in S3, and loading it into Databricks. The example dataset (~9.8M rows) was successfully migrated with validation confirming identical row counts across source and target.
- Continuous Replication Setup – Converting the migration job into a replication job with configurable scheduling (e.g., near real-time or batch intervals).
- Replication Demonstration – Validating replication for inserts, deletes, updates, and truncations. Changes made in the source were accurately reflected in the target, including:
- Insert of 280k records.
- Delete operations reducing dataset size.
- Bulk updates to values.
- Full table truncation.
The session concludes by highlighting SDF’s flexibility to migrate and replicate data not only to Databricks but to any supported target database. This enables both initial data synchronization and ongoing replication to maintain consistency across environments.
After you’ve watched the video, if you have any questions, contact us for further information.