Introduction to Smart Data Frameworks

This video is a high-level introduction to Smart Data Frameworks from Smart Associates.

Smart Data Frameworks is an off the shelf solution for enterprises looking to migrate between almost any database platform to almost any other database platform. It has been developed to allow faster, cheaper and better data integration with rapid return on investment.

SDF is a flatpack application, using container technology. It contains a number of pre-installed software components. It comes in 2 parts – an open source UI and the ETL framework, which is a commercial overlay.

The UI allows for interactive error handling and recovery so is very much the hands-on tool for migrating data. The ETL framework comprises the migration wizard and the migration dashboard with richer functionality for setting up and running complex data migrations automatically.

Smart Data Frameworks Open Source UI

The top screen is the DB Browser which is a generic SQL client with multiple workspaces for you to save the work you have done on supported databases.

ETL job editor is where ETL jobs and groups of jobs are defined. This includes all the steps in a job, their templates and their scheduling.

ETL monitor is a log-viewer and job status monitor. You can use it to look at running jobs or completed jobs. You can see all the SQL that was executed. along with performance statistics, errors and custom logs. So it is effectively the one place for viewing all information relating to jobs that are running or have already run.

The configuration manager is used for creating Database connections and among other things, here you can configure other projects, specific metadata, and the location of plugins.

The next screen is the data loader. This is for doing one-off loads in any of 20 supported databases. A useful feature of the data loader is the ability to load a CSV file without knowing the table DDL and to effectively reverse engineer the table. This is an invaluable tool when doing client demos, being able to work with the data they provide you.

The release manager is for packaging everything that you generate inside the framework. This feature allows you to move entire migration workloads to a different environment, for example from Dev, to Test, to Prod with relative ease. This includes all the objects that are editable, the templates, the jobs, and all of the metadata which you package into an individual release object. Release manager has been designed around GIT workflow and comes with rollback functionality and supports PTI workflow with Github interactivity.

1st run wizard pop ups when you run the app for the first time. It introduces the first couple of screens that you will need to use to get a functioning environment. You can go back later and run the wizard or see the documentation again

The final screen is Broadway configuration manager, which is when clients require the use of an HTML5 client to run SDF.

Smart Data Frameworks Commercial Screens

The Migration Wizard

The Migration Wizard is what we use initially to explore into a source database.

It reverse engineers all of the supported objects - so all of the tables, all of the columns & their types, views, functions, procedures, relationships and due dependencies. It pulls all of these items into a generic schema and then generates target DDLs based on whatever target platform is selected. It has mapping support – so you can redirect database schemas and tables to different targets. You can also modify DDLs using regular expressions, and execute the DDLs. Finally, you can create the ETL jobs for the data migration, and the data validation

The Migration Dashboard

This is what you’d use after initially using the migration wizard. So using the wizard you will set up all the jobs and the metadata and from there, the generated metadata will drive the migration dashboard.

In the dashboard you will see all your tables and their associated jobs, and their statuses. Here you can put tables on hold or change their status. You can filter on different statuses. You can switch to the ETL jobs from the dashboard to make changes, and you can open the ETL monitor to see logs.

You also do MD5 validation from here, so you can drill down into any data validation issue and also troubleshoot migration issues

After you've watched the video, if you have any questions, contact us for further information.