Running Flows

Overview

Once you have set up your flow in Design Flow mode, you can move on to the Runtime mode. In this mode, you will execute the transformation flow on your entire dataset by simply clicking the "Run" button. As your dataset can be large and dependent on the complexity of the flow you created, running the transformation over the whole data can take significant time.

To execute flow:

1. Verify the connections to your data sources and that you have destinations (local files or data warehouses).

   1. Flow cannot be executed without output nodes or charts

2. Press the "Run" button to execute the transformation flow on the entire dataset. We call it “Job”

3. Once a job is done you get a notification and can go to the job either from the Jobs page or from the Flow by pressing the Job button near the right corner of the toolbar.

Where does Tomat execute transforms?

Under the hood, Tomat generates SQL for each dialect agnostic transformation that could be executed on any warehouse or database.

In the Design mode, all transforms are executed locally using the engine and power of your laptop. No data is uploaded online.

In Runtime Tomat supports two types of execution engines: local and data warehouse. The type of execution engine used for the flow depends on the data sources and destinations you have selected:

• If all data sources and destinations are local files, the flow will be executed locally like in design time, but using the whole datasets.

• If any data source or destination is a table in a data warehouse, the flow will be executed in that warehouse. In this case, source local files will be temporarily uploaded to the warehouse and removed after the flow execution is completed. If there are output local files then they will be downloaded and saved on your laptop.

At the moment, Tomat chooses the execution engine automatically. But we are going to support a manual selection of the engine in the near future.