Introducing Qlik Cloud Data Integration
You can deliver data ready for consumption to Qlik Cloud or to cloud data warehouses, such as Snowflake, Google Cloud BigQuery, and Azure Synapse Analytics with Qlik Cloud Data Integration. Data sources can be on-premises or in the cloud. The data can be kept up-to-date without manual intervention using CDC (Change Data Capture) or batch technologies, such as scheduled reloads. You can create a data pipeline and perform fit-for-purpose transformations and create data marts.
You can access Qlik Cloud Data Integration home by selecting Data Integration from the launcher menu ().
For more information about the architecture of Qlik Cloud Data Integration, see Dataset architecture in a cloud data warehouse.
Subscription options
Qlik Cloud Data Integration subscriptions are based on a capacity model with the volume of Data Moved as the primary value meter.
Qlik Cloud Data Integration is available in subscription options from three tiers: Standard, Premium, and Enterprise. The higher editions provide more advanced data sources and transformations. All subscriptions include Qlik Cloud Analytics Standard.
For more information about subscription options, see Qlik Cloud Data Integration subscription options.
Data spaces
Data spaces are governed areas of your Qlik Cloud tenant that are used to create and store data projects. Inside the space, you can also create new data connections using connectors, and manage access to Data Movement gateways. All data assets will be created in the space of the data project that they belong to.
For more information, see Working in spaces in Qlik Cloud Data Integration.
Data projects
A data project is where you create your data integration flow, using data tasks. The data project is associated with a data platform that is used as target for all output. You can create a data project with either of the following use cases:
-
Data pipeline
Create a simple linear pipeline, or a complex pipeline consuming several data sources and generating many outputs.
-
Replication
Replicate data from supported data sources to any supported target, or land data to a data lake.
Data task
A data task is the main unit of work in a data project. You can create data tasks of the following types in a data project. You create a new data task by clicking on Add new in the top bar, and then clicking the appropriate task.
Data tasks in data pipeline projects
-
Landing
Copy data from a data source to a landing area. Data sources can be on-premises or in the cloud. The landing area can be a cloud target, or an Amazon S3 data bucket (only when creating QVD datasets).
You can keep data up-to-date without manual intervention by using CDC, or by performing full loads that are scheduled to reload periodically.
-
Registered data
Register data that already exists on the data platform. This lets you use data that is onboarded with other tools than Qlik Cloud Data Integration, for example, Qlik Replicate.
-
Storage
Create ready to consume datasets in a cloud data warehouse, or in Qlik Cloud, from the data copied by the landing data task. The datasets can be kept up-to-date with the landing data without manual intervention.
-
Transform
Create reusable data transformations based on rules and custom SQL as a part of your data pipeline. You can perform row-level transformations and create datasets that are either materialized as tables, or created as views that perform transformations on the fly.
-
Data mart
Create data marts to leverage your Storage data tasks or Transform data tasks. You can create any number of data marts depending on your business needs. Ideally, your data marts should contain repositories of summarized data collected for analysis on a specific section or unit within an organization.
Data tasks in replication data projects
-
Replication
Replicate data from supported data sources to any supported target.
-
Lake landing
Land data to a data lake.
Monitoring your data tasks
You can monitor the status and progress of your data tasks with monitor views. A monitor view lets you view the status of all data tasks in the tenant, or a subset of data tasks based on a filter. You can create several views to monitor different aspects of your data pipelines. For more information, see Monitoring and operating your data tasks.
Connections
Data connections are used to let data tasks access data sources, external storage and target platforms for data delivery and push-down transformations.
Viewing your connections
Click Data connections on the left to view all your data connections.
-
You can edit connections that you own.
Click ... and then Edit.
-
You can test a connection.
Click ... and then Test connection.
-
You can delete a connection.
Click ... and then Delete.
Creating a connection
The are several ways of creating connections:
-
Click Add new in the top bar, and then click Data connection.
-
Click Create new in data task setup wizards where you select a connection.
-
Click Create connection in Data connections view.
You can filter connectors by:
-
Category
Data warehouse, Cloud storage, Database and Application.
-
Type
Source or Target.
You can also select from recently used connectors.
You will need to select which type of data source, and then enter address and authentication information.
See also: