Publish: add target

Publish target definition configures connection and database credentials for each publish operation.

This process defines Target Type, Target Name, Connection URI and other properties relevant to the target type.

Target definition window displays with fields specific to the type of target being defined.

For example, Hadoop targets require a Hive URI as Hadoop objects necessitate the creation of a Hive table on top of the HDFS file.

All other file-based target types require a URI: File, HDFS, FTP, S3, ADLS) along with user-defined file specifications.

OpenConnector targets require a Custom Script Command and accompanying arguments.

Publish target types

Qlik Catalog provides the following publish target types:

• Amazon – S3 and Redshift: Object-based massively scalable, cloud-based data warehouse storage
• File on ADLS Storage: File-based targets published to MS cloud solution Azure Data Lake Store
• File on FTP Server: File-based targets published to an FTP server over an enterprise network
• File on Qlik Catalog Server: File-based targets are published to a directory on the local host
• File on S3 Storage: File-based targets published to scalable cloud storage
• File on SFTP Server: File-based targets published to an FTP server over an enterprise network via secure SSH tunnel
• File on WASB Storage: Objects publish to Azure's object storage option (Windows Azure Storage Blob) in containers with data in form of blobs
• File on WASBS Storage: Objects publish to Azure's object storage option (Windows Azure Storage Blob) in containers with data in form of blobs, using SSL certificates for improved security
• Hadoop – HDFS and Hive:  Entities are registered to Hive tables on top of HDFS
• HDFS: File-based targets published to Hadoop file system
• Open Connector/ RDBMS: File-based targets published to RDBMS via Sqoop transfer
• Qlik Cloud Services – S3: On-premises QVD data is published to pre-configured AWS S3 buckets accessible to Qlik Cloud

Add Amazon – S3 and Redshift target

Amazon – S3 and Redshift publish target requires an S3 target URI with Access Key and Secret Access Key and Redshift target URI.

File Format specifications are not required.

Add File onADLS storage, FTP server, Qlik Catalog server, S3 storage, SFTP server, WASB storage, WASBS storage publish target

These target types are addressed together here because the specifications are very similar. Enter the following values for file-based targets:

Test Connection and Save the target.

File on SFTP Server notes

ADLS Gen2 notes

ADLS Gen2 details for a Publish target:

• ADLS URI:
  syntax: abfss://<container>@<storage_account_name>.dfs.core.windows.net/
  example: abfss://abfstest@exadlsgen2test1.dfs.core.windows.net/
  
• Username:<storage_account_name>
• Password: Access key (available in ADLS account settings)

• The storage account in MS Azure must be created with specific options. On the Advanced tab in storage account settings: In the Security section, Enable storage account key access must be checked; and in the Data Lake Storage Gen2 section, Enable hierarchical namespace must be checked.

Add HADOOP – HDFS and Hive target

HADOOP – HDFS and Hive is the only publish target type requiring a Hive URI as Hadoop objects necessitate the creation of a Hive table on top of the HDFS file.

Example: Add Target for Hadoop-HDFS and Hive>

Add File on HDFS Storage target

Field Delimiter: \t

Record Terminator: \n

Header: none

Example: Add Target for File on HDFS Storage

Add Open Connector/ RDBMS target

Open Connector/ RDBMS Target type replicates data from HDFS shipping location to RDBMS targets through Sqoop CLI tool. Sqoop provides a command line interface where users can provide basic information like source (Podium %shippingLocation), destination (%prop.jdbcString), table name (%prop.table) and any other authentication details (ex. %prop.username) required.

In addition to standard file based target specs, Open Connector targets require a Custom Script Command and the entry of argumentsto be passed through the command.

Save the Target.

Example: Add Target for (Open Connector/ RDBMS)

Bash SQOOP export script:

#!/bin/bash
 
#postgres
#jdbc:postgresql://<localhost>:5432/username
#postgres
#sqoop_export_test
#/podium-mydir/sqoopexporttest/sqoopExportTest.txt
set -x
 
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
 
jdbcString=$1
username=$2
table=$3
receivingDir=$4
additional=$5
read password
 
/usr/bin/sqoop export \
--verbose \
--connect "$jdbcString" \
--username "$username" \
--password "$password" \
--table $table \
--fields-terminated-by '\t' --lines-terminated-by '\n' \
--export-dir "$receivingDir/*" \
$additional
 
exit $?

Add Qlik Cloud Services – S3 target

When setting up an Qlik Cloud publish target, users define the buckets accessible to Qlik Cloud with appropriate Access Key and Secret Access Keys.

Example: Edit Target for Qlik Cloud Services-S3