# Responsys (v1)
Information note **Responsys extraction is supported by Stitch**
This integration is powered by [Singer's](https://www.singer.io/) Responsys tap and certified by Stitch. [Check out and contribute to the repo on GitHub](https://github.com/singer-io/tap-responsys).
For support, contact [Support](https://community.qlik.com/t5/Support/ct-p/qlikSupport).
## Responsys feature snapshot
A high-level look at Stitch's Responsys (v1) integration, including release status, useful links, and the features supported in Stitch.
| **STITCH** | | | |
| --- | --- | --- | --- |
| **[Release status](https://help.qlik.com/en-US/stitch/integrations/identify-an-integration-version)** | Released on January 17, 2019 | **Supported by**
the Info icon
Indicates if the integration is supported by Stitch or the Singer community. | [Stitch](https://community.qlik.com/t5/Support/ct-p/qlikSupport) |
| **Stitch plan**
the Info icon
The type of Stitch plan required to use the integration. | Standard | **Supported versions**
the Info icon
Indicates if the integration is supported by Stitch or the Singer community. | n/a |
| **[API availability](https://help.qlik.com/en-US/stitch/developers/stitch-connect/guides/source-destination-reference)**
the Info icon
Indicates if the integration is available as a source in Stitch Connect. | the Success icon
This version of the integration is available via the API.
Available | **Singer GitHub repository** | [singer-io/tap-responsys](https://github.com/singer-io/tap-responsys) |
| **CONNECTION METHODS** | | | |
| **SSH connections**
the Info icon
Indicates if SSH tunnels are supported for connecting this database to Stitch. | the Success icon
This feature is supported for this version of the integration.
Supported | **SSL connections**
the Info icon
Indicates if SSL connections are supported for connecting this database to Stitch. | the Remove icon
This feature is not supported for this version of the integration.
Unsupported |
| **REPLICATION SETTINGS** | | | |
| **[Anchor Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/anchor-scheduling)**
the Info icon
A type of replication scheduling where an integration's Replication Frequency is anchored to a time you select. | the Success icon
This feature is supported for this version of the integration.
Supported | **[Advanced Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/advanced-scheduling)**
the Info icon
A type of replication scheduling that allows you to specify granular start times for data extraction using cron expressions. | the Success icon
This feature is supported for this version of the integration.
Supported |
| **Table-level reset**
the Info icon
Indicates if data can be reset at the table level. If unsupported, data can only be reset at the integration level. | the Remove icon
Table-level resets are not supported for this integration. Resets are only supported at the integration level.
Unsupported | **[Configurable Replication Methods](https://help.qlik.com/en-US/stitch/replication/replication-methods)**
the Info icon
Indicates if Replication Methods can be defined by you for tables in the integration. | the Remove icon
Configurable Replication Methods are not supported for this integration. Stitch will use pre-defined replication methods for tables you set to replicate.
Unsupported |
| **REPLICATION METHODS** | | | |
| **[Log-based Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/log-based-incremental)**
the Info icon
Log-based Incremental Replication is a replication method in which Stitch identifies modifications to records - including inserts, updates, and deletes - using a database's binary log files. | the Remove icon
This feature is not supported for this version of the integration.
Unsupported | **[Key-based Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/key-based-incremental)**
the Info icon
Key-based Incremental Replication is a replication method in which Stitch identifies new and updated data using a column called a Replication Key. | the Success icon
This feature is supported for this version of the integration.
Supported |
| **[Full Table Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/full-table)**
the Info icon
Full Table Replication is a replication method in which all rows in a table - including new, updated, and existing - are replicated during every replication job. | the Remove icon
This feature is not supported for this version of the integration.
Unsupported | | |
| **DATA SELECTION** | | | |
| **Table selection**
the Info icon
The ability to select individual tables for replication in the Stitch interface. If table selection is unavailable, all tables and all the columns they contain will be set to replicate by default. | the Success icon
This feature is supported for this version of the integration.
Supported | **Column selection**
the Info icon
The ability to select individual columns for replication in the Stitch interface. If column selection is unavailable, all columns will be set to replicate by default. | the Remove icon
Column selection is not supported for this integration. All available columns in tables set to replicate will be selected.
Unsupported |
| **[View replication](https://help.qlik.com/en-US/stitch/replication/replicating-database-views)**
the Info icon
Indicates if Stitch supports replicating views from this database integration. | the Remove icon
This feature is not supported for this version of the integration.
Unsupported | **Select all**
the Info icon
Indicates if the integration supports selecting all available tables and columns. | the Success icon
Supported
Supported |
| **TRANSPARENCY** | | | |
| **[Extraction Logs](https://help.qlik.com/en-US/stitch/replication/extractions/integration-extraction-logs)**
the Info icon
Logs detailing the Extraction phase of the replication process for a given integration. Includes error info, should an error occur. | the Success icon
This feature is supported for this version of the integration.
Supported | **[Loading Reports](https://help.qlik.com/en-US/stitch/replication/integration-loading-reports)**
the Info icon
Reports detailing the Loading phase of the replication process for a given integration. Includes the number of rows loaded for each table and error info, should an error occur. | the Success icon
This feature is supported for this version of the integration.
Supported |
## Connecting Responsys
### Responsys setup requirements
To set up Responsys in Stitch, you need:
* **Export files that adhere to [Stitch’s requirements](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#verify-connect-data-export-file-configuration).** Files that aren’t set up correctly may not be replicated successfully, or may cause issues during extraction.
### Step 1: Add Stitch's Public Key to your Responsys SFTP server
Stitch uses an SSH tunnel to securely connect to your Responsys SFTP server. This means that to connect successfully, you’ll need to add Stitch’s Public Key to your server.
* [Step 1.1: Retrieve your Stitch public key](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#retrieve-stitch-public-key)
* [Step 1.2: Determine your SFTP server hosting](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#sftp-server-hosting)
#### Step 1.1: Retrieve your Stitch public key
1. On the Stitch Dashboard page, click the **Add Integration** button.
2. Click the **Responsys** icon.
3. The Stitch public key will be at the top of the page that opens:
The Stitch Public Key in the Responsys settings pageThe Stitch Public Key in the Responsys settings page
#### Step 1.2: Determine your SFTP server hosting
Next, you’ll need to add the public key to your SFTP server. The steps for doing this depend on how your SFTP server is hosted:
My server is hosted by Oracle. (This is the default)
Contact Oracle and provide them with the public key to complete this step.
[close](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#collapseoracle-hosted)
My server is self-hosted.
If your SFTP server is self-hosted, or not hosted by Oracle, you’ll need to add the public key to the `authorized_keys` file. This will allow Stitch to conntect to the server using a trusted user.
1. Open a new terminal window and log into your SSH server.
2. Create a group named `stitch`:
```
sudo groupadd stitch
```
3. Create a user named `stitch`:
```
sudo useradd -m -g stitch stitch
```
4. Switch to the `stitch` user:
```
sudo su - stitch
```
5. Create the `.ssh` directory:
```
mkdir ~/.ssh
```
6. Using [`chmod`](https://en.wikipedia.org/wiki/Chmod), alter the permissions of the directory to allow access:
```
chmod 700 ~/.ssh
```
7. Navigate to the `.ssh` directory:
```
cd ~/.ssh
```
8. Using [`touch`](http://www.linfo.org/touch.html), create the `authorized_keys` file:
```
touch authorized_keys
```
9. Set the permissions for `authorized_keys`:
```
chmod 600 authorized_keys
```
10. Import the [Stitch public key](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#retrieve-your-public-key) into `authorized_keys`, replacing `` with the public key. Verify after replacing the key that it is on one line and doesn’t contain line breaks:
```
echo "" >> ~/.ssh/authorized_keys
```
[close](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#collapseself-hosted)
If you skip this step or the Public Key is added incorrectly, the following error will surface when you save the integration in Stitch:
```
"Message from SFTP server: Authentication failed. - Please ensure that the server is configured to accept the public key for this integration."
```
### Step 2: Verify Responsys Connect data export configuration
Connect Data Export Destination Specification pageConnect Data Export Destination Specification page
Stitch’s Responsys integration replicates `.csv` or `.txt` files created as part of a [Connect data export job](https://interact2.responsys.net/interact/help/rifh/en/Content/Connect_WizardDownload.htm). In the Responsys app, jobs are created and managed in the **Manage Connect** page, accessed via **Data > Connect**.
File settings are defined in the **Destination Specification** step of the Connect data export process, which can be seen in the image to the right.
The settings in the **Destination Specification** page must match the required values listed in the table below. This will ensure that data is replicated successfully from Responsys.
| **Setting in Responsys** | **Required value** |
| --- | --- |
| **File to be created** | We recommend using a **date/time prefix OR suffix** for file names, which will result in Responsys creating multiple files.
If a prefix or suffix isn’t used, the export will only ever create a single file. Every new export will replace the entire file, overwriting the data it contains. If this occurs before Stitch runs an extraction job, the overwritten data would be lost and unable to be replicated. |
| **File extension** | **.csv** or **.txt**
Stitch’s Responsys integration doesn’t currently support replicating other file types. |
| **Character set** | Unicode (UTF-8) |
| **File delimiter** | Comma
Stitch’s Responsys integration doesn’t currently support other file delimiters. |
| **Field enclosure** | Double quotes (`"`) |
| **Insert column header as first line** | **Box must be checked**
Stitch’s Responsys integration requires that the first row of every export file be a column header. |
| **Encryption/compression** | **Do not encrypt or compress file**
Stitch’s Responsys does not currently support replicating encrypted or compressed files. |
| **Additional ready file at completion of download** | Select **Create file with record count**.
In the **File extension** field, enter `ready` |
### Step 3: Connect Stitch
In this step, you’ll complete the setup by entering the connection details and defining replication settings in Stitch:
* [Step 3.1: Define the Responsys connection details](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#define-connection-details)
* [Step 3.2: Define the historical sync](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#define-historical-sync)
* [Step 3.3: Create a replication schedule](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#create-replication-schedule)
#### Step 3.1: Define the Responsys connection details
The info you need to complete the remaining fields depends on how your SFTP server is hosted:
My server is hosted by Oracle. (This is the default)
If your server is hosted by Oracle, you can find its connection details in the Responsys app by navigating to **Data > Connect > Destination Connectivity**. If you have multiple file locations, click the one you want to connect to Stitch.
Oracle-hosted SFTP server connection detailsOracle-hosted SFTP server connection details
The image on the right shows an example of the connection details in Responsys.
Fill in the remaining fields in Stitch as follows:
* **Host**: In Stitch, enter the value from the Responsys **Server** field. This will likely be `files.responsys.net`
* **Port**: Leave this as `22`.
* **Username**: In Stitch, enter the value from the Responsys **User name** field. In the example, this is `demo_scp`
* **Path**: In Stitch, enter the value of the Responsys **Directory path** field, or the file server path where completed Responsys export files are stored. In the example, this is `download`
[close](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#collapsefields-for-oracle-hosted)
My server is self-hosted.
If your server is hosted by Oracle, you can find its connection details in the Responsys app by navigating to **Data > Connect > Destination Connectivity**. If you have multiple file locations, click the one you want to connect to Stitch.
Self-hosted SFTP server connection detailsSelf-hosted SFTP server connection details
The image on the right shows an example of the connection details in Responsys.
Fill in the remaining fields in Stitch as follows:
* **Host**: Enter the host address (endpoint) used by the SFTP server.
* **Port**: Enter the SSH port used by the SFTP server. This is usually `22`.
* **Username**: Enter the name of the user you created in [Step 1.2](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#sftp-server-hosting). In the example, this is `stitch_user`
* **Path**: Enter the file server path where completed Responsys export files are stored.
[close](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#collapsefields-for-self-hosted)
#### Step 3.2: Define the historical sync
The Sync Historical Data setting defines the starting date for your Responsys integration. This means that data **equal to or newer than this date** will be replicated to your data warehouse.
Change this setting if you want to replicate data beyond Responsys’s default setting of **1 year**. For a detailed look at historical replication jobs, check out the [Syncing Historical SaaS Data](https://help.qlik.com/en-US/stitch/replication/syncing-historical-saas-data-resetting-replication-keys) guide.
#### Step 3.3: Create a replication schedule
Information note Replication schedules affect the time Extraction begins, not the time to data loaded. Refer to the [Replication Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling) documentation for more information.
In the **Replication Frequency** section, you’ll create the integration’s [replication schedule](https://help.qlik.com/en-US/stitch/replication/replication-scheduling). An integration’s replication schedule determines how often Stitch runs a replication job, and the time that job begins.
Responsys integrations support the following replication scheduling methods:
* [Replication Frequency](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/replication-frequency)
* [Anchor Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/anchor-scheduling)
* [Advanced Scheduling using Cron](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/advanced-scheduling) (Advanced or Premium plans only)
To keep your row usage low, consider setting the integration to replicate less frequently. See the [Understanding and Reducing Your Row Usage guide](https://help.qlik.com/en-US/stitch/getting-started/understanding-stitch-usage) for tips on reducing your usage.
### Step 4: Select data to replicate
Information note **Is an object missing or not replicating?** Verify that the object meets the requirements for [selection](https://help.qlik.com/en-US/stitch/replication/extractions/data-selection/setting-tables-and-columns-to-replicate#selection-requirements) and [replication](https://help.qlik.com/en-US/stitch/replication/extractions/data-selection/setting-tables-and-columns-to-replicate#replication-requirements).
The last step is to select the tables you want to replicate. [](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#schema)
**Note**: If a replication job is currently in progress, new selections won’t be used until the next job starts.
* Selection options
* Individual tables
* All tables
For Responsys integrations, you can select:
1. **Individual tables**
2. **All tables**
Click the tabs to view instructions for each selection method.
1. In the Integration Details page, click the **Tables to Replicate** tab.
2. Locate a table you want to replicate.
3. Click the **checkbox** next to the table’s name. A blue checkmark means the table is set to replicate.
4. After you set a table to replicate, a page with the table’s columns will display. De-select columns if needed. **Note**: Responsys tables replicate using Key-based Incremental Replication. Refer to the [Replication](https://help.qlik.com/en-US/stitch/integrations/databases/responsys#replication) section for more info.
5. Repeat this process for every table you want to replicate.
6. Click the **Finalize Your Selections** button at the bottom of the page to save your data selections.
Information note **Important**: Using the **Select All** feature will overwrite any previous selections. However, selections aren’t final until **Finalize Your Selections** is clicked. Clicking **Cancel** will restore your previous selections. Refer to the [Select All guide](https://help.qlik.com/en-US/stitch/replication/extractions/data-selection/setting-tables-and-columns-to-replicate#set-all-data-to-replicate) for more info about this feature.
1. Click into the integration from the Stitch Dashboard page.
2. Click the **Tables to Replicate** tab.
3. Navigate to the table level, selecting any databases and/or schemas that contain tables you want to replicate.
4. In the list of tables, click the box next to the **Table Names** column.
5. In the menu that displays, click **Track AllTables and Fields (Except Views)**:
The Track AllTables and Fields (Except Views) menu in the Tables to Replicate tabThe Track AllTables and Fields (Except Views) menu in the Tables to Replicate tab
6. Click the **Finalize Your Selections** button at the bottom of the page to save your data selections.
### Initial and historical replication jobs
After you finish setting up Responsys, its **Sync Status** may show as **Pending** on either the Stitch Dashboard or in the Integration Details page.
For a new integration, a **Pending** status indicates that Stitch is in the process of scheduling the initial replication job for the integration. **This may take some time to complete.**
Information note
**Initial replication jobs with Anchor Scheduling**
If using Anchor Scheduling, an initial replication job may not kick off immediately. This depends on the selected Replication Frequency and Anchor Time. Refer to the [Anchor Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling#job-scheduling-and-anchor-times) documentation for more information.
#### Free historical data loads
The first seven days of replication, beginning when data is first replicated, are free. Rows replicated from the new integration during this time won’t count towards your quota. Stitch offers this as a way of testing new integrations, measuring usage, and ensuring historical data volumes don’t quickly consume your quota.
Information note **Replication will continue after the seven days are over**. If you’re no longer interested in this source, be sure to **pause** or **delete** the integration to prevent unwanted usage.
## Responsys replication
While data from Responsys integrations is replicated using [Key-based Incremental Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/key-based-incremental), the behavior for this integration differs subtly from other integrations.
The table below compares Key-based Incremental Replication and [Replication Key](https://help.qlik.com/en-US/stitch/replication/replication-keys) behavior for Responsys to that of other integrations.
| | **Responsys** | **Other integrations** |
| --- | --- | --- |
| **What's replicated during a replication job?** | The entire contents of a modified file. | Only new or updated rows in a table. |
| **What's used as a Replication Key?** | The time a file is modified. | A column or columns in a table. |
| **Are Replication Keys inclusive?** | **No**. Only files with a modification timestamp value greater than the last saved bookmark are replicated. | **Yes**. Rows with a Replication Key value **greater than or equal to** the last saved bookmark are replicated. |
## Responsys Schema
Responsys tables will contain the same columns that the source files contains. These tables will also contain [common system columns](https://help.qlik.com/en-US/stitch/replication/loading/system-tables-columns), as well as columns specific to Responsys integrations:
| **\_sdc\_source\_file** | **Applicable only to Intacct and Responsys integrations**, this column contains the file path on the SFTP server of the table’s source file. |
| --- | --- |
| **\_sdc\_source\_lineno** | **Applicable only to Intacct and Responsys integrations**, this column contains the row number for the record in the source file. |