# Amazon DynamoDB (v1)

Information note **Amazon DynamoDB extraction is supported by Stitch**   
This integration is powered by [Singer's](https://www.singer.io/) Amazon DynamoDB tap and certified by Stitch. [Check out and contribute to the repo on GitHub](https://github.com/singer-io/tap-dynamodb).  
  
For support, contact [Support](https://community.qlik.com/t5/Support/ct-p/qlikSupport).

## Amazon DynamoDB integration summary

Stitch’s Amazon DynamoDB integration replicates data using the [Boto 3 1.9.57](https://boto3.amazonaws.com/v1/documentation/api/1.9.57/index.html) driver.

## Amazon DynamoDB feature snapshot

A high-level look at Stitch's Amazon DynamoDB (v1) integration, including release status, useful links, and the features supported in Stitch.

| **STITCH** |  |  |  |
| --- | --- | --- | --- |
| **[Release status](https://help.qlik.com/en-US/stitch/integrations/identify-an-integration-version)** | Released on January 6, 2020 | **Supported by**<br><br>the Info icon<br><br>Indicates if the integration is supported by Stitch or the Singer community. | [Stitch](https://community.qlik.com/t5/Support/ct-p/qlikSupport) |
| **Stitch plan**<br><br>the Info icon<br><br>The type of Stitch plan required to use the integration. | Standard | **Supported versions**<br><br>the Info icon<br><br>Indicates if the integration is supported by Stitch or the Singer community. | n/a |
| **[API availability](https://help.qlik.com/en-US/stitch/developers/stitch-connect/guides/source-destination-reference)**<br><br>the Info icon<br><br>Indicates if the integration is available as a source in Stitch Connect. | the Success icon<br><br>This version of the integration is available via the API.<br><br><br>Available | **Singer GitHub repository** | [singer-io/tap-dynamodb](https://github.com/singer-io/tap-dynamodb) |
| **CONNECTION METHODS** |  |  |  |
| **SSH connections**<br><br>the Info icon<br><br>Indicates if SSH tunnels are supported for connecting this database to Stitch. | the Remove icon<br><br>This feature is not supported for this version of the integration.<br><br><br>Unsupported | **SSL connections**<br><br>the Info icon<br><br>Indicates if SSL connections are supported for connecting this database to Stitch. | the Remove icon<br><br>This feature is not supported for this version of the integration.<br><br><br>Unsupported |
| **REPLICATION SETTINGS** |  |  |  |
| **[Anchor Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/anchor-scheduling)**<br><br>the Info icon<br><br>A type of replication scheduling where an integration's Replication Frequency is anchored to a time you select. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported | **[Advanced Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/advanced-scheduling)**<br><br>the Info icon<br><br>A type of replication scheduling that allows you to specify granular start times for data extraction using cron expressions. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported |
| **Table-level reset**<br><br>the Info icon<br><br>Indicates if data can be reset at the table level. If unsupported, data can only be reset at the integration level. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported | **[Configurable Replication Methods](https://help.qlik.com/en-US/stitch/replication/replication-methods)**<br><br>the Info icon<br><br>Indicates if Replication Methods can be defined by you for tables in the integration. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported |
| **REPLICATION METHODS** |  |  |  |
| **[Log-based Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/log-based-incremental)**<br><br>the Info icon<br><br>Log-based Incremental Replication is a replication method in which Stitch identifies modifications to records - including inserts, updates, and deletes - using a database's binary log files. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported | **[Key-based Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/key-based-incremental)**<br><br>the Info icon<br><br>Key-based Incremental Replication is a replication method in which Stitch identifies new and updated data using a column called a Replication Key. | the Remove icon<br><br>This feature is not supported for this version of the integration.<br><br><br>Unsupported |
| **[Full Table Replication](https://help.qlik.com/en-US/stitch/replication/replication-methods/full-table)**<br><br>the Info icon<br><br>Full Table Replication is a replication method in which all rows in a table - including new, updated, and existing - are replicated during every replication job. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported |  |  |
| **DATA SELECTION** |  |  |  |
| **Table selection**<br><br>the Info icon<br><br>The ability to select individual tables for replication in the Stitch interface. If table selection is unavailable, all tables and all the columns they contain will be set to replicate by default. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported | **Column selection**<br><br>the Info icon<br><br>The ability to select individual columns for replication in the Stitch interface. If column selection is unavailable, all columns will be set to replicate by default. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported |
| **[View replication](https://help.qlik.com/en-US/stitch/replication/replicating-database-views)**<br><br>the Info icon<br><br>Indicates if Stitch supports replicating views from this database integration. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported | **Select all**<br><br>the Info icon<br><br>Indicates if the integration supports selecting all available tables and columns. | the Remove icon<br><br>Amazon DynamoDB integrations don't currently support a default Replication Method, which is required to use the Select All feature.<br><br><br>Unsupported |
| **TRANSPARENCY** |  |  |  |
| **[Extraction Logs](https://help.qlik.com/en-US/stitch/replication/extractions/integration-extraction-logs)**<br><br>the Info icon<br><br>Logs detailing the Extraction phase of the replication process for a given integration. Includes error info, should an error occur. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported | **[Loading Reports](https://help.qlik.com/en-US/stitch/replication/integration-loading-reports)**<br><br>the Info icon<br><br>Reports detailing the Loading phase of the replication process for a given integration. Includes the number of rows loaded for each table and error info, should an error occur. | the Success icon<br><br>This feature is supported for this version of the integration.<br><br><br>Supported |

## Connecting Amazon DynamoDB

### Amazon DynamoDB setup requirements

To set up Amazon DynamoDB in Stitch, you need:

* **An Amazon Web Services (AWS) account.** Signing up is free - [click here](https://aws.amazon.com/) or go to `https://aws.amazon.com` to create an account if you don’t have one already.

* **Permissions in AWS Identity Access Management (IAM) that allow you to create policies, create roles, and attach policies to roles**. This is required to grant Stitch authorization to Amazon DynamoDB.

* **If using Log-based Incremental replication, streams must be enabled in Amazon DynamoDB for every table you want to replicate using this method.** Additionally, each stream must use the `New Image` or `New and Old Images` option in AWS. Refer to the [Replication](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#log-based-incremental-replication-details) section for more info.

### Step 1: Retrieve your Amazon Web Services account ID

1. Sign into your Amazon Web Services (AWS) account.

2. Click the **user menu**, located between the **bell** and **Global** menus in the top-right corner of the page.

3. Click **My Account**.

4. In the **Account Settings** section of the page, locate the **Account Id** field:

   ![An AWS account ID, highlighted in the AWS Account Settings page](https://help.qlik.com/en-US/stitch/images/integrations/s3-csv-account-id.png)An AWS account ID, highlighted in the AWS Account Settings pageAn AWS account ID, highlighted in the AWS Account Settings page

Keep this handy - you’ll need it to complete the setup.

### Step 2: Configure Log-based Incremental Replication

Information note **Note**: Skip this step if you’re not planning to use Log-based Incremental Replication. [Click to skip ahead](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#connect-stitch).

While Log-based Incremental Replication is the most accurate and efficient method of replication, using this replication method may, at times, require manual intervention or impact the source database’s performance. Refer to the [Log-based Incremental Replication documentation](https://help.qlik.com/en-US/stitch/replication/replication-methods/log-based-incremental) for more info.

You can also use one of Stitch’s other [Replication Methods](https://help.qlik.com/en-US/stitch/replication/replication-methods), which don’t require any database configuration. Replication Methods can be changed at any time.

For every table you want to replicate using Log-based Incremental replication, you’ll need to enable Amazon DynamoDB streams. Each stream must use the `New Image` or `New and Old Images` option in AWS or replication will be unsuccessful.

Refer to [Amazon’s documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html#Streams.Enabling) for instructions on enabling and configuring streams.

### Step 3: Add Amazon DynamoDB as a Stitch data source

1. If you aren’t signed into your Stitch account, [sign in now](https://app.stitchdata.com/).

2. On the Stitch Dashboard page, click the **Add Integration** button.

3. Locate and click the **Amazon DynamoDB** icon.

4. Fill in the fields as follows:

   * **Integration Name**: Enter a name for the integration. This is the name that will display on the Stitch Dashboard for the integration; it’ll also be used to create the schema in your destination.

     For example, the name “Stitch Amazon DynamoDB” would create a schema called `stitch_amazon_dynamodb` in the destination. **Note:** The schema name cannot be changed after the integration is saved.

   * **AWS Account ID**: Paste the AWS account ID you retrieved in [Step 1](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#retrieve-aws-account-id).

   * **AWS Region**: Select the region that your instance resides in. For example: `US East (N. Virginia)`

### Step 4: Create a replication schedule

Information note Replication schedules affect the time Extraction begins, not the time to data loaded. Refer to the [Replication Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling) documentation for more information.

In the **Replication Frequency** section, you’ll create the integration’s [replication schedule](https://help.qlik.com/en-US/stitch/replication/replication-scheduling). An integration’s replication schedule determines how often Stitch runs a replication job, and the time that job begins.

**Note**: If using Log-based Incremental Replication, keep in mind that Amazon purges Amazon DynamoDB streams after 24 hours. To ensure you don’t lose data, set the integration’s Replication Frequency to an interval less than 24 hours. For example: 12 hours.

If Stitch identifies a stream that has aged out, Stitch will automatically reset the table and queue a full re-replication.

Amazon DynamoDB integrations support the following replication scheduling methods:

* [Replication Frequency](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/replication-frequency)

* [Anchor Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/anchor-scheduling)

* [Advanced Scheduling using Cron](https://help.qlik.com/en-US/stitch/replication/replication-scheduling/advanced-scheduling) (Advanced or Premium plans only)

To keep your row usage low, consider setting the integration to replicate less frequently. See the [Understanding and Reducing Your Row Usage guide](https://help.qlik.com/en-US/stitch/getting-started/understanding-stitch-usage) for tips on reducing your usage.

### Step 5: Grant access to Amazon DynamoDB using AWS IAM

Information note **Note**: To complete this step, you must have permissions in AWS Identity Access Management (IAM) that allow you to create/modify IAM policies and roles.

Next, Stitch will display a **Configure Your DynamoDB Integration** page. This page contains the info you need to configure bucket access for Stitch, which is accomplished via an IAM policy and role.

**Note**: Saving the integration before you’ve completed the steps below will result in connection errors.

* [Step 5.1: Create an IAM policy](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#create-iam-policy)

* [Step 5.2: Create an IAM role for Stitch](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#create-stitch-iam-role)

* [Step 5.3: Check and save the connection in Stitch](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#check-save-stitch-connection)

#### Step 5.1: Create an IAM policy

[An IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#access_policies-json) is JSON-based access policy language to manage permissions to Amazon DynamoDB resources.

For more info about the permissions the auto-generated policy Stitch IAM policy grants, click the link below.

Amazon DynamoDB permissions

| **Permission name** | **Description** |
| --- | --- |
| **dynamodb:ListTables** | Required to list table names associated with the current account. |
| **dynamodb:DescribeStream** | Required to return information about a stream, including the current status of the stream, its Amazon Resource Name (ARN), the composition of its shards, and its corresponding Amazon DynamoDB table. |
| **dynamodb:ListStreams** | Required to obtain the stream ARNs for the tables associated with the current account. |
| **dynamodb:DescribeTable** | Required to obtain information about the current account's tables. |
| **dynamodb:GetRecords** | Required to return records from a shard. |
| **dynamodb:Scan** | Required to perform a scan on tables using Full Table Replication. |
| **dynamodb:GetShardIterator** | Required to perform Log-based Incremental Replication. |

To create the IAM policy:

1. In AWS, [navigate to the IAM service](https://console.aws.amazon.com/iam/home#/home) by clicking the **Services** menu and typing **IAM**.
2. Click **IAM** once it displays in the results.
3. On the IAM home page, click **Policies** in the menu on the left side of the page.
4. Click **Create Policy**.
5. In the **Create Policy** page, click the **JSON** tab.
6. Select everything currently in the text field and delete it.
7. In the text field, paste the IAM policy from the **Configure Your DynamoDB Integration** page in Stitch.
8. Click **Review policy**.
9. On the **Review Policy** page, give the policy a name. For example: `stitch_amazon_dynamodb`
10. Click **Create policy**.

#### Step 5.2: Create an IAM role for Stitch

Information note

**Required permissions**

To complete this step, you need the following AWS IAM permissions: `CreateRole` and `AttachRolePolicy`. Refer to [Amazon’s documentation](https://docs.aws.amazon.com/IAM/latest/APIReference/API_Operations.html) for more info.

the Info icon

**Roles can’t be used for multiple integrations**

If you’re creating multiple Amazon DynamoDB integrations, you’ll need to complete this step for each integration you’re connecting.

The **Role Name** Stitch uses to connect to the Amazon resource is unique to the integration. Attempting to re-use a role for multiple integrations will cause connection errors.

In this step, you’ll create an IAM role for Stitch and apply the IAM policy from the previous step. This will ensure that Stitch is visible in any logs and audits.

To create the role, you’ll need the **Account ID**, **External ID**, and **Role Name** values provided on the Stitch **Configure Your DynamoDB Integration** page.

1. In AWS, navigate to the [IAM Roles](https://console.aws.amazon.com/iam/home#/roles) page.
2. Click **Create Role**.
3. On the **Create Role** page:
   1. In the **Select type of trusted entity** section, click the **Another AWS account** option.
   2. In the **Account ID** field, paste the Account ID from Stitch. **Note**: This isn’t your AWS account ID from Step 1 - this is the Account ID that displays in Stitch on the **Configure Your DynamoDB Integration** page.
   3. In the **Options** section, check the **Require external ID** box.
   4. In the **External ID** field that displays, paste the External ID from the Stitch **Configure Your DynamoDB Integration** page: ![Account ID and External ID fields mapped from Stitch to AWS](https://help.qlik.com/en-US/stitch/images/integrations/aws-account-external-ids.png)Account ID and External ID fields mapped from Stitch to AWSAccount ID and External ID fields mapped from Stitch to AWS
   5. Click **Next: Permissions**.
4. On the **Attach permissions** page:
   1. Search for the policy you created in the [previous step](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#create-iam-policy).
   2. Once located, check the box next to it in the table.
   3. Click **Next: Tags**.
5. If you want to enter any tags, do so on the **Add tags** page. Otherwise, click **Next: Review**.
6. On the **Review** page:
   1. In the **Role name** field, paste the Role Name from the Stitch **Configure Your DynamoDB Integration** page: ![Role name field mapped from Stitch to AWS](https://help.qlik.com/en-US/stitch/images/integrations/aws-role-name.png)Role name field mapped from Stitch to AWSRole name field mapped from Stitch to AWS

      **Remember**: Role names are unique to the Stitch Amazon DynamoDB integration they’re created for. Attempting to use the same role for multiple integrations will cause connection errors.

   2. Enter a description in the **Role description** field. For example: `Stitch role for Amazon DynamoDB integration.`

   3. Click **Create role**.

#### Step 5.3: Check and save the connection in Stitch

Information note **Note**: Saving the integration before you’ve completed the IAM policy and role steps will result in connection errors.

After you’ve created the IAM policy and role, you can save the integration in Stitch. When finished, click **Check and Save**.

### Step 6: Select data to replicate

Information note **Is an object missing or not replicating?** Verify that the object meets the requirements for [selection](https://help.qlik.com/en-US/stitch/replication/extractions/data-selection/setting-tables-and-columns-to-replicate#selection-requirements) and [replication](https://help.qlik.com/en-US/stitch/replication/extractions/data-selection/setting-tables-and-columns-to-replicate#replication-requirements).

The last step is to select the tables and columns you want to replicate. [](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#schema)

**Note**: If a replication job is currently in progress, new selections won’t be used until the next job starts.

* Selection options
* Individual tables and columns
* Database views

For Amazon DynamoDB integrations, you can select:

1. **Individual tables and columns**

2. **Database views**

Click the tabs to view instructions for each selection method.

1. In the Integration Details page, click the **Tables to Replicate** tab.

2. Locate a table you want to replicate.

3. Click the **checkbox** next to the table’s name. A blue checkmark means the table is set to replicate.

4. On the page that displays, click the **Table Settings** button.

5. In the **Table Settings** page:

   1. Define the table’s **Replication Method**.

   2. **Optional**: Select or exclude fields by entering a projection expression in the **Fields to Replicate** section. Refer to the [Selecting DynamoDB Fields Using Projection Expression guide](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb/field-selection-using-projection-expressions) for instructions and examples.

   3. When finished, click **Update Settings**.

6. Repeat this process for every table you want to replicate.

7. Click the **Finalize Your Selections** button at the bottom of the page to save your data selections.

Setting a database view to replicate is similar to selecting a table, with a few differences. Refer to the [Replicating Database Views guide](https://help.qlik.com/en-US/stitch/replication/replicating-database-views) for detailed instructions.

At a high level, you’ll need to complete the following to select a database view:

1. [Verify the database user’s permissions](https://help.qlik.com/en-US/stitch/replication/replicating-database-views#step--verify-database-user-permissions)
2. [Select the view](https://help.qlik.com/en-US/stitch/replication/replicating-database-views#step--select-view-in-stitch)
3. **Optional**: [Define the view’s Primary Key](https://help.qlik.com/en-US/stitch/replication/replicating-database-views#step--define-view-primary-key)
4. [Define the view’s Replication Method](https://help.qlik.com/en-US/stitch/replication/replicating-database-views#step--define-view-replication-method)
5. [Save the view’s settings](https://help.qlik.com/en-US/stitch/replication/replicating-database-views#step--save-view-settings)

### Initial and historical replication jobs

After you finish setting up Amazon DynamoDB, its **Sync Status** may show as **Pending** on either the Stitch Dashboard or in the Integration Details page.

For a new integration, a **Pending** status indicates that Stitch is in the process of scheduling the initial replication job for the integration. **This may take some time to complete.**

Information note

**Initial replication jobs with Anchor Scheduling**

If using Anchor Scheduling, an initial replication job may not kick off immediately. This depends on the selected Replication Frequency and Anchor Time. Refer to the [Anchor Scheduling](https://help.qlik.com/en-US/stitch/replication/replication-scheduling#job-scheduling-and-anchor-times) documentation for more information.

#### Free historical data loads

The first seven days of replication, beginning when data is first replicated, are free. Rows replicated from the new integration during this time won’t count towards your quota. Stitch offers this as a way of testing new integrations, measuring usage, and ensuring historical data volumes don’t quickly consume your quota.

Information note **Replication will continue after the seven days are over**. If you’re no longer interested in this source, be sure to **pause** or **delete** the integration to prevent unwanted usage.

## Amazon DynamoDB replication

* [Details about Log-based Incremental Replication via Amazon DynamoDB streams](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#log-based-incremental-replication-details)

* [Details about Full Table Replication using scans and eventually consistent reads](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#full-table-replication-details)

* [Details about expected delays in Amazon DynamoDB replication](https://help.qlik.com/en-US/stitch/integrations/databases/amazon-dynamodb#replication-delays)

### Log-based Incremental Replication

Stitch’s Amazon DynamoDB integration uses [Amazon DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) to perform Log-based Incremental Replication. To use Log-based Incremental Replication, streams must be enabled on every table in Amazon DynamoDB you want to replicate using this Replication Method.

Refer to [Amazon’s documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html#Streams.Enabling) for instructions on enabling streams for Amazon DynamoDB tables.

**Note**: The **Manage Stream** option must be one of the following, or replication will be unsuccessful:

* `New Image`
* `New and old images`

Information note **Note**: DynamoDB streams are purged after 24 hours. To ensure you don’t lose data, set the integration’s Replication Frequency to an interval less than 24 hours. For example: 12 hours. If Stitch identifies a stream that has aged out, Stitch will automatically reset the table and queue a full re-replication.

### Full Table Replication

To perform Full Table Replications with Stitch’s Amazon DynamoDB integration, Stitch uses scans to return data. A scan returns data by accessing all items within a table. As queries require you to specify the hash key (Primary Key), Stitch uses scans to simplify setup and replication. For more information about scans, click [here](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html).

Additionally, Stitch’s Amazon DynamoDB integration only uses eventually consistent reads from your selected Amazon DynamoDB tables. **Note**: This means that you will not see all of your recent data right away due to a delay from Amazon, but it will eventually catch up and return the latest records. For more information on Amazon DynamoDB read consistency, refer to [Amazon’s documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html).

### Replication delays

Stitch can’t replicate data from your Amazon DynamoDB database until the shard is closed in your account. This can result in a delay in the replicaton of new data, as the new data is available only after the shard has been closed. Forcing an extraction in Stitch won’t have any effect on replicating new data unless the shard is closed.