Shopify
Shopify is an e-commerce platform. You can use Shopify as a data source when landing or replicating data with the new Shopify connector in Qlik Talend Cloud.
Shopify integration replicates data using the Shopify GraphQL Admin API.
See the Schema section for a list of objects available for replication.Preparing for authentication
To access your data, you need to authenticate the connection with your account credentials.
To set up your Shopify account, you need:
- Your Shopify store short name (the part before
.myshopify.com). - A Shopify app in the Shopify Dev Dashboard with a client ID and client secret.
- The app and target store in the same Shopify organization. The client credentials grant is only available for apps in your organization's Dev Dashboard installed in stores that you own. Apps distributed through the Shopify App Store or created from the store admin cannot be used with this connector.
- The app installed on the target store (installation is per store).
To create the app and retrieve your credentials:
- Log into your Shopify Dev Dashboard.
- Create a custom app for your organization.
- Configure the app's access scopes to grant read access to the resources you intend to replicate.
- Install the app on the store that you want to replicate.
- Navigate to Settings, and copy-paste your client ID and client secret into a secured file.
Access scopes to grant
Grant read access for each resource you intend to replicate. Metafields are covered by the scope of the resource they are attached to.
All tables are offered for selection regardless of which scopes have been granted, so the list you see when setting up a task does not tell you what the app can actually read. Scopes are enforced when the task runs. Grant the scopes for the tables you intend to select before running a task.
| Grant this scope | To replicate these tables |
|---|---|
read_orders | orders, order_refunds, order_shipping_lines, transactions, abandoned_checkouts, metafields_orders |
read_products | products, product_variants, collections, metafields_products, metafields_collections |
read_customers | customers, metafields_customers |
read_inventory | inventory_items, inventory_levels |
read_locations | locations, inventory_levels |
read_assigned_fulfillment_orders, read_merchant_managed_fulfillment_orders, read_third_party_fulfillment_orders | fulfillment_orders |
The events table returns activity log entries for store resources rather than one resource type of its own. Grant the scopes for the resources whose events you want to see.
Two scopes deserve particular attention:
read_all_orders: by default,read_ordersonly exposes orders created in the last 60 days. To replicate older order history, requestread_all_ordersfrom Shopify. This is a restricted scope that Shopify grants on request, not one you can simply select. Without it, a start date earlier than 60 days ago returns no data before the 60-day boundary.read_users: theauthorfield on theeventstable requires this scope, which Shopify makes available on Shopify Plus plans only. Where it is not granted,eventsreplicates without that one field and the rest of the table is unaffected.
Creating the connection
For more information, see Connecting to SaaS applications.
- Fill in the required connection properties.
Provide a name for the connection in Connection name.
Select Open connection metadata to define metadata for the connection when it has been created.
Click Create.
| Setting | Description |
|---|---|
| Data gateway | Select a Data Movement gateway if required by your use case. Information note This field is not available with the Qlik Talend Cloud Starter subscription, as it does not support Data Movement gateway. If you have another subscription tier and do not want to use Data Movement gateway, select None. For information on the benefits of Data Movement gateway and use cases that require it, see Qlik Data Gateway - Data Movement. |
| Start Date | Enter the date, in the format |
| Shopify Shop | Shopify store short name (the part before .myshopify.com). |
| Client ID | Client ID. |
| Client Secret | Client secret. |
Limitations and considerations
- Deletes are not captured. The connector replicates inserts and updates only. A record deleted in Shopify remains in the destination. Use a periodic full reload if you need deletes reflected.
- Order history older than 60 days needs an extra scope. See
read_all_ordersin the access scopes section above. - Only one replication task per store and app can run at a time. The
orderstable uses Shopify's bulk operations API, which allows only one bulk query per store per app at a time. If a second task starts against the same store while the first is still running, the connector waits and backs off for up to roughly ten minutes before failing. Schedule tasks so they do not overlap, or use a separate Shopify app per task. - Large order backfills can take a long time. A bulk operation is submitted per date window and Shopify processes it asynchronously. Reducing Date Window Size produces more, smaller operations, which usually finishes sooner on a large store.
- Start Date applies to the initial sync only. Once a table has a stored bookmark, later syncs resume from it. Changing Start Date does not retroactively re-replicate data.
- Incremental runs can re-deliver boundary records. Each incremental run filters from the previous run's high-water mark inclusive, so records whose timestamp exactly equals it are replicated again. Records are keyed on
id, so this resolves to an upsert in the destination rather than a duplicate. - Rate limits are handled automatically. Shopify applies per-store API limits and returns
HTTP 429 Too Many Requestswhen they are exceeded. The connector waits and retries. Server errors and timeouts are retried separately, up to 5 attempts, with exponential backoff. - Access tokens are refreshed automatically. Shopify access tokens expire after 24 hours. The connector requests a new one as needed. The client ID and client secret do not expire, but if the secret is rotated in the Dev Dashboard the new value must be saved to the connection.
- Selecting a table the app cannot read fails the task. Because every table is offered regardless of scope, it is possible to select one the app has not been granted access to. The task then fails when it reaches that table. The
fulfillment_orderstable is handled differently: the connector skips it, replicates the remaining selected tables, and reports that the connection needs to be re-authorized with the fulfillment order scopes. - Adding a scope later requires reinstalling the app. If you grant a new scope after installation, apply it to the app in the Dev Dashboard and reinstall the app on the store before rerunning the task.
Schema
Schemas and naming conventions can change from version to version. It is recommended to verify your version before continuing.
Depending on your destination, table and column names may not appear as they are outlined below.
Every table carries three additional columns identifying the store the record came from: _sdc_shop_id, _sdc_shop_name, and _sdc_shop_myshopify_domain.
Two tables use createdAt as the replication key rather than updatedAt: locations (because the Shopify locations query does not support filtering on updatedAt, so a modified location is not picked up by an incremental run) and transactions (because transactions are read through their parent order, not as a top-level resource).
abandoned_checkouts
The abandoned_checkouts table contains info about abandoned checkouts. Shopify considers a checkout to be abandoned when a customer has entered billing and shipping details, but hasn’t completed the purchase.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
collections
The collections table contains info about collections.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
customers
The customers table contains info about the shop’s customers. This includes their contact details, order history, and email marketing preferences.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
events
The events table contains info about events in the shop.
Replication Method | CDC |
Primary Key | id |
Replication Key | createdAt |
| Useful links |
fulfillment_orders
The fulfillment_orders table contains datails about fulfillment orders.
Replication Method | CDC |
Primary Key | id |
| Useful links |
inventory_items
The inventory_items table contains info about items in a shop.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
inventory_levels
The inventory_levels table contains info about quantities of an inventory item for a location.
Replication Method | CDC |
Primary Keys | id |
Replication Key | updatedAt |
| Useful links |
locations
The locations table contains info about locations.
Replication Method | CDC |
Primary Key | id |
Replication Key | createdAt |
| Useful links |
metafields_collections
The metafields_collections table contains info about metafields for collections.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
metafields_customers
The metafields_customers table contains info about metafields for customers.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
metafields_orders
The metafields_orders table contains info about metafields for orders.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
metafields_products
The metafields_products table contains info about metafields for products.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
order_refunds
The order_refunds table contains info about refunds associated with orders.
Replication Method | CDC |
Primary Key | id |
Replication Key | createdAt |
| Useful links |
order_shipping_lines
The order_shipping_lines table contains the shipping details that the customer chose for their order.
Replication Method | CDC |
Primary Key | id |
| Useful links |
orders
The orders table contains info about a shop’s completed orders.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
product_variants
The product_variants table contains info about variants that can be added to a product to represent one version of a product with several options.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
products
The products table contains info about a shop’s products.
Replication Method | CDC |
Primary Key | id |
Replication Key | updatedAt |
| Useful links |
transactions
The transactions table contains info about transactions.
Replication Method | CDC |
Primary Key | id |
Replication Key | createdAt |
| Useful links |