Supported Cloud Big Data platform distribution versions for Talend Jobs
Cloud Hadoop distributions
Talend supports the following cloud platforms for Big Data. Click your cloud platform to see the Big data support information.
- Google Dataproc
- Databricks on Azure
- Databricks on AWS
- Quoble on AWS
- Cloudera on Azure
- Cloudera on AWS
- Microsoft HD Insight
| Google Dataproc version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| v1.1 | Standard Talend MapReduce Spark v2.0 |
Hive BigQuery Google Storage Avro Delimited Parquet Postional XML JSON |
Hive BigQuery Google Storage Avro Delimited Parquet Postional XML JSON Google PubSub |
Hive |
| Databricks on Azure version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| 3.5 LTS | Standard Spark v2.2 |
Hive Azure Blob Storage ADLS Gen1 |
Hive Azure Blob Storage ADLS Gen1 |
DBFS |
| Databricks on AWS version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| 3.5 LTS | Standard Spark v2.4 |
Hive S3 DynamoDB |
Hive S3 DynamoDB Kinesis |
DBFS |
| Qubole version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| Qubole Spark 2 | Standard Spark v2.2 |
redshift S3 DynamoDB |
redshift S3 DynamoDB Kinesis |
S3 Hive |
| Cloudera version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| CDH5.11 | Spark v2.1 | ADLS Gen1 Azure Blob Storage HDFS |
ADLS Gen1 Azure Blob Storage HDFS |
| Cloudera version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| CDH5.11 | Spark v2.1 | S3 Redshift DynamoDB |
S3 Kinesis Redshift DynamoDB |
| Microsoft HD Insight version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| 3.4 | Talend MapReduce Spark v1.6 |
Hive | Hive | |
| 3.6 | Talend MapReduce Spark v2.1 |
Hive | Hive |