Supported Cloud Big Data platform distribution versions for Talend Jobs
Cloud Hadoop distributions
Talend supports the following cloud platforms for Big Data. Click your cloud platform to see the Big data support information.
- Amazon EMR
- Databricks on AWS
- Databricks on Azure
- Databricks on Google Cloud Platform (GCP)
- Google Dataproc
- Microsoft HDInsight
Old versions of the supported Big Data platforms are being retired by their vendors. Talend ceases to support a version once this version reaches its date of end of support set by its vendor.
Talend and its community provide you with the convenience to keep using a version its vendor ceases to support in Talend products. For this reason, this version can be still listed in the following tables and available in the products but Talend stops providing support for this version.
Talend supports the minor versions of the platform versions listed in the following tables.
Amazon EMR version | Supported frameworks | Supported Hadoop elements in Spark batch | Supported Hadoop elements in Spark streaming | Supported Hadoop elements in Standard |
---|---|---|---|---|
v5.29.0 (Hadoop 2.8.5) | Standard Spark 2.4 |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive Sqoop |
v6.2.0 (Hadoop 3.2.1) | Standard Spark 3.0 |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive Sqoop |
v6.3.x (Hadoop 3.2.3) | Spark Universal 3.1.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.4.x (Hadoop 3.2.3) | Spark Universal 3.1.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.5.x (Hadoop 3.2.3) | Spark Universal 3.1.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.6.0 (Hadoop 3.2.3) | Spark Universal 3.2.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.7.0 (Hadoop 3.2.3) | Spark Universal 3.2.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.8.0 (Hadoop 3.3.1) | Spark Universal 3.3.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.9.0 (Hadoop 3.3.1) | Spark Universal 3.3.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
v6.10.0 (Hadoop 3.3.1) | Spark Universal 3.3.x | HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
None |
The supported Amazon EMR versions for the tAmazonEMRManage component is 5.29.0.
Databricks on AWS version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
---|---|---|---|---|
5.5 LTS | Standard Spark 2.4 |
Hive S3 DynamoDB Snowflake MongoDB TDM components tDataprepRun |
Hive S3 DynamoDB Kinesis Snowflake MongoDB TDM components as beta tDataprepRun |
DBFS |
6.4 | Standard Spark 2.4.5 |
Azure Blob Storage ADLS Gen2 Snowflake DeltaLake |
Azure Blob Storage ADLS Gen2 |
DBFS |
7.3 LTS | Standard Spark 3.0.1 |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
DBFS |
8.x | Spark Universal 3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
9.x | Spark Universal 3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
10.x | Spark Universal 3.2.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
11.x | Spark Universal 3.3.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
12.x | Spark Universal 3.3.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
13.x | Spark Universal 3.4.x |
Avro Delimited DQ components Hive JSON Parquet Positional S3 XML |
Avro Delimited DQ components Hive JSON Parquet Positional S3 XML |
None |
Databricks on Azure version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
---|---|---|---|---|
5.5 LTS | Standard Spark 2.4 |
Hive Azure Blob Storage ADLS Gen1 ADLS Gen2 Snowflake DeltaLake MongoDB TDM components tDataprepRun |
Hive Azure Blob Storage ADLS Gen1 ADLS Gen2 Snowflake DeltaLake MongoDB TDM components tDataprepRun |
DBFS |
6.4 | Standard Spark 2.4.5 |
Azure Blob Storage ADLS Gen2 Snowflake DeltaLake |
Azure Blob Storage ADLS Gen2 |
DBFS |
7.3 LTS | Standard Spark 3.0.1 |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
DBFS |
8.x | Spark Universal 3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
9.x | Spark Universal 3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
10.x | Spark Universal 3.2.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
11.x | Spark Universal 3.3.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
12.x | Spark Universal 3.3.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
13.x | Spark Universal 3.4.x | ADLS Gen2 Avro Delimited DQ components Hive JSON Parquet Positional XML |
Avro Delimited DQ components Hive JSON Parquet Positional XML |
None |
Databricks on GCP version | Supported frameworks | Supported elements in Spark batch | Supported elements in Standard | |
---|---|---|---|---|
8.x | Spark Universal 3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
9.x | Spark Universal 3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
10.x |
Spark Universal 3.2.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
11.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
None |
12.x |
Spark Universal 3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
None |
13.x | Spark Universal 3.4.x | ADLS Gen2 Avro Delimited DQ components Hive JSON Parquet Positional XML |
Avro Delimited DQ components Hive JSON Parquet Positional XML |
None |
Google Dataproc version | Supported frameworks | Supported elements in Spark batch | Supported elements in Standard | |
---|---|---|---|---|
2.0.x |
Standard
Spark Universal 3.1.x |
Hive BigQuery Google Storage Avro Delimited Parquet Positional XML JSON |
Hive BigQuery Google Storage Avro Delimited Parquet Positional XML JSON Google PubSub |
Hive |
2.1 | Spark Universal 3.3.x |
Hive BigQuery Google Storage Avro Delimited Parquet Positional XML JSON |
Hive BigQuery Google Storage Avro Delimited Parquet Positional XML JSON Google PubSub |
Hive |
Microsoft HDInsight version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
---|---|---|---|---|
4.0 | Spark 2.3 and 2.4 | ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |
5.0 | Spark Universal 3.1.x | ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |
5.1 | Spark Universal 3.3.x | ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |