Spark Universal support for Hadoop distributions in Talend Studio
Last updated: 10/1/2026Spark Universal is a mechanism that allows Talend Studio to be compatible with every big data distribution for a given Spark version. You choose a Spark version and upload a Hadoop configuration JAR file that contains all the necessary information to connect to your cluster.
When you use Spark Universal in Talend Studio, only Scala 2.12 is supported.
For Spark Batch and Spark Structured Streaming Jobs, Universal is the default option in the Distribution list.
All Built-In Distribution values are deprecated in Talend Studio from 8.0.1 R2026-09 onwards, with removal planned for R2027-09. Use Universal instead.
The following Built-In distributions are deprecated: Amazon EMR, Azure Synapse, Cloudera, Databricks, Hortonworks, Microsoft HD Insight 4.0, and Custom.
The corresponding vendor platforms remain supported, but the Universal distribution must be used instead of the Built-In one.
Spark Streaming is deprecated in Talend Studio from 8.0.1 R2026-09 onwards, with removal planned for R2027-09. It is replaced by Spark Structured Streaming.
Support for HPE Ezmeral Runtime Enterprise through the Kubernetes mode is deprecated in Talend Studio from 8.0.1 R2026-09 onwards, with removal planned for R2027-09.
Spark Universal modes and environments support
For Spark Batch Jobs, the Runtime mode/environment list is ordered as shown in the following table, which also indicates the Spark versions supported by each mode or environment:
| Mode or environment | Spark 2.4.x | Spark 3.x | Spark 4.x |
|---|---|---|---|
| Amazon EMR | Not supported | Supported | Not supported |
| Amazon EMR Serverless | Not supported | Supported | Not supported |
| Azure HDInsight | Not supported | Supported: 3.1.x, 3.3.x | Not supported |
| Azure Synapse | Not supported | Supported: 3.2.x, 3.3.x | Not supported |
| Cloudera Data Engineering | Not supported | Supported: 3.1.x, 3.2.x | Not supported |
| Cloudera on Cloud | Not supported | Supported | Not supported |
| Cloudera on Premises | Not supported | Supported | Not supported |
| Databricks | Not supported | Supported | Supported |
| Databricks Serverless | Not supported | Not supported | Supported |
| Google Managed Service for Apache Spark (formerly Dataproc) | Not supported | Supported: 3.1.x, 3.3.x | Not supported |
| Kubernetes | Not supported | Supported | Not supported |
| Local | Not supported | Supported | Supported |
| Livy Knox | Not supported | Supported | Not supported |
| Spark-submit scripts | Not supported | Supported: 3.3.x | Not supported |
| Standalone | Not supported | Supported: 3.2.x, 3.4.x | Not supported |
| Yarn cluster | Supported | Supported except 3.4.x | Not supported |
| Mode or environment | Spark 3.x | Spark 4.x |
|---|---|---|
| Databricks | Supported | Supported |
| Local | Supported | Supported |
| Yarn cluster | Supported except 3.4.x | Not supported |
Spark Universal distributions support
Talend Studio supports the following distributions in Yarn cluster mode, depending on the Spark versions:
| Spark version | Supported distributions in Yarn cluster mode |
|---|---|
| Spark 2.4.x |
|
| Spark 3.0.x |
|
| Spark 3.1.x |
|
| Spark 3.2.x |
|
| Spark 3.3.x |
|
| Spark 3.5.x |
|
| Spark 3.x |
|
For example, if you want to connect to an Amazon EMR 6.2 cluster, you need to select Spark 3.0.x version and then upload the Hadoop configuration JAR file that contains all the *-site.xml files related to the cluster.
This list of distribution is not exhaustive, you can use Yarn cluster with any other distribution if the Spark version matches, but keep in mind that they have not been officially tested by Talend and thus not guaranteed to work.
Talend Studio supports the following distributions in Livy Knox mode:
| Spark version | Supported distributions in Livy Knox mode |
|---|---|
| Spark 3.5.x |
|
| Spark 3.x |
|