# Big Data: new features

## Spark Job designer enhancements

| Feature | Description | Available in |
| --- | --- | --- |
| ADLS Gen2 | Azure Data Lake Storage Generation2 is now supported<br>with the following Big Data platforms:<br><br><ul> <li>Databricks V5.5 LTS</li> <li>Cloudera CDH V6.1</li> <li>Hortonworks Data Platform V3.1</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Snowflake | The Snowflake components for Spark Batch are now generally available. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Native Datasets | In Spark Batch Jobs, support for native Spark Datasets has been added to more components to<br>obtain inherent performance gains. To benefit from this enhancement,<br>users must be using Spark V2.0 onwards with the following components:<br><br><ul> <li>tFileInputParquet and tFileOutputParquet</li> <li>tFileInputDelimited and tFileOutputDelimited</li> <li>tFileInputFullRow</li> <li>tFileInputPositional and tFileInputRegex</li> <li>tSortRow, tExtractDelimitedFields, tExtractPositionalFields,                                    tExtractRegexFields, tExtractXMLField, tExtractJSONFields,                                    tNormalize, tReplace, tReplicate, tSample, tUnite and                                    tSchemaComplianceCheck.</li> </ul><br><br>The following components require Spark V2.1 onwards to support Spark<br>Datasets.<br><br><ul> <li>tAggregateRow</li> <li>Left Outer Join in tMap, in addition to the tMap features that                                    have had support for Datasets since <span>Talend</span> Studio                                    V7.2.</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Delta Lake | The tDeltaLakeInput and tDeltaLakeOutput components are now generally available.<br>![](https://help.qlik.com/talend/en-US/release-notes/Resources/images/delta_lake_job.png) | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Apache Spark V2.4 | This new Aparch Spark version is supported with more<br>Big Data platforms in Spark Batch and Spark Streaming Jobs. The platforms<br>which now support Spark V2.4 are:<br><br><ul> <li>Cloudera CDH6.1.1</li> <li>Databricks V5.5</li> <li>Google Cloud Dataproc V1.4</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Job status | With Databricks, users are enabled to configure how<br>often the Studio asks a Spark cluster for Job status. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| tS3Configuration | With Amazon EMR, users can now apply an S3 bucket<br>policy. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| tAggregateRow | In Spark Batch Jobs, the Count (distinct) function and<br>the Sample Standard Deviation Algorithm function have been added. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| New driver versions | The support for the following driver versions has been<br>added to their related components:<br><br><ul> <li>Redshift JDBC driver V1.23.7.106</li> <li>MySQL driver V8.0.18</li> <li>Teradata JDBC driver V16.20.00.13</li> <li>MariaDB JDBC driver V2.5.3 in JDBC                                    components</li> <li>Snowflake JDBC driver V3.11.x</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| New components available | Two new components are now available: tAzureAdlsGen2Input and tAzureAdlsGen2Output. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |

## Support for Big Data platforms

| Feature | Description | Available in |
| --- | --- | --- |
| Databricks | <ul> <li>Databricks V5.5 LTS is now supported by Spark                                Jobs.</li> <li>Support for transient clusters of Azure Databricks                                has been added.</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Hortonworks Data Platform | <ul> <li>Hortonworks Data Platform V3.1 is supported.</li> <li>The Hortonworks Data Platform V3.x series is now                        generally available among the Dynamic Distributions.</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Google Cloud Dataproc | <ul> <li>Google Cloud Dataproc V1.4 is supported</li> <li>In Standard Jobs, tGoogleDataprocManage                                    supports all regions.</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Custom Hadoop configuration | When defining connections to Cloudera or Hortonworks in<br>Repository, users can now specify a custom JAR<br>file to provide the connection parameters of the Hadoop environment to be<br>used. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |

## Other components

| Feature | Description | Available in |
| --- | --- | --- |
| Kafka | Kafka V2.2.1 is now officially supported with:<br><br><ul> <li>Cloudera CDH V6.1</li> <li>Hortonworks Data Platform V3.1</li> <li>Kafka components in Standard Jobs</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Google BigQuery | <ul> <li>In tBigQueryBulkExec, users can now drop tables                                with either a service account or their OAuth 2.0 credentials.</li> <li>The BigQuery components now support Google cloud                                client API 1.25.10.</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| Couchbase | <ul> <li>tCouchbaseOutput now allows users to perform N1QL                                queries with parameters.</li> <li>Non-JSON documents are supported.</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| CXF | CXF V3.3.4 is now supported in the following<br>components:<br><br><ul> <li>tDBFSConnection, tDBFSGet, tDBFSPut</li> <li>tHCatalogInput, tHCatalogLoad, tHCatalogOperation,                                tHCatalogOutput</li> </ul> | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |
| MongoDB | The support for MongoDB V4.2.x has been added to the<br>MongoDB components in Standard Jobs. | <ul>Available in:<li>Big Data</li><li>Big Data Platform</li><li>Cloud Big Data</li><li>Cloud Big Data Platform</li><li>Cloud Data Fabric</li><li>Data Fabric</li><li>Real-Time Big Data Platform</li></ul><br><br>All<br>Talend<br>products with Big Data |