Moving data from ADLS Gen1 to ADLS Gen2 using Azure Databricks

Mount your Azure Data Lake Storage Gen2 (ADLS Gen2) filesystem to DBFS and use a Talend Job to move your data from ADLS Gen1 to ADLS Gen2.

The tAzureFSConfiguration component enables you to easily and flexibly connect to ADLS Gen1 or ADLS Gen2 in Spark Jobs. However, as only one tAzureFSConfiguration is allowed per Job, you cannot connect to your ADLS Gen1 system and ADLS Gen2 system at the same time in a single Spark Job.

For this reason, when you need to use a Talend Job to move data between those two systems, mount either ADLS Gen1 or ADLS Gen2 to DBFS before designing your Job.

This article demonstrates how to mount ADLS Gen2 and then design a Job to accomplish this move. If you need details about how to mount ADLS Gen1, see Mounting ADLS Gen1 from the Azure databricks documentation.