External data sources (Snowflake, Azure SQL Database)
AML Import Data (preview) and Data Connections (preview) were retired on 2026-09-30, so the accelerator does not use them. Microsoft’s replacement is Fabric plus an AML datastore (migration guide).
Design rule: AML only consumes a datastore or data asset. Getting data from Snowflake or Azure SQL into storage is a separate, optional step.
Options tested
“Tested” means validated offline on 2026-10-13 (no Azure resources were created and no Snowflake account was available). Items marked docs only rely on Microsoft documentation.
| # | Option | Result | Evidence |
|---|---|---|---|
| 1 | AML data connection / import (preview) | Ruled out | Retired 2026-09-30. The workspaces/connections ARM type still exists (Foundry uses it), but import jobs do not. |
| 2 | ADLS Gen2 datastore | Works | Bicep (workspaces/datastores, AzureDataLakeGen2) builds with typed API versions 2024-04-01 to 2025-12-01. Terraform azurerm_machine_learning_datastore_datalake_gen2 validates. |
| 3 | OneLake datastore | Works | Bicep datastoreType: 'OneLake' builds. No azurerm resource, so Terraform uses azapi_resource (validates). Needs Fabric admin settings and workspace access for the identity (docs only). |
| 4 | ADF or Synapse copy pipeline to ADLS Parquet | Works (IaC) | azurerm linked services snowflake and azure_sql_database validate. azurerm has no copy-activity resource, so the pipeline body must be activities_json. The pipeline run itself is untested. |
| 5 | Fabric pipeline to OneLake | Docs only | 170+ connectors. Needs a Fabric capacity (cost). Fabric items are not managed by AVM, azurerm or Bicep, so this is a prerequisite documented in the README, not generated infrastructure code. |
| 6 | Fabric mirroring | Docs only | Azure SQL: needs public network access on the logical server (no private-endpoint-only) and same tenant. Snowflake: username and password, or SSO only, no managed identity, preview. |
| 7 | OneLake shortcuts | Ruled out for these sources | Only S3 and ADLS Gen2 are supported, not Snowflake or Azure SQL. |
| 8 | Query in the training job | Works, with constraints | See below. |
In-job query details (Python 3.14, full pinned set)
snowflake-connector-python4.8.0 andmssql-python1.16.0 install alongside the pinned set;pip checkis clean.pyodbcis ruled out: it needs the systemunixodbclibrary and an ODBC driver on the compute image.mssql-pythonbundles its driver.- Use
fetch_arrow_all()for Snowflake, not the[pandas]extra. That extra requirespyarrow<24on Python 3.14, which conflicts with ourpyarrow==25.0.1pin (mlflow 3.17 capspyarrow<26). The connector only warns when pyarrow 25 is present; the Arrow fetch path still works. - A query-to-Parquet round trip with pandas and pyarrow works locally (sqlite stand-in). Connection failures surface as clean exceptions.
- Real connections to Snowflake or Azure SQL are untested.
- Credentials must come from Key Vault or managed identity (Azure SQL: Microsoft Entra auth). Never put them in YAML or the repo.
- The query path needs network egress to the source, so private-network scenarios need an approved outbound rule or private endpoint.
Recommendation
The data_source input supports taxi-csv (default), taxi-parquet, nyc-taxi-open and adls. The accelerator does not generate Snowflake, Azure SQL or OneLake sources.
The two bundled samples are the NYC taxi regression data from the MIT-licensed azureml-examples repository, with the license kept in data/NOTICE.md of the generated project. All bundled samples register as MLTable data assets. nyc-taxi-open is a sample of the public Azure Open Datasets parquet with its original column names; prep.py cleans and maps it, so the rest of the pipeline is unchanged. The adls source registers a single file as a uri_file asset today; an MLTable for it is tracked in the issue list.
- For Snowflake or Azure SQL, copy the data to ADLS Parquet with an ADF copy pipeline (option 4). It is Azure-native, needs no extra capacity and produces the Parquet that training needs. Fabric (options 5 and 6) is the alternative. Then set
data_source: adls. - Use an in-job query (option 8) only for small data.