Your First Azure Data Factory Pipeline: Blob to SQL, Triggers Included

The Ask: "Just Get the CSVs Into a Table"

A vendor drops a CSV of transaction records into Blob Storage every morning around 6 a.m. Someone on the finance team wants those rows sitting in a SQL table by 8, ready for a Power BI report that refreshes right before the daily stand-up. No transformation logic worth mentioning, just move the file and land it in a table shaped correctly. It's the kind of task that sounds like it barely needs a real tool, and it's also the exact task most people build their first Azure Data Factory pipeline to solve.

This walkthrough builds that pipeline, the same Data Factory skillset SkyTrainings' Azure Data Engineering course teaches, using Blob Storage as the source and Azure SQL Database as the destination. It assumes a resource group with a storage account and a SQL database already provisioned. The point here is the pipeline itself, not standing up the surrounding infrastructure.

Two Things Every Pipeline Needs Before It Can Run

Before dragging anything onto the canvas, Data Factory needs to know two things: where the data lives, and what shape it's in. A Linked Service is the connection, the storage account's credentials or the SQL server's connection string. A Dataset points at a specific object inside that connection, one CSV file or one table, and describes its schema. Skip either one and the pipeline has nothing to reference when you try to build an activity against it.

Copy Activity Handles Most of This. Data Flow Is for the Rest.

For a straight move like this one, a Copy Activity is the whole job: source, sink, a column mapping, done. Reach for a Mapping Data Flow only when the data needs actual transformation logic on the way through: joins across two sources, deduplication, conditional splits, aggregation before it lands. Data Flow runs on a Spark cluster under the hood, which means it costs more and takes longer to spin up than a Copy Activity. Using one for a job a straight copy could have handled is a common way a simple pipeline quietly gets expensive.

The Setting Almost Nobody Explains Well: Integration Runtime

Every activity in a pipeline runs on an Integration Runtime. For this pipeline, the default Azure IR is enough, since both Blob Storage and Azure SQL Database sit in the public cloud. That stops being true the moment a source lives behind a firewall: an on-premises SQL Server, or a database inside a VNet with no public endpoint. Reaching those requires standing up a Self-Hosted Integration Runtime, a small agent installed on a machine inside that network, which is the piece almost every "why can't Data Factory see my server" question traces back to. Worth knowing this distinction exists before hitting it blind at 2 a.m.

What Actually Happens When the Trigger Fires

A pipeline that runs perfectly in Debug mode can still fail the moment a real trigger fires, and the reason catches nearly everyone the first time.

Debug mode runs whatever is currently open in the canvas. A trigger only ever runs the last version you actually published, so an edit tested successfully in Debug and never published keeps running the old logic on schedule, silently, until someone notices the numbers look stale. That gap between "it worked when I tested it" and "it's still doing the old thing in production" is one of the most common support tickets on a team running more than a handful of pipelines.

Getting Past the First One

None of these pieces, a Linked Service, a Copy Activity, a trigger, is individually complicated. What takes real repetition is the instinct for where a failure actually lives. A blank sink table usually means check the trigger's publish state, not the activity's mapping. A timeout usually means check the Integration Runtime, not the credentials.

Build this pipeline hands-on, against a real Azure sandbox rather than screenshots, inside SkyTrainings' Azure Data Engineering course.