In our data-driven world, building a scalable, trustworthy data architecture is no longer optional—it’s essential.
As organizations collect data from more sources at higher volumes, the need for a structured, resilient foundation becomes clear. This is where the medallion architecture comes into play, offering a layered approach to organizing and refining data as it moves from raw ingestion to actionable insights.
At the heart of this architecture is the bronze layer, which acts as the starting point for all data operations.
The bronze layer of the medallion architecture serves as the cornerstone of a well-structured data estate, acting as both the initial landing zone for raw data and the single source of truth. Designed as an append-only repository, this layer accommodates diverse data formats, whether structured tables in the lakehouse or unstructured files in OneLake, ensuring a reliable foundation for downstream analytics.
This blog explores two architectural strategies for efficient data ingestion into the bronze layer. It also provides insights into leveraging tools like AzCopy to streamline data ingestion into OneLake, helping engineers build optimized pipelines that ensure accurate, secure, and reliable data flows across their systems.
Data ingestion – push method
For the purposes of our example let’s assume that there’s file data available on an on-premises server and that files are generated on an hourly basis.
The bronze layer acts as the foundational landing zone for raw data. It preserves lineage, allows replay or reprocessing as business rules change, and serves as a single source of truth.
A clear, logical storage hierarchy is crucial in structuring the bronze layer. Organize data by source system or domain (e.g. /bronze/<data domain> /<dataset>/) and partition by ingestion timestamp using folders like /year=2025/month=06/day=02/. This supports efficient filtering, retention policies, and downstream processing. Parquet is generally preferred, but json and csv formats are also acceptable.
At Inviso, we recommend structuring directories one level up from the ingestion rate. Hourly ingestion typically requires day-level folders, while daily ingestion may stop at month-level directories. Below is an example directory structure where we have point of sale data:
The bronze layer serves as an append-only landing zone for raw data, forming the foundation for all downstream processing. Transformations at this stage should be minimal, with the primary focus on table creation and logical modeling rather than heavy data manipulation. This approach ensures a consistent, auditable source that downstream teams can reliably debug or replay as needed. When implemented correctly, the bronze layer enhances long-term flexibility and significantly reduces risk across analytics and decision-making processes.
This model can be applied to both Azure Data Lake Gen2 and OneLake. For sample code, engineers can reference this repository for PowerShell scripts that can be ran on-premises to upload data to ADLS and to OneLake.
Data ingestion – pull method
The pull method involves leveraging cloud technology, like Fabric pipelines, to connect to on-premises systems and load it into the cloud. The key difference to the push method is that the execution occurs in the cloud.
Using Fabric pipelines (Azure Data Factory) engineers will require a data gateway to enable cloud to on-premises connectivity. Once the files are read using pipelines, the files can be ingested directly into append-only tables in the bronze lakehouse. These tables allow engineers to maintain a single source of truth and enable downstream incremental processing with the Fabric Spark runtime.
Incremental processing
When bronze data is ingested as files, there needs to be a secondary pipeline that transitions from files into tables. Typically, trigger-based copy activities using pipelines is the most efficient way of ingesting files into tables. Engineers can follow this process which automatically tracks which files are new or updated allowing you to specify a target into tables.
Once data is loaded into tables (in bronze or silver), then engineers can use Spark Streaming APIs to process data incrementally on a set schedule. Refer to this notebook for guidance on executing incremental pipelines that include complex transformations using the For Each Batch function and Triggers. This approach ensures the stream automatically shuts down after processing the current batch of available data.
This strategy enables engineers to optimize costs, reduce resource consumption, and improve SLAs by focusing exclusively on processing newly added or updated data since the previous batch. By leveraging Apache Spark Streaming APIs, teams can efficiently implement incremental data processing, which not only ensures timely updates but also minimizes the overhead associated with full data refreshes. Additionally, this methodology supports the seamless integration of complex transformations and real-time analytics.
Summary
Data ingestion into Azure Data Lake and OneLake can be accomplished through various methods. Organizations may employ different patterns based on specific use cases but should establish a standardized recommendation that teams within the organization can adhere to. Typically, organizations utilize the push method for on-premises sources or when the data source is managed by a separate team.
Standardizing your ingestion strategy not only improves efficiency and scalability, but also sets the foundation for consistent, high-quality data across your entire estate.
If you’re ready to take your data platform modernization to the next level, connect with our Inviso architects for expert guidance and support!
