Designing a Snowflake-based Data Lakehouse

Official answer

Define why a lakehouse is needed: open-format interoperability, existing object-storage data, shared analytics across engines, or a governed warehouse experience. Do not assume that calling a set of VARIANT tables a lakehouse resolves those requirements.

Organize raw, validated and serving layers with documented schemas, provenance and freshness. Preserve source data for controlled replay, then publish typed business models at explicit grains.

Choose storage/table management deliberately. Standard Snowflake tables provide Snowflake-managed analytical storage. Apache Iceberg tables provide an open table format with catalog and storage choices. Current supported Snowflake-managed Iceberg configurations can use Snowflake storage or an external volume; external-catalog configurations have their own supported access paths and limits.

Assign the catalog and writer authority for each table. Validate the exact engine/catalog combination, supported read/write operations, snapshot visibility and concurrent-commit behavior before enabling multiple writers. A shared bucket alone is not a transactionally coordinated table.

For customer-controlled storage, configure cloud IAM, encryption, lifecycle and recovery separately from Snowflake roles. Do not delete apparently old files while retained snapshots or other engines still reference them. Verify catalog refresh and reader compatibility after schema or partition changes.

Use workload isolation and supported SQL/Snowpark/ML tooling for consumers, with policy enforcement tested through every engine. Measure freshness, reconciliation, query latency, compute/storage/transfer cost and recovery time. Rehearse catalog, storage and ingestion failures before claiming the architecture is resilient.

Iceberg storage choices, external storage management, external catalog access