Data not showing from datalake

Question

Data not showing from datalake

Joseph Y. Lee 5

I'm building a dataflow to process data in our data lake, which is populated via a Dataverse link from Dynamics 365 into our Azure environment. When I query the data directly in the data lake, I can see all records, including the most recent updates. However, when I create a dataflow and set the source type to "Workspace DB" pointing to the same Dataverse-linked data lake, it only shows outdated records—specifically data from 2018—and none of the recent updates.

Smaran Thoomu 32,530 Reputation points Microsoft External Staff Moderator

2025-06-23T08:23:45.16+00:00

@Joseph Y. Lee We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Smaran Thoomu 32,530 Reputation points Microsoft External Staff Moderator

2025-06-23T08:23:45.16+00:00

@Joseph Y. Lee We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Chandra Boorla 15,455 Microsoft External Staff Moderator

@Joseph Y. Lee

Thanks for reaching out and providing detailed context.

From what you described, it sounds like your Dataflow is pointing to a "Workspace DB" source that may be referencing an outdated snapshot or cached version of your Dataverse-linked data. Since you're able to see the most recent records when querying the data lake directly, the issue likely lies in how the Dataflow source is configured or refreshed.

Here are a few troubleshooting steps that might help you:

Refresh Timing - The "Workspace DB" may not reflect real-time data and could rely on a scheduled refresh. Please verify when it was last updated.

Source Path Configuration - Ensure the Dataflow is correctly targeting the current location of the Dataverse-exported data. Sometimes data gets partitioned or moved based on export rules, and the path used in "Workspace DB" might still be pointing to an older folder.

Filters in Dataflow - Review the Dataflow for any filters or date-based constraints that might be unintentionally excluding newer records.

Schema or Metadata Changes - If there were any recent schema changes in the Dataverse tables, it's possible that the Dataflow is referencing a structure that no longer aligns with the updated dataset.

If possible, try configuring the Dataflow to read directly from the Parquet files in your Data Lake instead of using "Workspace DB" to confirm whether the issue is with the source layer.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Joseph Y. Lee 5 Reputation points

2025-06-20T16:17:59.6066667+00:00

@Chandra Boorla do you mind outlining the steps in how one can source the dataflow to read the parquet files directly?
Chandra Boorla 15,455 Reputation points Microsoft External Staff Moderator

2025-06-20T16:49:30.62+00:00
@Joseph Y. Lee

Thanks for the follow-up.

Since you're seeing only outdated records when using the "Workspace DB" source in your Dataflow, one effective workaround is to connect directly to the Parquet files stored in your Azure Data Lake—these contain the up-to-date data exported from Dataverse.

How to Read Parquet Files Directly in a Dataflow:

Identify the Storage Path Locate the folder path in your Data Lake where the Dataverse export writes Parquet files. These are usually partitioned by table and date (e.g., year=2025/month=06/day=20/...).

Create a Linked Service (if needed) In your Dataflow environment (e.g., Synapse or Power BI), create a Linked Service that connects to your Azure Data Lake Gen2 storage account.

Add a Source to Your Dataflow

Set the source type to Parquet.

Point it to the relevant folder path.

Enable recursive folder traversal if the data is partitioned by date or other subfolders.

Preview & Validate Load the file preview and confirm it includes the most recent records.

For further reference, this Microsoft Q&A thread explains the process clearly and may help address your use case as well: Read Parquet files from the folder in Dataflow – Microsoft Q&A For additional information, please refer Microsoft documenatation: Parquet format in Azure Data Factory and Azure Synapse Analytics

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Joseph Y. Lee 5 Reputation points

2025-06-20T17:05:21.8+00:00

@Chandra Boorla So we have delta loading every 30 minutes, therefore the folder directory would change as you stated how it's partitioned by table and date. How would the integrated dataset be able to handle it if the folder pathway keeps changing?
Chandra Boorla 15,455 Reputation points Microsoft External Staff Moderator

2025-06-20T17:33:53.1866667+00:00
@Joseph Y. Lee

Great follow-up question! You're absolutely right.

Since you're performing delta loads every 30 minutes and the data lake folders are partitioned by date and time (e.g., year=2025/month=06/day=20/hour=15/minute=30), it's true that the folder path changes frequently. To ensure your Dataflow picks up the latest files without needing to hardcode each new path, here are a few options you can consider:

Recommended approaches:

Recursive folder traversal - If you're using Azure Synapse or Data Factory Mapping Data Flows:

Point your source dataset to the root folder (e.g., ExportedData/<table_name>/).

Enable the "recursively" option in the source settings. This allows the Dataflow to automatically read all subfolders, including the latest partitions.

Use Wildcards (if supported) - Some platforms allow you to use wildcards in the path, such as:

ExportedData/<table_name>/year=*/month=*/day=*/

This helps include all nested folders dynamically, though you may still need to filter the records in the Dataflow if you only want the latest.

Parameterize the folder path (Advanced) - If you want to load only the most recent delta:

Use a pipeline activity (like Get Metadata or a lookup script) to identify the latest folder path.

Pass that as a parameter into your Dataflow or dataset dynamically.

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Joseph Y. Lee 5 Reputation points

2025-06-20T17:38:35.7166667+00:00

So to add context to this I have some questions and I'll be attaching screenshots. The parquet files are located in a separate folder labeled DeltaLake where they are partitioned by year (Screenshot A). However, within the same directory we have the date folders as well that are CSV.

Screenshot A

Screenshot B

How should we go about dealing with this discrepancy?
Chandra Boorla 15,455 Reputation points Microsoft External Staff Moderator

2025-06-20T18:15:34.02+00:00
@Joseph Y. Lee

Thanks for sharing the screenshots, this really helps clarify the structure of your data lake.

From what I see:

The **deltalake/** folder (Screenshot A) contains your Parquet-formatted Delta tables, partitioned by table and likely by date (e.g., year=/month=/day=). These are optimized for analytics and are the recommended source for your Dataflow.

The timestamped folders at the root level (Screenshot B) appear to be CSV snapshots of the data, possibly generated during Dataverse exports at specific intervals (e.g., 2025-06-20T16.26.54Z). These folders are separate from the deltalake/ directory and not intended for direct querying through a Dataflow.

Recommendation:

To ensure your Dataflow pulls the most current and structured data:

Use the Parquet files in the deltalake/ folder as your source.

Configure your Dataflow to point to the relevant *_partitioned subfolder (e.g., deltalake/custinvoicejour_partitioned/).

Enable recursive traversal in the Dataflow settings so it can automatically read all partitioned folders (e.g., by date or other partition keys).

Timestamp As Of (Optional): Use a pipeline parameter like @pipeline().parameters.latestTimestamp for incremental loading

Optionally, apply filters to only process recent records if needed.

Why this works:

Automatic partition handling

Reads all Hive-style partitions (e.g., year=2025/month=06/day=20/...) without needing to hardcode paths

Detects new partitions created every 30 minutes automatically

Delta Lake Advantages

ACID compliance for reliable incremental loads

Schema evolution support for handling upstream changes gracefully

Time travel capabilities with VERSION AS OF or TIMESTAMP AS OF for flexible debugging or backfills

The CSV-based timestamp folders can likely be ignored unless you have a specific use case for them (e.g., raw backups or archival review).

I hope this information helps.
Joseph Y. Lee 5 Reputation points

2025-06-20T19:12:41.94+00:00

@Chandra Boorla This is all very helpful and I will try this out. If I need further assistance can we set up a 1:1 meeting to go over the specifics as well?
Chandra Boorla 15,455 Reputation points Microsoft External Staff Moderator

2025-06-24T18:55:09.1566667+00:00

@Joseph Y. Lee

Following up to see if the above suggestion was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

As your feedback is valuable and can assist others in the community facing similar issues.

Share via

Data not showing from datalake

1 answer

Your answer