OrchardCore.DataPipelines: More Than Data Export — A Foundation for Data Integration and Reporting

One of the most valuable additions to the Orchard Core ecosystem is the new OrchardCore.DataPipelines feature.

At first glance, you might think this module is simply another way to export data from your Orchard Core application. While exporting data is certainly one of its capabilities, that perspective overlooks its real value.

OrchardCore.DataPipelines introduces a reusable infrastructure for defining data sets, manipulating and transforming data, performing calculations, and ultimately consuming that data in different ways.

It was designed around the well-established ETL (Extract, Transform, Load) data integration pattern, providing a structured and extensible way to work with data across your Orchard Core application.

Let's explore why this is such an important addition and how it opens the door to much more than data exports.

Understanding the ETL Approach

ETL stands for Extract, Transform, and Load. It is a widely adopted data integration process used to collect data from different sources, prepare it for a specific purpose, and deliver it to a destination.

OrchardCore.DataPipelines brings this concept into Orchard Core.

The process consists of three main stages:

1. Extract — Define Your Data Sources

The first step is identifying where your data comes from. Data Pipelines provides a consistent infrastructure for reading data from different sources within your Orchard Core tenant.

For example, you might retrieve:

• Content items of a particular content type.
• Users and their associated information.
• Results from saved queries.
• Data from custom sources provided by other modules.

These sources provide structured data sets containing fields that can be used in subsequent pipeline steps.

2. Transform — Manipulate and Enrich Your Data

Extracting data is only the beginning. In most real-world scenarios, the original data needs to be prepared before it becomes useful.

This is where the transformation capabilities become particularly powerful.

You can:

• Filter, select, and sort data.
• Group, join, and combine data sets.
• Deduplicate data.
• Introduce calculated fields using formulas.
• Derive new values from existing fields.

For example, imagine retrieving customer orders. You could calculate order totals, group orders by customer, filter by a specific date range, or combine order information with customer details.

Instead of writing custom code for every data-processing requirement, you can define these operations as reusable pipeline steps.

3. Load — Deliver or Consume Your Data

Once the data has been extracted and transformed, it is ready to be consumed.

The Data Pipelines module provides several ways to deliver the resulting data, including:

• Generating CSV files and Excel workbooks.
• Saving files to the media library.
• Sharing download links.
• Sending files by email.
• Uploading files to FTP or SFTP servers.
• Sending data to web APIs.
• Creating or updating Orchard Core content items.

The important part is that the destination is not fixed. The infrastructure is extensible, allowing developers to introduce their own destinations and integrate with virtually any external system.

You can learn more about the architecture and available capabilities in the Orchard Core Data Pipelines documentation.

Why Is This Module So Important?

Here is where I believe the real value of OrchardCore.DataPipelines becomes apparent.

Its importance is not simply that it allows you to export data from your Orchard Core application. Its importance is that it provides the infrastructure to define data sources, manipulate that data, and make it available for different purposes.

Think about it for a moment.

Before having a shared data-processing infrastructure, individual features that needed to work with data would often have to implement their own mechanisms for retrieving, filtering, transforming, and calculating values.

This can lead to duplicated logic, inconsistent implementations, and additional maintenance overhead.

Data Pipelines provides a common foundation that other modules and features can build upon.

And here is an important architectural detail that makes this possible.

Data Pipelines Core: You Don't Necessarily Need to Export Anything!

The Data Pipelines functionality is divided into two features:

• OrchardCore.DataPipelines.Core
• OrchardCore.DataPipelines

The distinction between these two features is extremely important.

OrchardCore.DataPipelines.Core provides the underlying infrastructure that other features can reuse. This includes:

• Data sources.
• File formats.
• The formula language and its functions.
• Data operations.
• Shared links.

It is designed to be enabled as a dependency of other features rather than directly by administrators.

On the other hand, OrchardCore.DataPipelines provides the actual pipeline designer, execution capabilities, and steps for reading, transforming, generating, and delivering data.

This separation introduces an interesting possibility:

What if you want to retrieve data, transform it, calculate values, and do something with the results without exporting anything?

You absolutely can!

You don't need the Data Pipelines UI or its export-oriented functionality to benefit from the underlying data infrastructure.

Another Orchard Core feature can depend on OrchardCore.DataPipelines.Core, access the available data sources, apply transformations, perform calculations, and consume the resulting data directly within its own functionality.

In other words, the Core feature is not tied to exporting data. It is a reusable data-processing foundation.

And this architectural decision is what brings us to another interesting feature.

From Data Pipelines to Report Builder

Fun fact: The idea of building a Report Builder was actually one of the original motivations behind introducing OrchardCore.DataPipelines!

We wanted to provide enterprise users with a flexible way to gain insights into their business data without having to write custom queries or develop individual reporting interfaces for every business requirement.

Think about the possibilities.

If we already have:

• A consistent way to define and retrieve data sources.
• A way to identify and work with individual fields.
• A way to manipulate, filter, and combine data.
• A way to calculate new fields and aggregate existing values.

Why should we limit ourselves to exporting that data?

Why not visualize it?

We could present the same underlying data as tables, charts, metrics, or combinations of different visualizations.

And that is exactly the idea behind CrestApps.OrchardCore.Reports.Builder.

Introducing CrestApps.OrchardCore.Reports.Builder

The Report Builder takes advantage of the infrastructure provided by OrchardCore.DataPipelines.Core to turn data from your Orchard Core tenant into meaningful, interactive reports.

Rather than requiring developers to create custom reports for every business requirement, the Report Builder provides a user-friendly, drag-and-drop interface that allows users to design their own reports.

Users can:

• Select the data sources they want to work with.
• Explore the available fields and choose the information they want to display.
• Join data sets to bring information from different sources together.
• Create calculated fields using formulas.
• Apply filters, sorting, and aggregations.
• Format and customize the presentation of their data.
• Visualize information using tables, charts, metrics, and other visual components.

For example, an enterprise might want to build:

• A sales report showing total revenue by month.
• A customer report showing order history.
• An operational dashboard displaying key business metrics.

Different departments may need to look at the same underlying data in completely different ways.

The sales team might prefer charts showing revenue trends, while management might want high-level metrics and financial summaries. Another department might need a detailed table containing individual records.

The Report Builder makes these scenarios possible without requiring a separate implementation for each one.

The key is that the data retrieval and transformation capabilities are already provided by OrchardCore.DataPipelines.Core. The Report Builder simply consumes that infrastructure and provides a reporting-oriented experience on top of it.

You can explore the Report Builder documentation to learn more.

The Bigger Picture: A Foundation for More Than Reporting

Although reporting is an excellent example of what can be built using Data Pipelines Core, it is certainly not the only possibility.

Once you have a consistent way to retrieve data, manipulate it, and perform calculations, you can build many other features on top of the same infrastructure.

Imagine:

• Dashboards: Retrieve and aggregate business data to display real-time or periodically refreshed insights.
• Business Intelligence: Build analytical features that combine information from multiple data sources.
• Integrations: Transform data into the format required by external services and APIs.
• Automated Business Processes: Evaluate data and use the results to trigger custom application logic.
• Custom Reporting Solutions: Build specialized reporting experiences tailored to particular business domains.
• Data Synchronization: Retrieve, transform, and prepare data for synchronization with external systems.

The possibilities are not limited to the destinations already provided by the Data Pipelines module. Developers can extend the infrastructure to support their own data sources, transformations, and consumers.

This is the real benefit of having a shared foundation: different features can solve different problems without reinventing the underlying data-processing capabilities.

Final Thoughts

I believe OrchardCore.DataPipelines is an important step forward for the Orchard Core ecosystem.

It introduces more than an export mechanism. It establishes a reusable architecture for working with data, from retrieving it and transforming it to making it available for different application scenarios.

And perhaps the most important architectural decision is the separation between OrchardCore.DataPipelines.Core and the Data Pipelines UI.

It means that developers are not forced into a particular workflow or destination. They can leverage the underlying infrastructure and build entirely different experiences on top of it.

The Report Builder is one example of this philosophy in action. It demonstrates how the same data infrastructure can power a completely different user experience, turning application data into meaningful business insights.

Define your data once. Transform it as needed. Consume it wherever it makes sense.

That is the bigger vision behind OrchardCore.DataPipelines, and I am excited to see what the Orchard Core community builds on top of it.