Is my dashboard lying to me? Data cataloguing and origin trace back through lineage for SAP sources


📢Note: Before implementing data extraction or write back to/from SAP systems please verify your licensing agreement.

Dear community,

For many companies, SAP ERP is at the heart of daily business operations and therefore, there is a desire by many stakeholders to leverage SAP data for insights. Data often gets curated in shiny slide decks and interactive dashboards, combining multiple data sources, pivot tables, and excel sheets.

How do you really know where the data came from, what it actually contains, how it was processed and where it is being used? No, hearsay is not enough for an auditor or your CEO 😉.

Below a sneak peek 🕵🏾 into the webcast we did on the topic. Scroll further down to get more context and understand the spectrum within SAP + Microsoft.

See below an example that offers tool-driven insight into the “journey” of a data set from a central finance table through a data lake into a curated data set before it gets reported on in a dashboard. On the ECC side you could imagine the equivalent with a table like BSEG.

Be aware that the emphasis of this visual is on the processing outside of the SAP system rather than the structures comprising the source table or view.

Screenshot from Data Lineage view from Microsoft Purview

SAP has a set of opinionated applications to deal with master data management, cataloguing and governance with varying focus and origin to address those needs. Some of them integrate with non-SAP sources like Azure Data Lake, Oracle DBs, AWS, or Google tooling too.

Here is a list of popular products that I personally came across in this space with some explanatory details without any claim for completeness.

SAP Data Custodian Cloud based tool to provide transparency and control to customers in public cloud scenarios. Enables users to influence where their data is stored and processed to support alerts regarding security breaches or policy violations.
SAP NetWeaver Master Data Management Module on top of SAP NetWeaver to deal with master data integration, operations, and quality of interest for the ERP backend.
SAP Master Data Governance (MDG) Allows you to define, enforce, monitor, and improve master data across domains for central or decentralized master data management in hybrid landscapes focussing on S/4HANA. On-Prem or cloud based.
SAP Data Quality Management (DQM) Part of MDG with focus on accessibility, correctness, and completeness. In addition to that: nice features like quality rule mining.
SAP AI Business Service Intelligent service hosted on SAP Business Technology Platform to enable document classification, document information extraction, data attribute and business entity recommendation etc.
SAP Information Steward Tool to make master data accessible in an efficient way increasing data quality. It is tailored for SAP BW and BusinessObjects customers. The tool can be integrated with MDG.
SAP Data Intelligence Cloud Data management solution acting as orchestration layer and data catalogue powered by SAP Business Technology Platform.
SAP Data Warehouse Cloud SAP calls it a “semantic service”, that aims to describe the data of a company in business rather than technical terms. It is powered by SAP Business Technology Platform and connects to a wide array of SAP and non-SAP sources like Azure Data Lake or Azure Data Explorer to name a few.
Microsoft Purview A unified data governance solution that helps you manage and govern your on-premises, multi-cloud, and software-as-a-service (SaaS) data. Enables a holistic, up-to-date map of your data landscape (including SAP) with automated data discovery, sensitive data classification, and end-to-end data lineage. Empowers data consumers to find valuable and trustworthy data.

📖Further Reading

On the Microsoft-side you find Microsoft Purview – an open eco system supporting sources like Snowflake, AWS, Google Looker, Oracle, PostgreSQL, MySQL, Salesforce and of course Azure. For SAP it offers connectors for ECC, BW, S/4HANA and HANA as a database.

Have a look at my post on the TechCommunity for more details and further reading on Microsoft Purview for SAP use cases.

🥇Final Words

What do you think @community? Does the freedom of choice in this space make your decision harder or easier?

As always feel free to ask lots of follow-up questions.

Best Regards

Martin