Enrich contact data on SAP Customer Data Platform with master data from SAP S/4HANA Cloud using a Kyma Serverless Function

This example demonstrates how to set up a Kyma Serverless Function as an SAP Customer Data Platform pre-mapping Extension endpoint and to use it to enrich contact data extracted from an Amazon S3 Cloud Object Storage bucket with master data from SAP S/4HANA Cloud Business Partner (A2X) microservice, before ingesting it to SAP Customer Data Platform (CDP).

An SAP Customer Data Platform pre-mapping Extension can be used to synchronously intercept the data ingestion flow of SAP Customer Data Platform events that ingest data from supported sources such as Commerce applications, Cloud Storage Solutions, CRM Solutions, Custom Web Applications and SAP Customer Data Cloud.

In this example, email addresses and unique identifiers of contacts are ingested from an Amazon S3 Cloud Object Storage bucket into SAP Customer Data Platform. During the data ingestion process, the flow is intercepted by an Extension (the Kyma Serverless Function) that enriches the request with additional data from SAP S/4HANA Cloud such as the name, phone number and organizations of each of the contacts.

The Kyma Serverless Function used in this example will work with other supported sources as well such as Google Cloud Storage or Microsoft Azure Blob. The only change required is to connect a different source to SAP Customer Data Platform instead of an Amazon S3 Cloud Object Storage bucket that is demonstrated here.

Note: All the functionality presented here are subject to change and may be changed by SAP at any time for any reason without notice.

Scenario

This example includes a Kyma Serverless Function, cdp-extension, that is exposed as an SAP Customer Data Platform Extension endpoint, and demonstrates how to:

Solution Architecture

Sequence Diagram

Suggested introductory readings

Pre-requisites

Deployment steps

Deploy the Kyma Serverless Function

  1. Go to the kyma-runtime-extension-samples repository and clone it. This repository contains a collection of Kyma sample applications including this example (in the cdp-extension subfolder). Download the code by choosing the green Code button, and then choosing one of the options to download the code locally. Alternately, you can also run the following command using your command-line interface within your desired folder location:git clone https://github.com/SAP-samples/kyma-runtime-extension-samplesNote: The source code of this example is in the cdp-extension subfolder of this repository.
  2. Update the values of the following environment variable in the ./kyma-runtime-extension-samples/cdp-extension/k8s/function.yaml file: SAP_API_HUB_API_KEY
    • SAP_API_HUB_API_KEY – This is the API Key of SAP API Business Hub. Login to api.sap.com. Then, go to your profile settings and click on Show API Key to get the value for this variable.
  3. Make the following changes in the ./kyma-runtime-extension-samples/cdp-extension/k8s/api-rule.yaml file:
    • Replace DATA_CENTER with the Data Center that is closest to you. As an example, the US Data Center is us1.gigya.com. For other locations, check Finding your Data Center.
    • Replace BUSINESS_UNIT_ID with your Business Unit ID, which can be located in your SAP Customer Data Platform console URL after /business-unit/.
  4. Create a Kubernetes namespace with the name cdp.
    • kubectl create namespace cdp

    Note: As a prerequisite, please follow the steps listed in the following tutorial: Download the Kyma Runtime kubeconfig

    Alternately, use the Kyma Console User Interface to create a new namespace

    • Open the Kyma console and click on Add new namespace. Enter its name as cdp and click the Create button.
  5. Create/update Kubernetes resources of the cdp-extension serverless function.
    • kubectl apply -f ./kyma-runtime-extension-samples/cdp-extension/k8s/function.yaml
    • kubectl apply -f ./kyma-runtime-extension-samples/cdp-extension/k8s/api-rule.yaml

    Note: As a prerequisite, please follow the steps listed in the following tutorial: Download the Kyma Runtime kubeconfig

    The resources are represented as declarative YAML objects. Applying the resources will perform the following steps:

    • Deploy the Kyma serverless function
    • Expose the serverless function using a Kyma API Rule with JWT Access strategy that will serve as the SAP Customer Data Platform Extension endpoint

    Alternately, deploy the Kyma serverless function and API Rule using the Kyma Console User Interface:

    • Open the Kyma console and select the cdp namespace.
    • Click on Workloads. Then, click on Deploy new workload and select Upload YAML.
    • Then, click on Browse to select the following YAML file, and click on Deploy: ./kyma-runtime-extension-samples/cdp-extension/k8s/function.yaml
    • Repeat the above steps and select the following YAML file. Then, click on Deploy: ./kyma-runtime-extension-samples/cdp-extension/k8s/api-rule.yaml
  6. Go to the Kyma Console –> cdp namespace –> Discovery & Network –> API Rules. Copy the host URL of the cdp-extension API Rule and use it in the last step below.

Update the Schema in the SAP Customer Data Platform console

  1. Login to the SAP Customer Data Platform console.
  2. Go to Dashboard –> Connect –> Customer Schema
  3. Select Profile Entity. Then, click on JSON and enter the following JSON:

    Note: If you already have some custom fields, then only add the phoneNumberDetails and contactPersonBPID fields listed in the JSON below.

    { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "firstName": { "type": "string" }, "lastName": { "type": "string" }, "primaryEmail": { "type": "string" }, "primaryPhone": { "type": "string" }, "masterDataId": { "type": "array", "items": { "type": "string" } }, "crmId": { "type": "array", "items": { "type": "string" } }, "ciamId": { "type": "array", "items": { "type": "string" } }, "emails": { "type": "array", "items": { "type": "string" } }, "phones": { "scopes": [], "type": "array", "items": { "type": "string" } }, "cookieId": { "type": "array", "additionalItems": true, "items": { "type": "string" } }, "searchHistory": { "type": "array", "additionalItems": true, "items": { "type": "string" } }, "phoneNumberDetails": { "type": "object", "additionalProperties": false, "properties": { "phoneNumber": { "type": "string" }, "phoneNumberExtension": { "type": "string" }, "phoneNumberType": { "type": "string" } } }, "contactPersonBPID": { "type": "string" } } }

  4. Select Activities –> Click on the icon with three lines –> Create New Activity. Enter its name as “groups” and click Save. Then, click on JSON and enter the following JSON:
    { "type": "array", "items": { "type": "object", "properties": { "customerBPID": { "type": "string" }, "relationshipNumber": { "type": "string" }, "relationshipCategory": { "type": "string" } } } }

Add a source and create a new event

  1. Login to the SAP Customer Data Platform console.
  2. Go to Dashboard –> Connect –> Sources –> Connect Application –> Select connect within AWS S3 (under Cloud Storage).



  3. Enter the connection details and your AWS credentials as depicted in the screenshots below.




  4. Once the aws-cdp-integration source has been created, click to select it.
  5. Then, click on Create New Event.

  6. In the Settings screen, enter the settings as per the screenshots below and click Next.


  7. In the Model screen, click on JSON and enter the following JSON:
    { "type": "object", "additionalProperties": true, "properties": { "profile": { "type": "object", "additionalProperties": true, "properties": { "UID": { "type": "string" }, "email": { "type": "string" }, "firstName": { "type": "string" }, "lastName": { "type": "string" } } }, "data": { "type": "object", "additionalProperties": true, "properties": { "phoneNumber": { "type": "string" }, "phoneNumberExtension": { "type": "string" }, "internationalPhoneNumber": { "type": "string" }, "phoneNumberType": { "type": "string" }, "contactPersonBPID": { "type": "string" }, "groups": { "type": "array", "additionalItems": true, "items": { "type": "object", "additionalProperties": true, "properties": { "customerBPID": { "type": "string" }, "relationshipNumber": { "type": "string" }, "relationshipCategory": { "type": "string" } } } } } } } }

  8. Perform the following mappings in the Mapping screen:
    EVENT SCHEMA CUSTOMER SCHEMA
    profile.UID PROFILE.ciamId
    profile.email PROFILE.primaryEmail
    profile.firstName PROFILE.firstName
    profile.lastName PROFILE.lastName
    data.phoneNumber PROFILE.phoneNumberDetails.phoneNumber
    data.phoneNumberExtension PROFILE.phoneNumberDetails.phoneNumberExtension
    data.internationalPhoneNumber PROFILE.primaryPhone
    data.phoneNumberType PROFILE.phoneNumberDetails.phoneNumberType
    data.contactPersonBPID PROFILE.contactPersonBPID
    data.groups GROUPS



  9. Click Next. Finally, in the Scheduled Polling screen, click on Extensions.

  10. Under Pre-mapping, enter the name and URL of the extension, which is the host URL of the cdp-extension API Rule. Set the Timeout to 10 seconds, select Stop as the Failure Policy.

  11. Click Save and Done.

Test the solution

  1. Upload the sample input file to your AWS S3 bucket: Upload the input_20211119_001.json file (located in the ./kyma-runtime-extension-samples/cdp-extension/input folder of the cdp-extension repo) to the input folder of your AWS S3 bucket.

    Note: The input_20211119_001.json file contains mock email addresses that are currently used by the SAP S/4HANA Cloud Business Partner (A2X) microservice for Business Partner, Supplier, and Customer master data in SAP S/4HANA Cloud system.

  2. Login to the SAP Customer Data Platform console.
  3. Go to Dashboard –> Connect –> Sources –> aws-cdp-integration –> Load users from AWS S3 –> Ingest Now.

  4. If SAP API Hub is up and running and your Extension has been setup correctly, then the data is enriched during ingestion with additional data from SAP S/4HANA Cloud. To verify this, go to Customers –> Search. Then, check to confirm that the contact data has been enriched and ingested as expected.

Troubleshooting steps

Check the logs of your Kubernetes pods

To see the logs of a specific function, open the function in the Kyma console (Go to Workloads > Functions > cdp-extension) and you will see the logs in an expandable window at the bottom of the page.



Alternately, go to Workloads > Pods in the Kyma Console (within the cdp namespace) to see the list of all running pods. Then, click on the three dots to the right of the running pod of the cdp-extension function, and click on Show Logs to see the logs of the pod.

Or use the following kubectl command to get the list of pods running in the cdp namespace.

kubectl get pods -n cdp

Then, to see the logs of any of the pods, use the following syntax:

kubectl logs <pod-name> -n <namespace-name>

Example:

kubectl logs cdp-extension-jqckp-57787d8cd4-d54nw -n cdp

Conclusion

As you’ve seen, this example demonstrates how to enrich data being ingested to SAP Customer Data Platform using a Kyma Serverless Function that makes external API calls to SAP S/4HANA Cloud Business Partner (A2X) microservice.

The external API calls get specific additional information about the contacts, which is also ingested. You can also extend this example to validate specific information about the contacts from the SAP S/4HANA Cloud Business Partner (A2X) microservice. For example, you could verify the postal code and city of contacts.

As a next step, you could extend this example to support your use case and you could also try using another source instead such as Google Cloud Storage.

Since this is a synchronous flow, it is important to ensure that the extension executes as fast as possible. In this example, parallel API calls have been made to the SAP S/4HANA Cloud Business Partner (A2X) microservice to save processing time and to reduce the latency.

I’d like to request you to kindly provide your feedback or ask clarifying questions related to this post in the comment section below. Additionally, I’d like to invite you to submit any broader Kyma related questions in the Q&A area of the SAP BTP, Kyma runtime topic.

If the SAP BTP, Kyma runtime topic interests you, here are some other links that you may like:

Lastly, if you liked this post, I’d like to request you to kindly hit the like icon, leave a comment below or share this post.