Final XR Venture Story: UX Design Award Nomination for Augmented Reality app for Service Technicians of BEUMER Group

Dear community,

In this blog post you will read about the final activity from the XR venture that I led for almost two years from 2020 to 2021 together with Thomas Eckert and a team of passionate cloud and 3D developers.

In June 2020, the XR venture team proved that Augmented Reality (AR) object tracking works excellent with the Evomixx machine at the SAP Experience Center in Walldorf. We all were really surprised about the accuracy of the tracking and additional fun it brings for the end user. Instead of spending time to look, search, and scroll for the right spare part, you just hover with the iPad over the machine and get all the valuable insights from the ERP you need for your daily work.

Figure%201%3A%20AR%20object%20tracking%20with%20Evomixx%20machine%2C%20June%202020

Figure 1: AR object tracking with Evomixx machine, June 2020

One year later, my team managed to embed the AR object tracking capability into an app built with SAP BTP SDK for iOS. I wrote about the findings in my second XR venture blog.

But is AR object tracking technology really beneficial for our customers and their service technicians? Does this work on real products?

There was only one way to find it out! Finding a customer who would be interested in participating in a joint PoC. Thanks to the help of Frederick Breckwoldt, I was connected to Michael Brehe, Head of PLM at BEUMER Group. I presented Michael the idea and he immediately liked my proposal and invited us to come to BEUMER Group headquarters to try out AR object tracking on their products.

Arrived at Beckum, I liked very much to hear from Michael that every part in the 3D visualization has a corresponding material number and instance ID (TOS-Id) in SAP S/4HANA. Having a digitalized process is the key to unlock the benefit of AI and AR technologies.

After lunch we went to the factory. So, for me as a software developer, I always enjoy these moments to see where SAP software is running smoothly and reliably in the background while machines and operators are producing real, physical, and great products.

The first product to try out the object tracking was the BEUMER autoca®. This BEUMER product is used at airports to move luggage to its destinations. The object tracking result was again amazing! The autoca® got tracked and the digital model snapped exactly into the physical counterpart and held its position even while it moved slowly up on the maintenance track.

Autoca%AE%20successfully%20tracked%20while%20moving

Autoca® successfully tracked while moving.

The second BEUMER product was the BEUMER stretch hood® A. It is used to package pallets with plastic foils. It was over three meters high. Could it be tracked as well? The answer was yes, although the object was a little harder to catch because of its large size.

Thrilled by these results, we brainstormed together about the additional benefits. The X-ray view capability seems to be very beneficial. It gives service technicians a deep understanding of the machine without the need to remove covers or other machine parts. For instance, the stock availability of spare parts built deep inside in a machine could be checked before the real work begins.

Creating a maintenance report will be easier if the desired equipment can be selected by touch gestures on a tablet. Animated 3D instructions may help to increase the first-time-fix rates for beginners as well.  But it can help also potential buyers to identify in what colors a machine part is available or what product configurations are possible.

We agreed to focus on the maintenance process for the BEUMER autoca®. Their maintenance is crucial to avoid any delays in the luggage transportation process at airports. But how should such an object-tracking-enabled application be designed to work best for service engineers? What hardware should be used?

The hardware question was easy to answer: Mobile tablets are currently the best choice. In the future, the choice would go rather for a hands-free Mixed Reality head mounted device. But today such Mixed Reality glasses have not reached the same tracking experience as tablets mainly due to their weaker performance and limited field of view. Smartphones can do object tracking as well but, due to the smaller screen size, they are not suitable when it comes to process a maintenance report. To answer the second question how the user interface should be designed, I was very happy to get support from Isabell’s design team.

Isabelle Ostermeier is head of UX design for small and medium enterprises (SME) at SAP. Isabelle was fascinated by this new technology and its potential improvement in user experience. She suggested to create a design study to apply for the UX Design Award in autumn 2022. Her design team in Shanghai would help Jan Schneiker to create low- and later high-fidelity mockups of the application for the first iteration of end-user interviews. Jan Schneiker joined the XR venture team to write a master thesis to become Master of Engineering at a state university of applied sciences in Berlin (called BHT). He did the end user interviews with service engineers at several airports, created the UX award submission video and documented the findings in his mater thesis.

After each interview we improved the mockups and added additional features: We learned that AR object tracking is fine but a regular 3D view needs to be offered as well. Michael Brehe had the idea to uniquely identify the autoca® instance with near field communication technology (NFC), so we included this feature into the app design as well.

Select correct autoca® instance with NFC technology

At the end of the submission phase, Jan created a product video with content from BEUMER Group and our design mockups. I created a one-minute-long pitch video for the jury. This was a funny challenge for me as I was at the airport Frankfurt, five minutes before my flight departure to Rome. So it was a one-shot try with me and the virtual autoca®. I am glad that the autoca® behaved well. 😊

Me and the autoca® making a pitch video for the UX Design Award jury

In July 2022, we were happy to hear that our project together with 124 other projects got nominated out of more than 400 submissions. In September 2022 the winner was announced but it was not us. All credits to the winner, our submission was good but not excellent to get a real chance.

But more important than winning the award was for us to make everyone aware of the fact that AR object tracking technology can bring a lot of benefit to service technicians. This mission has been accomplished. Now it is time to bring this technology to SAP products, such for the SAP Service and Asset Manager.

Today I am proud to work as a product manager for Mixed Reality with the mission to bring AR object tracking to the mobile solution roadmaps. This will take a little time, as integrating such a feature into products is a completely other story than doing it in a PoC.

You will read more in upcoming blog posts about these road map items and when this amazing technology is available for you. So, stay tuned!

At the end of my blog, I would like to share our team selfie that we made when we submitted for the UX award. Thank you to all team members for your dedicated contribution – working with you was great fun. We all can be proud of our joint outcome!

From left to right: Michael Brehe (BEUMER Group), Isabell Ostermeier (SAP), Jan Schneiker (SAP), Feifei Yu (SAP), Michael Spieß (SAP), Puneet Mahali (SAP), David Schrott (SAP), Jia Li (SAP), Feng Chen (SAP)

You like to watch award video (2:46)? Follow this link!

Interested to watch AR object tracking in action? Follow this link!