Tracking down customer journey with in-store cameras and sensors 

The information gained from tracking a customer journey through a store can be used for many purposes, from optimizing store layout to deciding on additional placements to adjusting offers based on the time of day. 

For a proof-of-concept, Nuremberg Institute for Market Decisions (NIMD) has developed a typology of shopping trips based on path-tracking data collected over one year in a German supermarket, using UWB antennas that monitor battery-operated tags implemented in shopping carts and baskets. Researchers identified eight different shopping trip types based on characteristics like distance covered on a shopping trip, speed, and the proportion of trips spent in specific areas of the store. 

According to NIMD, “unstructured refills” are characterized by a long distance, whereas “single purpose” and “last-minute trips” are attributed to a high speed. Combined with sales and survey data, more detailed behavioral differences between segments can be determined. Such data allows retailers to create targeted and highly personalized recommendations for specific segments, including reminders for abandoned items, optimized category management and secondary placements for single-purpose trips, and special product aisles for last-minute shoppers.

One of the leading European retail companies once turned to us for help building a custom AI and Big Data solution to solve their challenges, namely:

1. High inefficiency in defining customer clusters by gender and age, resulting in slow follow-ups across departments and locations due to variability in operations.
2. Stressful implementation of immediate response programs.
3. Lack of understanding of customer behavior.

Using CCTV cameras and video analytics servers (including Facial Recognition, Queue Detector, People Counter, Activity Visualizer and Analytics system package for retail), our software engineering team built a solution to map each client’s in-store journey, identify promptly their needs and shopping behavior and solve even the most sophisticated requests. In particular, we employed:

• Windows service for data synchronization.
• Custom-built React JS-based web application to store, view and compare data against various criteria.
• Third party software integration for face recognition, queue and people count analysis.
• Third party cameras with face recognition, people count and heatmap processing functionalities.

This custom retail analytics application helped our retail client:

• Increase customer insights generation by 90%, 
• Increase the immediate response rate by 50%,
• Increase accuracy of customer patterns by 60%. 

Creating a unique customer faceprint with Computer Vision

customer identification at the point of sale (POS)— via loyalty cards—is no longer good enough for retailers. They want to know who the people in the store are before they make a purchase to provide a personalized shopping experience. However, most stores don’t have the luxury of recognizing every face that walks through the door.

This is where facial recognition software can step in by scanning thousands of checkpoints on a person’s face to identify their unique “faceprint” (just like we all have our own unique fingerprints).

Identifying customers in this way gives your employees insight into the types of products customers can purchase and how much they can spend so they can roll out a VIP service for these high-value customers.

Facial recognition technology can help retailers better respond to customer needs by deciphering any changes in your customers’ emotions and then notifying your employees so they can tackle upset or confused customers right away.

Another obvious value of this technology is loss prevention, which is especially useful given that US retailers lose almost $50 billion annually due to theft (shoplifting accounts for 36.5% of these losses). Your security staff may not notice a stalking shoplifter, but the software certainly can. It can compare faces against a database of known shoplifters to provide proactive theft prevention. Facial recognition in retail reduces shoplifting by 34%. This means better value for your customers and more money in your pocket.

All this may seem rather Orwellian. While facial recognition is a way to give people the personalized experience they want (for example, 41% of consumers expect representatives in a brand’s physical store to know what they’ve bought online), there will also be shoppers who find it a flagrant violation of their private life. Thus, to address any ethical issues, facial recognition should only be used if someone has explicitly consented (instead of, for example, using assumed consent).

Shoppers can sign up for the scheme and use their smartphones to scan their faces or maybe even visit a special booth in the store. Whatever the mechanism, it is vital that retailers maintain trust by giving shoppers control of the experience.

Automatic analysis of customer faces captured on the camera allows retailers to make conclusions about emotional states in real-time. Combined with eye-tracking technology, retailers can, for example, determine a customer’s need for information and support. 

Optimizing customer experience and sales with Virtual Reality (VR)

Online shopping website providers can easily use A/B tests to optimize customer experience and turnover. It is much more difficult and expensive for regular stores to experiment with display, assortment, or prices. VR offers retailers an environment for controlled experiments. For example, you can quickly and efficiently compare different store layouts in VR.

A big advantage compared to real A/B tests is a high degree of experimental control over the situation and the ability to obtain comprehensive information about user actions and perception. Our experience shows that VR is well suited for predicting purchasing decisions.

In collaboration with the Karlsruhe Institute of Technology (KIT) and CITEC at the University of Bielefeld, a team of researchers used a collaborative choice-based solution to compare choices made in an online shopping environment on a desktop screen with those made in front of a fully immersive 3D supermarket. 

Designing and programming 3D models for VR still requires a lot of time and money. However, subsequent adjustments to existing virtual worlds can be made more easily once programmed.

Supporting informed decision-making with Augmented Reality (AR)

Perhaps the flagship technology for creating the physical future comes in AR/VR. And the customers seem to agree. According to one report, nearly two-thirds (61%) of American adults surveyed said augmented reality has impacted where they shop.

The COVID-19 measures have played a huge role in boosting demand for augmented reality systems, enabling shoppers to adopt a try-before-you-buy approach in absentia and helping to replicate and even evolve the online store experience. 

IKEA, for example, has created an app called “Ikea’s Place” that allows shoppers to access 3,200 IKEA products using the live view feature on their smartphone and use AR functionality to see what certain items will look like by placing them virtually in rooms of your house. home before making a purchase.

AR also promises to radically change the in-store experience. In fashion retail, smart mirrors will provide a previously unthinkable level of interactivity. Shoppers won’t even have to undress to try out a brand new wardrobe, with virtual fitting rooms allowing them to layer items and outfits on top of each other and get friends’ opinions by sharing images on social media.

Oak Laboratories uses similar technology to create interactive locker rooms. Smart mirrors recommend other products that shoppers often pair with products shoppers try, helping to complement their outfits and providing an in-store online shopping experience. This process also produces a lot of rich data. For example, you can see the conversion rates for certain items from the fitting room to checkout and adjust products and inventory accordingly.