The method for a deeper understanding of user experience using customer rating analytics: A study in tourism sector

Abstract

The level of competition between businesses in the post-COVID period will be fierce, especially those in the tourism sector according to a survey from the World Economic Forum, wherein the tourism industry is the industry o suffer heavy losses during the COVID-19 pandemic. Businesses are actively applying technology and digital transformation to increase revenue, cut costs, and optimize resources. In general, in a business, there will be many factors that need to be taken care of and one of them is customers because customers are the ones who directly generate revenue for the business, helping the business to maintain its existence and develop. Therefore, understanding customer experience is an incredibly important step in business. From this, businesses can evaluate the effectiveness of the business model. This research approaches a new Net Promoter Score (NPS) to measure how willing customers are to recommend a company's products or services to others. This score contributes to helping enterprises reassess their business models and indirectly reflecting the growth ability of enterprises. The novelty of this research is the application of Business Intelligence (BI) solutions to support multidimensional analysis, thereby solving the weakness in the traditional NPS, which is based on only one factor. Also from multidimensional analysis, businesses will visualize customers more clearly, thereby taking appropriate and timely actions. This research has three main objectives, including (1) Collect data and measure NPS automatically; (2) Design the Dashboard system to solve complex problems in retrieving, exploiting and synthesizing information to measure NPS, which supports businesses to make timely and more effective decisions; (3) Experiment the solution with data crawled from Agoda. Customers' NPS remarks can be investigated further. Enterprise will discover what customers are lacking in their product, which features they like, which features they don't comprehend, and what additions they should build in the future.

Introduction

The prolonged and complicated COVID-19 pandemic has affected all aspects of life, the economy, and society, in which the business community has been dramatically affected, especially for small and medium enterprises. In the current post-COVID period, many businesses have been rapidly reorganizing production, adopting digital transformation, and changing business methods to adapt to the crisis's difficulties and create development opportunities. In this regard, measuring customer loyalty to evaluate the effectiveness of the business model is an essential step. Whilst the positive feedback will ensure that the business model is working well, the negative feedback will help managers rethink the business model, service quality, infrastructure, etc., so that appropriate adjustments can be made to regain trust from customers who help businesses witness revenue and profit growth.

Of all the sectors, travel and tourism are perhaps hit hardest by the COVID-19 pandemic. Research 1 shows that in 2020 alone, the travel & tourism industry worldwide has lost 4.5 trillion USD due to the COVID-19 pandemic, with 62 million jobs lost. In Vietnam, data from the General Statistics Office (GSO) shows that in 2020, the number of international tourists only reached 3.7 million people, which decreased by 79.5% compared to 2019, and domestic tourists also reduced by 34.1% (56 million in 2020 compared to 85 million in 2019), the Vietnam National Administration of Tourism has estimated losses of up to $19 billion ($13 billion in 2020 compared to $32 billion in 2019) 2 .

In the context of the COVID-19 pandemic being gradually controlled by the government, the tourism industry and the economy are gradually going through a difficult period and entering a recovery phase. However, the pandemic has affected people's living habits, activities, and travel habits. With the trend of approaching safe tourist destinations, small group travel, short-term travel, and near-date scheduling, the trend of choosing resort activities close to nature will take the throne. Therefore, businesses must adopt changes to keep up with user trends. This research approaches a new way of assessing the NPS index, applying BI solutions to meet the need of supporting management to make timely and more effective decisions. Given the competition between businesses increases, businesses that take timely actions primarily based on data to make decisions will help businesses increase their competitive advantage. This study also uses data from the Agoda platform (Agoda is an online booking service provider for hotels mainly in the Asia Pacific region) to experiment with the solution. The research will crawl Review data of users from hotels in Vietnam on the Agoda platform to conduct an NPS analysis.

Related works

Frederick F. Reichheld proposed NPS in his research 3 to measure customer loyalty to their products and services by surveying users based on the question "How likely is it that you would recommend [company X] to a friend or colleague?". Surveyors would answer the question by giving a score from 0 to 10. Based on the survey results, three groups of customers will be defined based on their scores, including 1) Detractors are those who give scores from 0 to 6. This group of customers is considered those who are not satisfied with the service. They are ready to leave the brand; 2) Passives are those who give scores from 7 to 8. This group of customers do not love the service but also do not complain about the service; 3) Promoters are those who give scores from 9 to 10. This group of customers are delighted with the service and are ready to recommend to friends and relatives use the service they have used again.

Figure 1 . How to calculate Net Promoter Score 4

4" width="300" height="200">

Figure 1 describes how the NPS is calculated by subtracting the Promoters group's rate from the Detractors group's ratio. The NPS will be in the range [-100:100]. The more NPS value indicates that the customers of [company X] are more loyal to the company's products and services.

Figure 2 . Business Intelligence Solution (Source: The authors synthesize and propose)

Howard Dresner described BI (Business Intelligence) as concepts and methods for improving business decision-making using fact-based support systems 5 . BI solution ( Figure 2 ) consists of 4 main components: 1) Data Sources: Data can be stored in different environments, heterogeneous in origin and type such as excel, text, website, database; 2) ETL Process (Extract, Transform and Load): Data collected from multiple sources. However, only necessary information need to be extracted from each data source . From that information, it needs to be converted into a particular structure, then pushed into the Data Warehouse; 3) Data Warehouse is a collection of data that is structurally homogenous, relatively stable, and updated over time. The data warehouse can collect many Data Mart, which is subject-oriented data; 4) Report, Dashboard, and Analysis: Data from the Data Warehouse is displayed in the form of dashboards, reports for analytical methods to support decision making.

In this research, the solution does not load data directly to Data Warehouse but will load into the database all the data that has been crawled from Agoda, combined with geographically hierarchical data. Depending on the analysis needs, specifically, in this case, NPS analysis, the ETL process will be performed to extract, transform and load the necessary data from the database to the Data Warehouse.

For research 6 and research 7 , the NPS index was applied in healthcare. Meanwhile, research 8 applied the NPS index in evaluating educational programs. These researches have shown the importance of NPS in measuring customer loyalty. Research 9 also shows relationship and efficient management, including 1) High levels of customer satisfaction increase NPS; 2) High levels of satisfaction measured in key areas such as reception, cleanliness and room comfort, and Gastronomy increase NPS. In addition, research [10] also shows the importance of NPS when predicting sales value.

However, many researchers also argue that the NPS index is just a fallacy index. It does not explain the root cause of the detractors' group 10 . Besides, relying on just one metric is risky, and companies encourage a more multidimensional analysis 11 . According to the traditional model ( Figure 3 ), the NPS index is measured through a survey sheet, the usual analytical process according to the manual method 4 .

Figure 3 . The traditional approach to assessing the NPS 4

4" width="300" height="200">

This is considered a weakness of the NPS index. From this weakness, the research proceeds to develop a new approach to assessing the NPS index by using BI solutions so that the NPS index is considered under a multidimensional perspective.

This research also uses online survey data due to its advantages, including 1) Easier access to new populations; 2) Larger sample sizes, 3) Lower cost; 4) More timely data collection and reliable data; 5) Anonymity of participants 12 . In this regard, the research relied on the customer rating score on the Agoda platform which considered a score to evaluate the NPS index. This data can be more reliable because users are not forced to rate or review. The research looks like a user survey in the traditional NPS measure method for each customer review. From these reviews, by applying BI solution, the research can extract the dimensions including time, province/city, hotel, travel type, geography... From there, the NPS evaluation process will be more multidimensional. This is a contribution of research to fill in the research gap presented above.

Methodology

This research proposes a solution consisting of 3 main stages ( Figure 4 ), including Phase 1) (1) (2) Crawl data and load Master Data into the Database; Phase 2) (3) Through the ETL process, build the Data Warehouse according to the Star Schema; Phase 3) (4) Building a Dashboard system to measure NPS according to End User's needs using Power BI tool. The system as a whole is considered a complete BI solution.

Data in this research includes: 1) (1) Master Data which is geographically hierarchical data; 2) (2) Data extracted from Agoda by crawling with information including city, hotel, type of travel, reviews, rating, etc... This data will be loaded into the database, where raw data is aggregated. From there, depending on the business analysis needs of each business, specifically in this case measuring the NPS index, the research will (3) Extract the necessary data, then Transform and Load it into the Data Warehouse. This is also a dataset to import into Power BI Desktop to perform Data Visualization (4). The research was conducted to create Dashboards to support businesses in making decisions based on the NPS index through data dimensions such as time, hotel, geography, room type, travel type, language comment, and stay length. The system will be automatically updated via activation trigger on a weekly (5) or daily basis, depending on analysis needs.

Figure 4 . The new approach to assessing the NPS index by BI solution (Source: The authors synthesize and propose)

Data Crawling

Input: Names of 63 provinces/cities in Vietnam.

Output: Hotel information, comments, ratings, etc... of each hotel in each province/city on the Agoda platform.

This research collected data from 2007 to 2020 with 1,272,080 observations on the Agoda platform. Research and develop a Framework to approach a new way of assessing NPS. Agoda is seen as a unit for experimentation. The change in the enterprise does not affect the presented model ( Figure 4 ).

Table 1 Structured data collected from Agoda

Table 1 describes the data crawled from Agoda, including information related to hotels, users, and services, along with ratings and reviews. In this information, "Rating" is the score for each survey sheet in the NPS index measurement model. Based on "Rating" the research divided the dataset into Detractors, Passives, and Promoters.

ETL Process

All information collected from Agoda is stored in a Database built with SQL Server Database Management System (DBMS). Geographically hierarchical data was also loaded into the SQL Server DBMS in this research. Depending on the analysis business, the developers will Extract, Transform and Load the necessary data to proceed with the construction of the Data Warehouse.

From the collected data, combined with the need to analyse the NPS index, the research proceeds to transform the raw data into Views that are considered Dimensions and Fact.

This research only considers ETL as a small stage, and the data is at a superficial level so that it can be transformed by T-SQL right on the SQL Server DBMS. From the Power BI tool, researchers import data from the Views into the Power BI model to directly do Data Modelling.

In the process of data transformation, the research considers converting the rating score to a group of points to evaluate the NPS index. Rating on the Agoda platform is in the range of [0.0 : 10.0]. Therefore, the research conducted to convert the score for the Detractor group is [0.0 : 6.0), Passives is [6.0 : 8.0) and Promoter is [8.0 : 10.0].

Data Modelling

Figure 5 . Model of star data warehouse (Source: Experimental Results)

The research conducted Extract, Transform necessary data and Load into Power BI Desktop to build a Data Warehouse model ( Figure 5 ). The model includes 10 Dimensions, including 1) Dim_Date – Description of time information; 2) Dim_NPS – Describes three groups of NPS index: Detractors, Passives, and Promoters; 3) Dim_Group_Rating – Rating by category classified by users; 4) Dim_Stay Length – Describes how long the customer has rented the service at Agoda; 5) Dim_Reviewer – Description of the reviewer's information; 6) Dim_Hotel – Description of hotel information; 7) Dim_Room – Description of room information; 8) Dim_Travel – Description of travel type information; 9) Dim_Geography – Description of the hierarchical geographical data; and 10) Dim_Language – Describes the language user selects for review. These dimensions combine to create Fact_Comment, which stores rating information.

Results and Discussion

The Dashboard was born to help synthesize information without reviewing many small reports, helping administrators read data to make quick decisions. Therefore, the construction of the Dashboard also needs to adhere to some principles. Otherwise, the Dashboard will be flooded with information, making it difficult for decision-makers and not knowing where to start.

Data and information on the Dashboard should be represented in 3 levels: 1) Overview: NPS index; the Detractors, Passives, and Promoter index; Observations; 2) Focus: The graph shows the level, frequency, comparison; 3) Detail: Tables represent data at a detailed level. According to the viewport, the data level should also be uniformly presented from left to right or top to bottom.

Research has been conducted to build a Dashboard system applying the principle of 3 levels of information, including 1) Overview; 2) Focus; 3) Details. This principle is presented in detail in Leva Dobraja et al. (2020) 13 .

Figure 6 . Principles of building Dashboard 13

13" width="300" height="200">

The research proposes a design pattern according to the above principle ( Figure 9 ). The design can be divided into five main areas, including 1) Dashboard title; 2) Filtering criteria according to the necessary dimensions for the Dashboard, helping to support analysis. 3) The leading indicators of the Dashboard follow the principle of level 1; 4) The tables and graph comparing the NPS index and the Detractors, Passives, and Promoter values, this area also follows the level 2 principle in Dashboard building; 5) The detailed information display area is usually presented in the form of a Matrix/Table, showing the level of detail of the data according to the level 3 principle.

The Dashboards will follow this template. However, depending on the specific conditions of the analysis requirements, some vital areas may be added or removed.

This research has collected data from Agoda for experimentation. The Dashboards below are assumed to be built from the board of directors' requirements, the Agoda platform's management board. The analysis results are intended to help support decision-making for management.

Figure 7 . Dashboard describes overview of NPS (Source: Experimental Results)

Figure 7 depicts the NPS index of the Agoda platform with an NPS of 62.78%, in which the Detractors, Passives, and Promoters indexes are 8.16%, 20.91%, and 70.93%, respectively. The total number of observations without filtering is 1,272,080 observations. In general, Agoda's NPS index is very high. This NPS index shows that 62.78% of people who have used the Agoda platform are ready to recommend friends and relatives to use Agoda's services. However, as a manager, you need to go deeper into the analysis. When analysing the dimension "Stay Length". The data shows that the NPS is extremely low for users at 22 days, only 9.09% (50.00 – 40.91), including 22 observations. From here, the manager needs to deeply analyse the review and rating to see if there are any abnormalities in the service, management or time, type of service, and which hotel has a low rating to support the hotel in the future, such as improving services, facilities, etc...

This dashboard also follows the 3-level template. However, because it is an overview Dashboard, the 3rd level (Detail) is not shown. Instead, simple drag and drop operations on Power BI tools will help managers when they need to observe detailed data. This is also the reason why research chooses the Power BI tool to support data visualization.

Figure 8 . The table details some observations related to users with a stay of 22 days (Source: Experimental Results)

Figure 8 shows some observations related to user reviews who had a stay length of 22 days. Viewing data at a detail level to check-in time, type of travel, hotels, etc... Agoda platform managers will work with suppliers (Hotels) to improve services or provide customer feedback if there is a mistake affecting Agoda's reputation.

Figure 8 is also considered the 3rd level of 3 levels of Dashboard building to support administrators in data-driven decision-making.

Figure 9 . Dashboard describes NPS index around Travel type dimension (Source: Experimental Results)

Figure 9 shows the NPS through the "Travel Type" base. The KPIs at the overview level are shown in the same way as in ( Figure 7 ). When analysing more thoroughly in terms of time, the Dashboard shows that the NPS has increased steadily from 2007 (42.29%) to 2020 (69.26%), increasing by 26.97%. Analysis of each type of Travel Type (Business Travel, Couple, Family with a Baby, Group, Solo Travel) in general also shows that the NPS index has increased steadily. Only the "Family with a Baby" type has shown signs of decreasing from 2007 (75%) to 2020 (65%) down 10%. These are considered insights to help the Agoda platform work with partners to understand the causes and reasons why NPS is tending to decrease in this type of service and find ways to overcome it.

Figure 9 fully demonstrates the components in the design that the research proposes. At the 3rd level (Detail), this Dashboard shows NPS by each Travel Type combined with time and geographically hierarchical data.

In each Dashboard, the values in the filter area will also be beneficial to assist managers in analysing data and uncovering insights from which to have more in-depth analysis.

Figure 10 . Dashboard for detailed analysis of NPS by geographic hierarchy (Source: Experimental Results)

Figure 10 shows an NPS analysis of geographically hierarchical data. By combining hierarchical geographic data (Domain Region Province/City), the research has built Dashboard in the form of a decomposition tree. The overall NPS is 62.78% for the entire Agoda platform. NPS continues to be viewed by each Domain (Vietnam geographical data divides Vietnam into three domains), including North has an NPS of 69.51% (including three regions), Central has an NPS of 67.42% (including three regions), South has an NPS of 51.58% (including two regions). According to this data, it can be commented that the NPS in the South is much lower than the overall NPS of the Agoda platform and compared to the other two domains, North and Central. Based on this insight, the Agoda platform management will also have a deeper analysis of domain South's data to find out the cause. In addition, for a region with many provinces, the NPS index is considered for each province. As ( Figure 10 ), the NPS value is determined in the range [-100:100], all the NPS values of the domain, region, and province/city are positive, only that of "Thai Binh province" is -29.41%. It can be commented that the hotels in Thai Binh province on the Agoda platform are being rated very low. Customers will abandon hotels in Thai Binh province on the Agoda platform and will not be willing to refer customers to these hotels, and especially in Thai Binh province where they have negative feedback on the quality and service of these hotels on the Agoda platform. This is a valuable insight for the Agoda platform and partners in Thai Binh province. Besides, the color and decomposition tree in this visual also helps administrators easily capture this insight.

The Dashboard system is designed on the Power BI Desktop system in this research. Report data is taken from the Power BI dataset from the data warehouse model designed according to star schema. After completing the data visualization operations and building the required Dashboards, the reporting system will go through the testing process and publish the product to the Power BI service. Users will use the product through this system, not the Power BI Desktop file. The final step in configuring the Power BI service is to schedule automatic updates. This scheduling is similar to the time the user needs to update the data through the data crawl process. Scheduling can trigger by day, by week, or by a fixed timeline.

In performing data visualization, the DAX (Data Analysis Expressions) language is studied and applied to process and calculate many values. This research also chooses Power BI as the platform for data visualization because of its low cost and good performance. Power BI provides hundreds of chart templates such as pie charts, column charts, line charts, domain charts, combo charts, or many other tables, maps, and charts. In addition, Power BI also allows users to create their own report templates for each request using ReactJS to execute and then embed reports in Power BI. Besides, users can also write code in Python or R to visualize data into charts.

Conclusion

In summary, the research results have achieved the objectives that set out both the meaning of improved method, model, and practice. In which the results include three main contributions: 1) The research proposed a new way to evaluate the NPS index through BI solutions with multiple data dimensions to fill the gap of previous studies, which is the evaluation of the NPS index based on only one factor; 2) The research proposed a template and built a Dashboard system on the Power BI tool to evaluate NPS in multiple data dimensions applied the 3-level information principle; 3) Successfully tested the solution by evaluating the NPS index for the Agoda platform. This is a premise that research can be applied in enterprises.

One weakness in this research is that the rating entered by the user can be biased, which might affect the NPS result. However, with platforms like Agoda, users will often have their reviews besides ratings. It would be inappropriate where there is a case where the rating is 10.0, but the review is full of criticisms about the service and facilities for example. This is the driving force for research to continue to be developed. A solution to re-verify ratings based on reviews will help improving the solution and making the NPS more reliable.

ACKNOWLEDGMENTS

This research is funded by Vietnam National University Ho Chi Minh City (VNU-HCM) under grant number DS2022-34-01.

ABBREVIATIONS

BI: Business Intelligence

DAX: Data Analysis Expressions

DBMS: Database Management System

ETL: Extract, Transform and Load

GSO: General Statistics Office

NPS: Net Promoter Score

T-SQL: Transact - Structured Query Language

COMPETING OF INTERESTS

The authors declare that they have no conflicts of interest.

AUTHORS’ CONTRIBUTIONS

All members proposed and built the method first. Thien Le carried out the collecting, cleaning data and build data warehouse. Van-Ho Nguyen and Thien Le calculated NPS based crawled data, and Thanh Ho and Thanh-Tuyen Huynh visualized data into dashboards. Thanh Ho, Thien Le and Van-Ho Nguyen evaluated the results and prepared the draft manuscript, then Thanh Ho and Mai Thi Cam Tu were responsible for language editing and response to the reviewers.