Examining the Impact of search Engine Ranking and Personalization on Consumer Behavior: Combining Bayesian Modeling with Randomized Field Experiments Anindya Ghose, Panagiotis G. Ipeirotis, Beibei Li2 Department of Operation and Information Management Wharton School of Business, University of Pennsylvania Department of Information Systems Stern School of Business, New York University Abstract In this paper, we examine how different ranking and personalization mechanisms on product search engines influence consumer online search and purchase behavior. To investigate these effects, we combine archival data analysis with randomized field experiments. Our archival data analysis is based on a unique dataset containing approximately 1 million online sessions from Travelocity over a 3-month period. Using a hierarchical Bayesian model, we first jointly estimate the relationship among consumer click and purchase behavior, and search engine ranking decisions To evaluate the causal effect of search engine interface on user behavior we conduct randomized field experiments. The field experiments are based on a real-world hotel search engine application designed and built by us. By manipulating the default ranking method of search results, and by enabling or disabling a variety of personalization features on the hotel search engine website, we are able to empirically identify the causal impact of search engines on consumers' online click and purchase behavior The archival data analysis and the randomized experiments are consistent in demonstrating that ranking has a significant effect on consumer click and purchase behavior. We find that hotels with a higher reputation for providing superior services are more adversely affected by an inferior screen position. In addition, a consumer utility-based ranking mechanism yields the highest click and purchase propensities in comparison to existing benchmark systems such as ranking based on price or customer ratings. Our randomized experiments on the impact of active vS. passive personalization mechanisms on user behavior indicate that although active personalization(wherein users can interact with the recommendation algorithm) can lead to a higher click-through rate compared to passive personalization, it leads to a lower conversion rate when consumers have a planned purchase beforehand. This finding suggests that active personalization strategies should not be adopted ubiquitously by product search engines. On a broader note, our inter-disciplinary approach provides a methodological framework for how econometric modeling, randomized field experiments, and IT-based artifacts can be integrated in the same study towards deriving causal elationships between variables of interest
1 Examining the Impact of Search Engine Ranking and Personalization on Consumer Behavior: Combining Bayesian Modeling with Randomized Field Experiments Anindya Ghose 1,2, Panagiotis G. Ipeirotis 2 , Beibei Li 2 1 Department of Operation and Information Management Wharton School of Business, University of Pennsylvania 2 Department of Information Systems Stern School of Business, New York University Abstract In this paper, we examine how different ranking and personalization mechanisms on product search engines influence consumer online search and purchase behavior. To investigate these effects, we combine archival data analysis with randomized field experiments. Our archival data analysis is based on a unique dataset containing approximately 1 million online sessions from Travelocity over a 3-month period. Using a hierarchical Bayesian model, we first jointly estimate the relationship among consumer click and purchase behavior, and search engine ranking decisions. To evaluate the causal effect of search engine interface on user behavior, we conduct randomized field experiments. The field experiments are based on a real-world hotel search engine application designed and built by us. By manipulating the default ranking method of search results, and by enabling or disabling a variety of personalization features on the hotel search engine website, we are able to empirically identify the causal impact of search engines on consumers’ online click and purchase behavior. The archival data analysis and the randomized experiments are consistent in demonstrating that ranking has a significant effect on consumer click and purchase behavior. We find that hotels with a higher reputation for providing superior services are more adversely affected by an inferior screen position. In addition, a consumer utility-based ranking mechanism yields the highest click and purchase propensities in comparison to existing benchmark systems such as ranking based on price or customer ratings. Our randomized experiments on the impact of active vs. passive personalization mechanisms on user behavior indicate that although active personalization (wherein users can interact with the recommendation algorithm) can lead to a higher click-through rate compared to passive personalization, it leads to a lower conversion rate when consumers have a planned purchase beforehand. This finding suggests that active personalization strategies should not be adopted ubiquitously by product search engines. On a broader note, our inter-disciplinary approach provides a methodological framework for how econometric modeling, randomized field experiments, and IT-based artifacts can be integrated in the same study towards deriving causal relationships between variables of interest
1. Introduction Many businesses today have started looking at consumers'online search queries and click log data, to understand how consumers seek and evaluate relevant information during their online shopping forays. In fact, the knowledge created from customer interactions with product search engines allows firms to customize their business and services in an interactive way to gain and retain customers(Henzinger 2007, Gretzel et al. 2006). Consequently, product search engines have evolved into one of the most important strategic platforms for information seeking and marketing communications. Moreover, because of the information overload reinforced by the recent explosion of social media(e.g, online word-of-mouth, social communities, geo-/social-tagging, photo/video sharing and blogs), product search engines perhaps provide the best way for consumer to seek information and act upon it Outside of search, one of the most important ways for shoppers to discover products has been through recommendation engines( Chittor 2010). Personalization and recommendation engines have been around for a while and have been a strong driver of sales. For example, Amazons recommendation system was said to account for up to 35 percent of sales in 2006. However, while individual online retailers have increased their usage of recommendation systems, product search engines have still not made any headway into providing personalized results in response to consumer queries for products Over the last few years, a tremendous amount of research has focused on how to improve the content quality of the search results, for example, by optimizing retrieval of relevant documents from the Web, mainly as a response to a keyword query(e.g, Lavrenko and Croft 2001, Pang and Lee 2008). Nevertheless, due to the multi-dimensional preferences of consumers for many products and services, several questions remain unanswered in this space. How can product search engines present their results in a manner that facilitates efficient information exchange and effective marketing activities? Should product search engines allow consumers to interact with the recommendation algorithm to personalize their search results? Therefore, two challenges appear to
2 1. Introduction Many businesses today have started looking at consumers’ online search queries and click log data, to understand how consumers seek and evaluate relevant information during their online shopping forays. In fact, the knowledge created from customer interactions with product search engines allows firms to customize their business and services in an interactive way to gain and retain customers (Henzinger 2007, Gretzel et al. 2006). Consequently, product search engines have evolved into one of the most important strategic platforms for information seeking and marketingcommunications. Moreover, because of the information overload reinforced by the recent explosion of social media (e.g., online word-of-mouth, social communities, geo-/social-tagging, photo/video sharing and blogs), product search engines perhaps provide the best way for consumer to seek information and act upon it. Outside of search, one of the most important ways for shoppers to discover products has been through recommendation engines (Chittor 2010). Personalization and recommendation engines have been around for a while and have been a strong driver of sales. For example, Amazon's recommendation system was said to account for up to 35 percent of sales in 2006. However, while individual online retailers have increased their usage of recommendation systems, product search engines have still not made any headway into providing personalized results in response to consumer queries for products. Over the last few years, a tremendous amount of research has focused on how to improve the content quality of the search results, for example, by optimizing retrieval of relevant documents from the Web, mainly as a response to a keyword query (e.g., Lavrenko and Croft 2001, Pang and Lee 2008). Nevertheless, due to the multi-dimensional preferences of consumers for many products and services, several questions remain unanswered in this space. How can product search engines present their results in a manner that facilitates efficient information exchange and effective marketing activities? Should product search engines allow consumers to interact with the recommendation algorithm to personalize their search results? Therefore, two challenges appear to
be crucial for product search engines today. First, what ranking mechanism should be used to effectively present the search results? Second, what personalization mechanism should be applied to deliver the search results to the population of heterogeneous consumers? These are the goals of our research. More specifically, first, we aim to examine how differences in search engine ranking mechanisms affect consumer search and purchase behavior online. Second, we examine how different levels of personalization affect consumer behavior and search engine performance. In articular, we compare between two types of personalization mechanisms: active personalization and passive personalization In our context, a ranking system that personalizes results based on the average utility from a given hotel and enables consumers to proactively interact with the recommendation algorithm prior to the display of results from a search query are classified as active". In contrast, a ranking system that personalizes results based on the average utility from a given hotel, but does not allow customers to interact with the recommendation algorithm prior to displaying results is classified as passive Towards examining these questions, we combine Bayesian modeling on archival data analysis with randomized field experiments. Our research focuses on the hotel industry. We apply archival data analysis to gain insights towards our first research objective of studying the impact of ranking mechanisms on consumer click and purchase behavior. Using a panel data set from 2008/11 to 2009/1, containing approximately 1 million online user search sessions including detailed information on consumer searches, clicks, and transactions, obtained from Travelocity, we propose a hierarchical Bayesian framework in which we build a simultaneous equation model to jointl examine the inter-relationship between consumer click and purchase behavior, and search engine ranking decisions As of today, no hotel search engine, has explicitly, adopted a personalization-based approach to hotel ranking because they are still grappling with the issue of whether this is useful or not. Hence there is no known archival data in any product search engine that has information on the effect of personalization on user behavior. Therefore, we design and conduct randomized field experiments
3 be crucial for product search engines today. First, what ranking mechanism should be used to effectively present the search results? Second, what personalization mechanism should be applied to deliver the search results to the population of heterogeneous consumers? These are the goals of our research. More specifically, first, we aim to examine how differences in search engine ranking mechanisms affect consumer search and purchase behavior online. Second, we examine how different levels of personalization affect consumer behavior and search engine performance. In particular, we compare between two types of personalization mechanisms: active personalization and passive personalization. In our context, a ranking system that personalizes results based on the average utility from a given hotel and enables consumers to proactively interact with the recommendation algorithm prior to the display of results from a search query are classified as “active”. In contrast, a ranking system that personalizes results based on the average utility from a given hotel, but does not allow customers to interact with the recommendation algorithm prior to displaying results is classified as passive. Towards examining these questions, we combine Bayesian modeling on archival data analysis with randomized field experiments. Our research focuses on the hotel industry. We apply archival data analysis to gain insights towards our first research objective of studying the impact of ranking mechanisms on consumer click and purchase behavior. Using a panel data set from 2008/11 to 2009/1, containing approximately 1 million online user search sessions including detailed information on consumer searches, clicks, and transactions, obtained from Travelocity, we propose a hierarchical Bayesian framework in which we build a simultaneous equation model to jointly examine the inter-relationship between consumer click and purchase behavior, and search engine ranking decisions. As of today, no hotel search engine, has explicitly, adopted a personalization-based approach to hotel ranking because they are still grappling with the issue of whether this is useful or not. Hence, there is no known archival data in any product search engine that has information on the effect of personalization on user behavior. Therefore, we design and conduct randomized field experiments
based on a unique hotel search engine application designed and built by us. This also helps us make causal claims about the relationship between the search-based personalization strategies and consumers' purchase behavior In a randomized experiment, a study sample is divided into one group that will receive the intervention being studied(the treatment group) and another group that will not receive the intervention (the control group). Randomized experiments have major advantages over observational studies in making causal inferences Randomization of subjects to different treatment conditions ensures that the treatment groups, on average, are identical with respect to all possible characteristics of the subjects regardless of whether those characteristics can be measured or not. In our first experiment, we have designed four treatment groups. Each group is exposed to the same search ranking mechanism except for a different default ranking method. In the second experiment we have two treatment groups and one control group The control group is granted full access to the search mechanism with active personalization that allows them to interact with the search engine recommendation algorithm. In contrast, for the treatment groups, the two key personalization features are disabled for each group(which we refer to as passive personalization) Our randomized experimental results are based on a total of 730 unique user responses over two- week period via Amazon Mechanical Turk(AMT) crowd-sourcing platform. We use a customized behavior tracking system to observe the detailed information of consumer search, evaluation and purchase decision making process. The use of randomized experimental design should allow a degree of certainty that the research findings cited in studies that employ this methodology reflect the effects of the interventions being measured and not some other underlying variable or variables Hence, we need to be careful in designing these experiments. by manipulating the default ranking method, and by enabling or disabling a variety of personalization features on the hotel search engine website, we are able to extract the causal effect of search engine ranking and personalization on consumer behavior In some cases, rather than comparing with the control group, multiple treatment groups can be compared with each other(Ranjith 2005). This is the method we use in our first experimental study
4 based on a unique hotel search engine application designed and built by us. This also helps us make causal claims about the relationship between the search-based personalization strategies and consumers’ purchase behavior. In a randomized experiment, a study sample is divided into one group that will receive the intervention being studied (the treatment group) and another group that will not receive the intervention (the control group) 1 . Randomized experiments have major advantages over observational studies in making causal inferences. Randomization of subjects to different treatment conditions ensures that the treatment groups, on average, are identical with respect to all possible characteristics of the subjects, regardless of whether those characteristics can be measured or not. In our first experiment, we have designed four treatment groups. Each group is exposed to the same search ranking mechanism except for a different default ranking method. In the second experiment, we have two treatment groups and one control group. The control group is granted full access to the search mechanism with active personalization that allows them to interact with the search engine recommendation algorithm. In contrast, for the treatment groups, the two key personalization features are disabled for each group (which we refer to as passive personalization). Our randomized experimental results are based on a total of 730 unique user responses over twoweek period via Amazon Mechanical Turk (AMT) crowd-sourcing platform. We use a customized behavior tracking system to observe the detailed information of consumer search, evaluation and purchase decision making process. The use of randomized experimental design should allow a degree of certainty that the research findings cited in studies that employ this methodology reflect the effects of the interventions being measured and not some other underlying variable or variables. Hence, we need to be careful in designing these experiments. By manipulating the default ranking method, and by enabling or disabling a variety of personalization features on the hotel search engine website, we are able to extract the causal effect of search engine ranking and personalization on consumer behavior. 1 In some cases, rather than comparing with the control group, multiple treatment groups can be compared with each other (Ranjith 2005). This is the method we use in our first experimental study
Our main findings are the following. First, we find a significant ranking effect on both click throughs and conversions. A hotel that appears on a higher position on the screen and on an earlier webpage attracts a more clicks and conversions from consumers. On average, a one position increase on the screen is associated with a 731% increase in hotel click-throughs and a 4.56% increase in conversions. Moreover, we find that hotels with a higher reputation for providing superior services are more adversely affected by an inferior screen position(i.e, being ranked on the bottom part of the screen) than others Second, we find that the total number of hotels in a certain market has a negative effect on hotel click-throughs and conversions. This suggests that the more hotels available for a consumer to choose from, the less likely the consumer will choose any of them. a plausible explanation is elated to theories of consumer cognitive cost. prior theoretical work has shown that information overload and non-negligible search cost can discourage decision makers of searching, and end up with not searching or not choosing(Kuksov and Villas-Boas 2010). Our empirical finding nicely dovetails with the theoretical conclusion by Kuksov and villas-Boas in that more alternatives can lead to fewer choic Third, our experimental results on ranking mechanism are highly consistent with those from the Bayesian model based archival data analysis, suggesting a significant and causal effect of search engine ranking on consumer click and purchase behavior. Specifically, a consumer utility-based ranking mechanism yields the highest click and purchase propensities in comparison to existing benchmark systems such as ranking based on price or customer ratings Finally, we find active personalization mechanism that requires consumer interactions to specify both search context and individual preference can attract higher online attention from consumers and leads to higher click-through rate for search engine, compared to the two passive mechanisms where the two personalization choices are disabled one at a time. Surprisingly, search engine with active personalization mechanism performs the worst in the conversion rate. This finding suggests although active personal ization helps consumers discover what they want to buy hence increasing
5 Our main findings are the following. First, we find a significant ranking effect on both clickthroughs and conversions. A hotel that appears on a higher position on the screen and on an earlier webpage attracts a more clicks and conversions from consumers. On average, a one position increase on the screen is associated with a 7.31% increase in hotel click-throughs and a 4.56% increase in conversions. Moreover, we find that hotels with a higher reputation for providing superior services are more adversely affected by an inferior screen position (i.e., being ranked on the bottom part of the screen) than others. Second, we find that the total number of hotels in a certain market has a negative effect on hotel click-throughs and conversions. This suggests that the more hotels available for a consumer to choose from, the less likely the consumer will choose any of them. A plausible explanation is related to theories of consumer cognitive cost. Prior theoretical work has shown that information overload and non-negligible search cost can discourage decision makers of searching, and end up with not searching or not choosing (Kuksov and Villas-Boas 2010). Our empirical finding nicely dovetails with the theoretical conclusion by Kuksov and Villas-Boas in that “more alternatives can lead to fewer choices.” Third, our experimental results on ranking mechanism are highly consistent with those from the Bayesian model based archival data analysis, suggesting a significant and causal effect of search engine ranking on consumer click and purchase behavior. Specifically, a consumer utility-based ranking mechanism yields the highest click and purchase propensities in comparison to existing benchmark systems such as ranking based on price or customer ratings. Finally, we find active personalization mechanism that requires consumer interactions to specify both search context and individual preference can attract higher online attention from consumers and leads to higher click-through rate for search engine, compared to the two passive mechanisms where the two personalization choices are disabled one at a time. Surprisingly, search engine with active personalization mechanism performs the worst in the conversion rate. This finding suggests although active personalization helps consumers discover what they want to buy hence increasing
the sale, it should not be adopted ubiquitously. When consumers already have a planned purchase in mind(as in our setting), active personalization may actually cause the conversion rate to drop 2. Literature review Our research is related to the fields of search engine ranking and online position effect. Over the past few years, two opposite views have been held towards the position effect in product search. On one hand, consumers are endowed with cognitive limitation. Eye-tracking studies have long shown chat people tend to scan the search results in order(e. g, Aula and Rodden 2009). Hence, the same link will have a higher click-through rate(CTR) if it is positioned towards the top of the page versus the bottom (e. g, Srikant et al 2010). Studies have also found empirical evidence suggesting significant effect of rank order in the context of search engine-based keyword advertising (e.g Ghose and Yang 2009, Rutz and bucklin 2007) However, very little empirical work actually examines the rank order effect on product demand in searching and purchasing commercial products. A few existing studies such as Baye et al. (2009) examine the ranking effect on click-through rate as a substitute for the actual demand (conversions) Other studies tend to focus only on a single search dimension, for example, examining the competition of retailers ranked on price search engines(e.g, Ellison and Ellison 2009) In contrast to the above theoretical work, consumers have been found to be"variety-seeking"in their economic choice making process(e. g, McAlister 1982, Givon 1984 ). Especially, recent studies have shown that when consumers search and shop for commercial products online, they tend to examine the variety reflected in the set of product search results as a whole for their choice decision(Agrawal et al 2009, Panigrahi and Gollapudi 2011). This is different from the traditional web search (i.e, which returns web pages) where people often examine the results in a top-down order. As a consequence, the rank order of the product search results(i.e, which contain normally commercial products) may not have significant effects as in the web page search context (Bhattacharya et al 2011), whereas only the diversity of products in the search results set matters Therefore, one of our major goals in this research is to examine whether there exists a significant
6 the sale, it should not be adopted ubiquitously. When consumers already have a planned purchase in mind (as in our setting), active personalization may actually cause the conversion rate to drop. 2. Literature Review Our research is related to the fields of search engine ranking and online position effect. Over the past few years, two opposite views have been held towards the position effect in product search. On one hand, consumers are endowed with cognitive limitation. Eye-tracking studies have long shown that people tend to scan the search results in order (e.g., Aula and Rodden 2009). Hence, the same link will have a higher click-through rate (CTR) if it is positioned towards the top of the page versus the bottom (e.g., Srikant et al 2010). Studies have also found empirical evidence suggesting significant effect of rank order in the context of search engine-based keyword advertising (e.g., Ghose and Yang 2009, Rutz and Bucklin 2007). However, very little empirical work actually examines the rank order effect on product demand in searching and purchasing commercial products. A few existing studies such as Baye et al. (2009) examine the ranking effect on click-through rate as a substitute for the actual demand (conversions). Other studies tend to focus only on a single search dimension, for example, examining the competition of retailers ranked on price search engines (e.g., Ellison and Ellison 2009). In contrast to the above theoretical work, consumers have been found to be “variety-seeking” in their economic choice making process (e.g., McAlister 1982, Givon 1984). Especially, recent studies have shown that when consumers search and shop for commercial products online, they tend to examine the variety reflected in the set of product search results as a whole for their choice decision (Agrawal et al 2009, Panigrahi and Gollapudi 2011). This is different from the traditional web search (i.e., which returns web pages) where people often examine the results in a top-down order. As a consequence, the rank order of the product search results (i.e., which contain normally commercial products) may not have significant effects as in the web page search context (Bhattacharya et al 2011), whereas only the diversity of products in the search results set matters. Therefore, one of our major goals in this research is to examine whether there exists a significant
anking effect in product search. By combining archival data analysis with a set of randomized experiments, our research can thus provide critical insights on the impact of search engine ranking and design on users' search and purchase behavior from a causal perspective Existing research also holds two different opinions toward the effects of personalization supportive and skeptical(Arora et al. 2008 ). From the supportive perspective, Malthouse and Elsner(2006)show in a field test that personalizing the copy used in a book offer increases response rates significantly. Rossi et al. (1996) quantified the benefits of adopting one-to-one pricing by utilizing household purchase history data and empirically found that individual personalization improves 7.6% over mass optimization. Ansari and Mela(2003)found that targeting the content can potentially increase the expected number of click through by 62%.Arora and Henderson(2007)showed targeting at individual level can enhance the efficiency of embedded premium. From the skeptical perspective, Zhang and Wedel(2009)investigate the profit potential of various promotion programs customized at different levels in online and offline stores. They found that the incremental benefits of one-to-one promotions over segment- and market-level customized promotions were small in general, especially in offline stores Furthermore, one major concern in one-to-one marketing is invasion of privacy( Chellappa and Sin 2005, Arora et al. 2008 ) Chellappa and Sin(2005)developed a parsimonious model to predict consumers' usage of online personalization as a result of the tradeoff between their value for personalization and concern for privacy. They found that a consumer's intent to use personalization services is positively influenced by her trust in the vendor. A recent experimental study by aral and Walker(201 1)looked at application adoptions among 1. 4 million friends of over 9,000 users on Facebook. com, and found that active-personalized invitations are less effective in generating peer influence and social contagion compared to passive-broadcast notifications In summary, existing studies indicate although personalization can lead to customer satisfaction and profits, it may not work universally. Moreover, the level of personalization design is sensitive to the context and consumer behavior. Therefore. another goal of our research is to examine consumer online search
7 ranking effect in product search. By combining archival data analysis with a set of randomized experiments, our research can thus provide critical insights on the impact of search engine ranking and design on users’ search and purchase behavior from a causal perspective. Existing research also holds two different opinions toward the effects of personalization: supportive and skeptical (Arora et al. 2008). From the supportive perspective, Malthouse and Elsner (2006) show in a field test that personalizing the copy used in a book offer increases response rates significantly. Rossi et al. (1996) quantified the benefits of adopting one-to-one pricing by utilizing household purchase history data and empirically found that individual personalization improves 7.6% over mass optimization. Ansari and Mela (2003) found that targeting the content can potentially increase the expected number of click through by 62%. Arora and Henderson (2007) showed targeting at individual level can enhance the efficiency of embedded premium. From the skeptical perspective, Zhang and Wedel (2009) investigate the profit potential of various promotion programs customized at different levels in online and offline stores. They found that the incremental benefits of one-to-one promotions over segment- and market-level customized promotions were small in general, especially in offline stores. Furthermore, one major concern in one-to-one marketing is invasion of privacy (Chellappa and Sin 2005, Arora et al. 2008). Chellappa and Sin (2005) developed a parsimonious model to predict consumers’ usage of online personalization as a result of the tradeoff between their value for personalization and concern for privacy. They found that a consumer’s intent to use personalization services is positively influenced by her trust in the vendor. A recent experimental study by Aral and Walker (2011) looked at application adoptions among 1.4 million friends of over 9,000 users on Facebook.com, and found that active-personalized invitations are less effective in generating peer influence and social contagion compared to passive-broadcast notifications. In summary, existing studies indicate although personalization can lead to customer satisfaction and profits, it may not work universally. Moreover, the level of personalization design is sensitive to the context and consumer behavior. Therefore, another goal of our research is to examine consumer online search
and purchase behavior under different levels of personalization mechanisms on product search engines 3. Data Our dataset consists of detailed information on a total of 969.033 online sessions from Travelocity. com, including consumer searches, clicks and conversions that occurred within these sessions over 3 months from 2008/11 to 2009/1. Besides, we supplement our search and transaction data with hotel service-, location- and customer review-based information collected using various machine learning techniques such as image classification and text mining tools. This provides us a final dataset with a total of 29, 222 weekly observations for 2117 hotels in the US. More specifically, our dataset combines four major sources 3.1.ConsumerSearchClickandConversionDatafromTravelocity.com We have complete information on consumer searching and shopping behavior. a typical online session involves the initialization of the session, the search query, the results (in a particular rank order) returned from that search query, the sorting method, the click(s)on hotel(s)if there exists any, the login and actual transaction(s) if any conversion occurs, and the termination of the session We count a display for a hotel if that hotel appears visible to a consumer on the web page in online search session. Meanwhile, a"click? "is counted if the hotel is selected by a consumer, and a conversion"is counted only if a consumer has finished the payment in that online session. Since our major goal is to exam the effect of rank order displayed on a page, we focus only on the sessions with at least one display a display often leads to a click but it may not lead to an actual purchase. Each hotel that counts for a display is associated with a page number and a screen position, which capture the corresponding page order and( within-page) rank order of that hotel in he search results. Notice that when Travelocity displays the hotel search results on a web page, it 2 In some cases, users may initiate a session and look for general travel information, for example the area of the city rather than search for any hotels, thus there will be no hotels displayed on any web page. We excluded such sessions in our analysis
8 and purchase behavior under different levels of personalization mechanisms on product search engines. 3. Data Our dataset consists of detailed information on a total of 969,033 online sessions from Travelocity.com, including consumer searches, clicks and conversions that occurred within these sessions over 3 months from 2008/11 to 2009/1. Besides, we supplement our search and transaction data with hotel service-, location- and customer review-based information collected using various machine learning techniques such as image classification and text mining tools. This provides us a final dataset with a total of 29,222 weekly observations for 2117 hotels in the US. More specifically, our dataset combines four major sources: 3.1. Consumer Search, Click and Conversion Data from Travelocity.com We have complete information on consumer searching and shopping behavior. A typical online session involves the initialization of the session, the search query, the results (in a particular rank order) returned from that search query, the sorting method, the click(s) on hotel(s) if there exists any, the login and actual transaction(s) if any conversion occurs, and the termination of the session. We count a “display” for a hotel if that hotel appears visible to a consumer on the web page in an online search session. Meanwhile, a “click” is counted if the hotel is selected by a consumer, and a “conversion” is counted only if a consumer has finished the payment in that online session. Since our major goal is to exam the effect of rank order displayed on a page, we focus only on the sessions with at least one display2 . A display often leads to a click, but it may not lead to an actual purchase. Each hotel that counts for a display is associated with a page number and a screen position, which capture the corresponding page order and (within-page) rank order of that hotel in the search results. Notice that when Travelocity displays the hotel search results on a web page, it 2 In some cases, users may initiate a session and look for general travel information, for example the area of the city, rather than search for any hotels, thus there will be no hotels displayed on any web page. We excluded such sessions in our analysis
only shows 25 hotels per page. This restricts the rank order for each hotel within the range from 1 to 25. Meanwhile, to facilitate consumer search, Travelocity provides a sorting criterion called Travelocity Pick" by default. Besides, it also provides multiple alternative sorting criteria: Price, Hotel Class, Hotel Name, and Customer Review Rating. To capture consumers' particular sorting preferences that may potentially influence the position effect, we include a control variable in our study to indicate how frequently a hotel appears in a result list under a"special sort In addition, we also have supplemental data collected from three other sources. We only briefly discuss them below 3. 2. Hotel characteristics Location Characteristics: We used geo-mapping search tools(Bing Maps APD) and social geo- tags (from geonames. org) to identify the external amenities (e.g, shops, bars) and public transportation in the area around the hotel. We also used image classification together with Mechanical Turk to examine whether there is a nearby beach, a nearby lake, a downtown area, and whether the hotel is close to a highway. We extracted these characteristics within an area of 0.25- mile. 0.5 mile. 1-mile and 2-mile radius Service Characteristics: This category contains hotel class, number of internal amenities and number of rooms. Hotel class is an internationally accepted standard ranging from 1-5 stars, representing low to high hotel grades. Number of internal amenities is the aggregation of hotel internal amenities, such as bed quality, hotel staff, food quality, bathroom amenities and parking facility. We extracted this information from the Tripadvisor website using fully automated parsing Since hotel amenities are not directly listed on the Tripadvisor website, we retrieved them by following the link provided on the hotel web page, which randomly directs the user to one of its cooperating partner websites(e.g, Travelocity, Orbitz) Review Characteristics: We collected customer reviews from Travelocity. com. The online reviews and reviewers information were collected on a daily basis up to January 31, 2009(the last Recently Travelocity has upgraded the webpage design by showing 10 hotels per page. However, during our examination time period this number was 25
9 only shows 25 hotels per page3 . This restricts the rank order for each hotel within the range from 1 to 25. Meanwhile, to facilitate consumer search, Travelocity provides a sorting criterion called “Travelocity Pick” by default. Besides, it also provides multiple alternative sorting criteria: Price, Hotel Class, Hotel Name, and Customer Review Rating. To capture consumers’ particular sorting preferences that may potentially influence the position effect, we include a control variable in our study to indicate how frequently a hotel appears in a result list under a “special sort.” In addition, we also have supplemental data collected from three other sources. We only briefly discuss them below. 3.2. Hotel Characteristics Location Characteristics: We used geo-mapping search tools (Bing Maps API) and social geotags (from geonames.org) to identify the external amenities (e.g., shops, bars) and public transportation in the area around the hotel. We also used image classification together with Mechanical Turk to examine whether there is a nearby beach, a nearby lake, a downtown area, and whether the hotel is close to a highway. We extracted these characteristics within an area of 0.25- mile, 0.5 mile, 1-mile, and 2-mile radius. Service Characteristics: This category contains hotel class, number of internal amenities and number of rooms. Hotel class is an internationally accepted standard ranging from 1-5 stars, representing low to high hotel grades. Number of internal amenities is the aggregation of hotel internal amenities, such as bed quality, hotel staff, food quality, bathroom amenities and parking facility. We extracted this information from the Tripadvisor website using fully automated parsing. Since hotel amenities are not directly listed on the Tripadvisor website, we retrieved them by following the link provided on the hotel web page, which randomly directs the user to one of its cooperating partner websites (e.g., Travelocity, Orbitz). Review Characteristics: We collected customer reviews from Travelocity.com. The online reviews and reviewers’ information were collected on a daily basis up to January 31, 2009 (the last 3 Recently Travelocity has upgraded the webpage design by showing 10 hotels per page. However, during our examination time period, this number was 25
date of transactions in our database). In addition to the total number of reviews and the numeric reviewer rating, we extracted indicators that measure the stylistic characteristics of the reviews for robustness checks. We examined two text-style features: subjectivity and readability of reviews Ghose and Ipeirotis 2011). Also, since prior research suggested that disclosure of identity information is associated with changes in subsequent online product sales( Forman et al 2008 ),we measured the percentage of reviewers for each hotel who reveal their name or location information on their profile pages Table 1. Definitions and Summary Statistics of variables Variable Definition Mean std dev Min max Search Click and Conversion Data PRICE Transaction price per room per night 12045 732525.77 978 DISPLAY Number of displays 213.6538228 14849 CLICK Number of clicks 3.55 CONVERSION Number of conversions 1.26 0.66 PAGE Page number of the hotel 20.86 13.4 0011 192 RANK Screen position of the hotel within a page 1209 4.32 25 Hotel location-Related characteristics BEACH Beachfront within 0.6 miles 18 LAKE Lake or river within 0.6 miles 2 TRANS Public transportation within 0.6 miles HIGHWAY Highway exits within 0.6 miles 74 DOWNTOWN Downtown area within 0.6 miles 67 000000 EXTAMENITY Number of external amenities within 1 mile 4.57 792 27 i.e., restaurants, shopping malls, or bars CRIME City annual crime rate 193.19126.70 13 Hotel service-Related characteristics CLASS Hotel class 3.36 1.37 AMENiTyCNT Total number of hotel amenities 1154 7.56 12 23 ROOMS Total number of hotel rooms 212.30250.70 122900 Hotel review-Related characteristics REVTEWCNT Total number of reviews 21.06 29.28 202 RATING Overall reviewer rating SPECialSORt Number of times using a sorting method 37726 Total number of hotels in a city 24.03 BRAND Dummies for 9 hotel brands: Accor. Be western, Cendant, Choice, Hilton, Hyatt ntercontinental. Marriott and Starwood Number of observations(Weekly-Level): 29, 222 Time period:I1/2008-1/31/2009
10 date of transactions in our database). In addition to the total number of reviews and the numeric reviewer rating, we extracted indicators that measure the stylistic characteristics of the reviews for robustness checks. We examined two text-style features: subjectivity and readability of reviews (Ghose and Ipeirotis 2011). Also, since prior research suggested that disclosure of identity information is associated with changes in subsequent online product sales (Forman et al 2008), we measured the percentage of reviewers for each hotel who reveal their name or location information on their profile pages. Table 1. Definitions and Summary Statistics of Variables Variable Definition Mean Std. Dev. Min Max Search, Click and Conversion Data PRICE Transaction price per room per night 120.45 73.25 25.77 978 DISPLAY Number of displays 213.65 382.28 1 4849 CLICK Number of clicks 2.99 3.55 0 56 CONVERSION Number of conversions 1.26 0.66 0 9 PAGE Page number of the hotel 20.86 13.44 1 192 RANK Screen position of the hotel within a page 12.09 4.32 1 25 Hotel Location-Related Characteristics BEACH Beachfront within 0.6 miles .18 .38 0 1 LAKE Lake or river within 0.6 miles .22 .42 0 1 TRANS Public transportation within 0.6 miles .30 .46 0 1 HIGHWAY Highway exits within 0.6 miles .74 .44 0 1 DOWNTOWN Downtown area within 0.6 miles .67 .47 0 1 EXTAMENITY Number of external amenities within 1 mile, i.e., restaurants, shopping malls, or bars 4.57 7.92 0 27 CRIME City annual crime rate 193.19 126.70 3 1310 Hotel Service-Related Characteristics CLASS Hotel class 3.36 1.37 1 5 AMENITYCNT Total number of hotel amenities 11.54 7.56 2 23 ROOMS Total number of hotel rooms 212.30 250.70 12 2900 Hotel Review-Related Characteristics REVIEWCNT Total number of reviews 21.06 29.28 1 202 RATING Overall reviewer rating 3.84 .85 1 5 Control Variables SPECIALSORT Number of times using a sorting method 204.64 377.26 0 4810 H Total number of hotels in a city 24.03 56.48 1 922 BRAND Dummies for 9 hotel brands: Accor, Best western, Cendant, Choice, Hilton, Hyatt, Intercontinental, Marriott, and Starwood -- -- 0 1 Number of Observations (Weekly-Level): 29,222 Time Period: 11/1/2008-1/31/2009