eBay Haitao App Foundational Research
eBay Haitao App Foundational Research
01_intro
Project background
I was one of the first core members, including a UXR, a PM, a UX designer, and an iOS engineer, starting the eBay Haitao app project serving Chinese imported shoppers. I carried out a mixed-methods approach and implemented both generative and evaluative studies to help build, test, and launch the eBay Haitao app.
The eBay Haitao app was the New Product Development Team’s second initiative in China after launching the eBay Haitao website and achieving half-a-million-buyer goal in one year. For the business objectives, the app aimed to increase buyer engagement and drive repeat purchases. For the user experience purpose, Chinese consumers favor mobile apps over websites as they seek convenience in their dynamic lifestyles.
In addition to above business and user experience stand points, the product team also used the app to experiment novel ways to engage users. The efforts resulted in a gaming feature, micro-interactions and dynamic browse experiences throughout the app, and a reward program to encourage social sharing.
Methodologies
Design workshop: Ran 3 ideation and 1 synthesis workshops to create and decide 10 app concepts.
Moderated interview: Conducted 10 interviews in China to gain in-depth user feedback on the concepts.
MaxDiff survey: Co-managed survey execution (N=600) with an agency to evaluate design concepts.
RITE study: Led Rapid Iterative Testing and Evaluation study with the product team in China.
Moderated usability testing: Managed a Chinese agency to perform 3 rounds of in-lab usability testing.
02_Concept ideation
Concept generation matrix
The UX researchers and designers used Concept Generation Matrix to brainstorm ideas for the app. The goal of the app was to increase user engagement. We came up with the service criteria which would make Chinese consumers feel engaged on the top row, and a list of features that utilized eBay’s new technology on the left column. The concept generation matrix helped the team avoid random discussions and focused on the key objectives and eBay competence.
Brainstorm workshop Concept generation matrix
Top 10 concepts and 3 themes
The team clustered all of the ideas into themes and used “most user-centered“, “most provocative“, and “most viable“ to vote. After aggregating the scores, the top 10 concepts covered a wide variety of features, such as visual search, vote-for-deals, AI shopping assistant, and celebrity live streaming shopping.
03_concept evaluation
Moderated interviews in china
The research team led 3 product managers, 2 user experience designers, and 4 local marketing team members to conduct 10 moderated interviews in 3 days. Afterwards, we ran a 2-day analysis and synthesis workshop. The study not only informed the key stakeholders of the pros and cons of each app concept, but also helped the team build empathy for the Chinese consumers and improved the connection between the product team and the local business unit.
Interview session Analysis and synthesis workshop
online Survey with MaxDiff and kano exercises (N=600)
At eBay UX research team, we embrace data triangulation — incorporating both qualitative and quantitative studies to make informed recommendations. Besides the moderated interviews, we hired an agency to run an online survey (N=600) with a MaxDiff exercise and a KANO exercise.
MaxDiff Exercise
Respondents were asked to rate the feature that they desired the most and the least among three features in different scenarios.
There were 20 possible scenarios and each respondent was taken through 9 of the 20 scenarios with three features within the Max-Diff exercise.
The scenarios were randomized and balanced so an equal number of respondents saw each scenario.
KANO Exercise
Respondents were asked three rating scale questions: 1) How they would feel if the feature was available, 2) How they would feel if the feature was NOT available, and 3) Feature importance.
These positive/negative question pairs are analyzed together, and knowing the two points enables determining which emotional reaction the user is experiencing to a given feature.
Respondents were asked to rate the features on a 9 point importance scale which is used in helping to understand feature priority.
MaxDiff and KANO index
App concepts Evaluation
Combining the data from of both of the moderated interviews and the online survey, I used desirability, feasibility, and viability to evaluate the potential of the app concepts. In sum, I recommended the product team to focus on helping users finding the best and personalized deals and events as the direction to develop the eBay Haitao app in China. Both studies validated that users had strong demand for deals and it’s highly feasible by leveraging eBay’s core competence.
The Visual Search + AR concept using augmented reality was well received among younger generation, however, the search result accuracy was still a major challenge for our AI team. I suggested to keep this in our future roadmap while expecting the machine learning to continue improve itself. The live stream shopping concept gained poor scores from both studies due to authenticity concern, and it’s less likely for eBay to work with Chinese online celebrities given we don’t have much knowledge in this area.
04_REFINE CONCEPT
RAPID ITERATIVE TESTING & EVALUATION (RITE)
After confirming the app direction to be helping user findings the best and personalized deals, the research team led another round of brainstorming and had 6 new concepts associated with the theme. We decided to bring the core team members to China and conduct Rapid Iterative Testing & Evaluation (RITE) study.
RITE is a powerful yet low-risk way to test, validate, and improve on fluid design concepts. It is quicker than waterfall, allowing for broader range of ideas to be tested than evaluative research. RITE also engages and builds consensus of user needs among stakeholders.
Business goal
Increase Chinese imported consumers’s re-engagement and repeat purchase.
Research goal
1. Evaluate the app’s value propositions
Rewards - Attract by gamified coupon experiences
Authenticity - Avoid counterfeit products and bad sellers
Discovery - Seek recommendations from interest group and shopping guides
Social - Share great deal information with friend circles
2. Identify features, patterns, and interactions that work / don’t work with imported shoppers
Must solve the shopper problems
Must satisfy business goal
Create a starting point to build out the eBay Haitao app
RITE Process
I led a team of a PM, a designer and an engineer to conduct a 5-day RITE study in China. Each day we began with testing designs with 3 participants. Each member was assigned a role to take notes, observe user behaviors and emotions, or sketch quick ideas based on user feedback.
Then the team debriefed their observations and reexamined the hypotheses. We formed new hypotheses and created new designs for the next day. The whole process was repeated for 4 days and we kept the last day for final analysis and synthesis workshop.
Results
Acquired directional feedback of the concepts in a rapid pace.
Built rapport with the core product team from the beginning of a product life cycle.
Increased the stakeholder’s relatedness and empathy for the users they are building product for.
Reinforced team focus on the most critical questions giving a short timeline to sweep design iteration.
05_Test prototypes in the app development process
Moderated usability testing
Built on the user research results, the product team developed the app’s beta version in 10 weeks. Meanwhile, the research team managed a Chinese agency to launch three rounds of moderated usability testing.
One of the biggest challenges was to train a local moderator to run the interviews. I sent prescriptive interview script with questions, screens, and purpose of the question, and went through the script with the moderator step by step.
Using emoji to prompt attitudinal feedback
The research team noticed that Chinese participants were less expressive when asked to describe their perceptions. Therefore, we had experimented ways to help them talk more. At first we used keyboard emoji to let users rate their experience, but when asked to explain why they chose the score, they would squeeze a short sentence and then remained quiet.
Knowing that the keyboard emoji had not worked as well as expected, we tried WeChat emoji instead. Chinese people are used to add emoji in the end of a sentence to emphasize their emotion when chatting on WeChat. This turned out to be an effective way to prompt Chinese consumers’s attitudinal feedback.
06_Results
Gamify coupon experience to incentivize social Sharing
The app featured an gamified coupon experience by driving airplane into clouds to earn points. Before arriving in this feature, we have tested 6 different games to collect user feedback and realize earned rewards are valued more than free ones. Users were also incentivized to share with their friends.
Provide trust signals to ensure authenticity
The app featured value propositions, including 500 million global inventory, 23 years of e-commerce experience, and 30 days money back guarantee, on the landing page. On item page, sales amount, trusted seller badge, product rating, and direct shipping badge are highlighted to increase trustworthiness.
Introduced category browse to discover ebay’s breadth of inventory
Based on the team’s previous research, we learned that Chinese shoppers are used to browse products through category when they don’t have any specific target in mind. The app introduce a category browse experience where users can discover product through main categories and sub-category. There is also quick filters, including sales amount, ratings, and price, to help user refine the product results.
Streamline registration process by using phone number
The app implemented a registration process using phone number instead of email address. The result was a dramatic lift of the completion rate from 50% to 90%.
