More than 20 million students are looking to improve their lives through learning, and depend on our ability to help them find the right course. With such a large collection to choose from, search is the easiest & preferred way to find what they’re looking for.
For 1.5 years, I lead the design of the Search experience.
My responsibilities included user research, concept ideation, aligning key stakeholders on product goals, designing user flows, visual design, prototyping, and collaborating with broader Marketplace Discovery teams.
I worked alongside a Product Manager and 5 engineers. Our extended team also included a researcher and data scientist.
Search lives within the Marketplace team; other teams include: Browse, Course Page, Reviews, Mobile, and Tagging.
When I first joined the Search team, it was an engineering only team working on ranking and algorithm problems.There was no product / user-facing strategy, and feature ideas were solution-based and difficult to evaluate.
I wanted us to align as a team on how we evaluate and ideate on new features. To do that, we needed research support to understand what our users needed.
Old Search Page
Mainly with a career-related goal in mind
Almost ~90% of searches were topics (i.e. photoshop, python)
1. Students see irrelevant courses for their goal and assumes Udemy has no courses that can help them
2. Students don’t have enough information to understand if a topic or course will help them achieve their goal
3. Students worry that there is a better course on the marketplace despite finding a good course (FOMO)
4. We have no results that match what they searched for
Mapping out user problems to the user flow
Based on the main problems, we formed 4 areas of work:
1. Bring students to relevant results
2. Help students understand their results
3. Bring focus to recommended courses
4. Reduce dead ends
Our old search system was so simplistic, it didn’t reflect the many states that a searcher could be in.
The new search page needed to support both existing features, and potential features we haven't designed yet.
Countless team brainstorms and prototypes later, we aligned on key features in each bucket of work:
1. Course-Based Filters, Related Searches
to bring students to relevant results
2. Course Quickview, Topic Overview
to help students understand their results
3. Recommendations, Badges
to bring focus to recommended courses
4. Include Lecture Results, No Results Page
to reduce dead ends
One of the main reasons that students weren’t seeing relevant courses is because they searched a broad topic that can be taught for different reasons. For example, students could search the topic “photoshop” for photo editing, graphic design, ux design, mobile design or marketing.
To surface more relevant results, we wanted to enable our students to specify the path they’re interested in.
While I was excited about a couple of directions, in practice, Course Topic Filters was the most promising because they were systematically collected, and had coverage on almost every course in the marketplace.
Our course descriptions don’t always help students understand if a course can help them achieve their goals. From survey responses, we knew students needed additional pieces of information to properly evaluate a course's helpfulness.
We ultimately decided to go with an option that was fair to all courses, felt lightweight, didn't require an extra step, and was easy to compare between multiple courses.
One common reason students reach “dead-ends” is because they searched for a niche topic that our search couldn’t find an entire course on. Those topics may be mentioned within a course, but we were only scanning course titles & subtitles to find results. We wanted to see if searching within courses would help reduce dead-ends.
From user-testing, we learned that full course matches were still more compelling. But in cases where there weren’t many full course matches, the lecture matches are also compelling. Thus, we enabled lecture matches only on queries that had less than 12 results.
With all these newly planned features, we also had to start thinking about a new layout that can support it all. The current layout was too simplistic and restricted how the features could work together. The trickiest part with changing the layout was balancing how prominent a feature should be with how often it’s used, how helpful it is to the user, and the conversion impact.
Armed with these insights, I iterated on the 2nd direction "Expose Dynamic Filters" and took designs to high fidelity. During this time we also went through a brand redesign.
Given the broader marketplace strategy for developing higher quality metadata and tags, a more flexible and dynamic filter system seemed like a better bet. During this time, we also went through a brand refresh that gave us a more robust colour palette to work with.
We rolled out the new layout in 3 parts to be able to track more accurately what was working and what wasn’t. After many iterations, these are the final positive RPU numbers on each element. Everything else was neutral.
(c) Beatrice Law 2018