Problem

The Engagement team lacked visibility into users’ perception of the experience with certain features under the tribe’s scope. At the time, we had NPS and CSAT , but these metrics had a broader scope, assessing the product as a whole (and were not exactly UX metrics). 

Project Goals and Questions

Test and implement experience metrics on the platform to establish a baseline, identify potential gaps and inconsistencies, and monitor the user experience on an ongoing basis.

1. Which experience metrics are best suited for the tribe’s context?

2. At which stage of the experience will we collect this information?

3. What will be the frequency of analysis, and where will we document it?

Methods

Desk Research: We conducted an exploratory study on different types of experience metrics and selected SEQ, UMUX, and CES for further analysis. We compared the standard survey questions and calculation methods for each evaluation and, considering the tribe’s context, chose UMUX-Lite as our starting point.

Mapping touchpoints: This is where we encountered the main challenge of this initiative: it was not possible to deploy surveys at specific moments of the user experience (e.g., saving, clicking on recommendations, searching) due to the lack of integration between product trackings and research tools.

​​

Stakeholder Alignment: To enable the creation of experience metric surveys at specific product touchpoints, I reached out to the team for support. Partnering with a research colleague and a member of the engineering team, they developed a solution to integrate tracking with the research tool. We compiled a list of the most relevant tracking events for the team’s context and prioritized those that aligned with the current initiatives of each squad.

​

Online Survey: With the tracking integration in place, we implemented the UMUX survey within the product.

Research Impact

Impact on Research Operations 

We believe the value of this initiative extends beyond the implementation of metrics. With the tracking solution validated, we were able to raise the maturity level of research within the product by enabling the deployment of surveys at specific points in the user journey—something that was not possible prior to this project.

Today, other teams can easily implement metrics and surveys within the product, and we now have experience metrics surveys running continuously.

Considering the challenges we faced with tracking integration and low survey engagement, we were unable to fully complete the proposed metrics implementation. We conducted a preliminary analysis of the results and shared them with the team, but the project was discontinued due to a team reorganization and a shift in the tribe’s scope.

This is a summarized version—please feel free to reach out to me at rute.harada@gmail.com for more detailed insights and learnings.

My Learnings

It’s not always possible to complete projects as planned. Several factors can influence the success of an initiative, but this doesn’t mean that value isn’t being delivered. Challenges and mistakes also provide opportunities for learning and growth!

Collaboration is key! Each team member brings their own expertise and knowledge, and this is essential to achieve our goals and improve research practices.