Omada Health project

Omada Health was seeing a drop off in weekly lesson completion — an important KPI that we were paid on. I conducted research across several efforts to move this key metric.

• Alignment with Product and Coaching — lesson completion was a Product KPI, and the Coaching team had a lot of anecdotal experience with participants complaining about lesson topic sequence (e.g. wanting to focus on exercise sooner).

• Reviewing existing research — in past qualitative research, participants had shared feedback on lessons; recent archetype work based on key attitudes and approaches to both health changes and the Omada program that focused on areas of need and interest.

• SQL analytics & collaboration with Data Science to review their data — a review of lesson completion / drop off patterns; used this data for pre-survey segmentation on lesson completion and other engagement indicators (e.g. engaged with coach but not completing lessons; engaged in neither; engaged with lesson but not coach).

• Moderate-N (n=~60) qual-heavy survey — focused on general impressions of and feedback on lessons.

• Survey analysis & early hypothesis generation — we hypothesized that the archetypal patterns we saw in other recent research could be useful in adjusting lessons to improve completion (e.g. certain archetypes were more focused on emotional eating vs exercise).

• Approach: We decided to run an A/B test by segmenting the participants into groups based on the archetypes we’d seen and adjusting the lessons to address those archetypes; control group saw the original lessons.

• Quant-focused survey (n=~200) — combination of validated psychology instruments and customized questions and statements to gauge which archetype (if any) the participant best fit in — designed to

• Result: Null — no improvement with adjusted lessons.

• Low-N qual interviews — talking to participants who had been in the treatment arm and dropped off on lessons.

• Moderate-N survey

• Collaboration with Content for to create prototype lessons for the 4 key archetypes we had seen across general qual research

• Quant survey n=100 on archetypes — goal to develop potential questions to identify likely participant archetype (mix of validated clinical instruments and custom questions)

• Low-N follow-up interviews recruited from survey takers —

• Hypothesis refinement

• A/B test design

• A/B test — supported by data science (tools only available to data scientists at Omada)

Everlane project

Everlane had launched two styles of shoes and was trying to figure out where to go next. Shoes were a huge investment to de-risk.

Goal: Make an educated recommendation for the next shoe style. Get beyond digital images to assess in-person qualities, e.g. material, that we knew could fuel returns.

• SQL analytics of purchases of the two existing styles

• Moderate-N (given Everlane’s size at that time!) qual-heavy survey (n=~50) of shoe purchasers — what they liked / didn’t, fit, material , etc of the current Everlane shoes

• Moderate-N qual-heavy survey (n=~50) of moderate-to-high spend customers — including detail on the last 10 shoes they purchased — type, style, price, color, photo of the shoe, etc.

• Survey analysis & early hypothesis generation — price point by type and style, current purchase trends, patterns of challenges in the current Everlane shoes

• In-person quant survey (n=~200 in NYC and SF) — I purchase ~30+ competitive shoes and separated them into small groupings. Each grouping focused on a different aspect of the shoe (e.g. leather look and feel, heel height, toe shape), with the clusters containing various options for each aspect. Respondents circulated among the sections, entering their preferences into the survey on their phone.

• Survey analysis — there were some surprisingly strong groupings

• Recommendation and socialization

Result: The results didn’t align with the design lead’s preferences, which led to some interesting conversations; however, the team later launched a shoe with many of the attributes recommended by the research, and it was a strong success.