People Experience Research & Usability Testing

Calfresh logo written in lowercase letters in green, and split between "cal" and "fresh" by four overlapping circles, each colored differently - orange, light brown and two reds - with curving green lines across them. Beneath the letter and circled desigh are the words, "BETTER FOOD FOR BETTER LIVING."
SAN FRANCISCO DEPARTMENT OF BENEFITS & FAMILY SUPPORT (BFS)

BFS provides medical, financial, employment and child protection for San Franciscans. CalFresh (the Food Stamps program) is one of these services. It provides money to pay for groceries. As an internal research and design resource in the San Francisco Human Service Agency (where BFS is located), I managed the research strategy for this project. I then transitioned from overseeing the work in the first round of research to being the lead individual contributor for the next two research sprints. This included leading all (1) research, (2) research operations and (3) deliverables.

Role: Manager & Lead Researcher

Year: 2021-2022

Challenge: The digital services team in my agency was upgrading our organization’s website from the Drupal 7 to 9 programming language. How could this opportunity also improve how the public experiences the site’s design and content? What if we started by researching people’s experiences with the highly trafficked CalFresh webpages, including non-native English speakers?

Project Scope: We worked with the product owner to align on the project’s (1) goals, (2) questions to be answered, (3) prototypes to be tested, (4) timeline, (5) deliverables, (6) success metrics and (7) prioritization of research insights.

Quick Wins: We successfully advocated for the engineering team to use the U.S. Web Design System (USWDS) as the basis for the upgrade. The quality design and accessibility-compliant components of USWDS were in Drupal. They could be repurposed for our website’s needs. We also used data successfully to champion testing (1) on mobile devices (that’s how most people use our site), and (2) with our Traditional Chinese-reading population (who are about one in five of our CalFresh clients).

Screenshot of the prototypes we tested for the Traditional Chinese, mobile version of the SFHSA.org homepage. "San Francisco Human Services Agency" is in the top left, written in black lettering. Next to it is its logo in blue lines forming a square with rounded edges. In the square is a circle and in that looks like a petal with the ends of four ovals touching in the middle of the circle, each pointing in a different direction - north, south, east and west - and four tiny circles between the ovals. In the upper right corner is "Menu" written in Traditional Chinese in white lettering on a blue rectangle. Beneath that is a row of languages people can click for their preferred language to read the website. From left to right, written in each language is "Chinese, Spanish, Russian, Vietnamese, Filipino, and English." Beneath that is SFHSA.org and its Traditional Chinese equivalent.
MOBILE VERSION IN TRADITIONAL CHINESE THAT WAS TESTED

Evolving Strategy: The original research lead left the project before the first round of data gathering was done. I moved from a managerial role to being the lead, individual contributor, which included two more rounds of remote research. Early changes I made involved (1) a more systematic approach to note-taking and analysis, (2) deeper collaboration with the project team to analyze and synthesize the data, and (3) prototypes to show the team based on the research insights.

"Analytic Review, Synthesis & Two New Mobile Designs" is written on the top of the screenshot of a MURAL application's canvas. Underneath that is: "Team analysis and synthesis session | Friday, May 5, 2022" Below that is: "Instructions: (1) Intro to MURAL. (2) Map overview (3) Start with analytical review (4) Q&A (5) Review insights informing two new designs. (6) Q&A (7) Next Steps" There are seven piles of sticky notes or other content below those headings. (1) A neat row of 8 x 6, pink sticky notes. (2) Usability Testing Observations & Insights (Interpretations). Instructions: Orientation to the table below by following the numbers 1-5. (3) Original Mobile Versions Tested. (4) Insights Informing Two New Designs (5) Original, Two, Desktop Versions Tested (6) Questions about the analysis or designs? Ideas to make them better? Instructions: Please drag a note to the place where you have a question or idea and then type it out. (7) Next Steps
REMOTE ANALYSIS AND SYNTHESIS SESSION WITH THE PROJECT TEAM

Equity Considerations: Research participants came from a variety of demographic backgrounds to provide a more holistic understanding of how people experience the website on mobile and non-mobile devices.

Outputs: (1) New prototypes, (2) a research report, and (3) refined research templates and tools.

Impact: Based on the insights, (1) translations errors have been fixed, (2) a translation service is being hired to manage this work for the five, non-English languages, (3) a mobile-first rather than mobile responsive website transition is happening for our product team, (4) a new approach to rapid testing has been created.

Zoomed in part of the screenshot image to the left featuring two mobile designs: Version A and B. They have the logo and wording, "San Francisco Human Services Agency" in the top left. In the top right is "Menu" written in white letters within a blue rectangle. Beneath that are five languages to translate the page: Chinese, Spanish, Russian, Vietnamese, Filipino and English." Then is "CalFresh" with one sentence explaining what it is. The images then differ. The one on the left has three anchor links at the top followed by clickable accordions or rectangles to open more content. In the mobile version to the right are the same three anchor links - Contact, Need Food Now, News - followed by three images with text that read, "Apply or Check Eligibility," "Get & Use Your EBT Card," and "Forms & Deadlines." There are purple sticky notes in black lettering near these two images of mobile prototypes. The stickies explain the design and content changes.
TWO NEW PROTOTYPES i designed WITH EXPLANATIONS FOR CHANGES