Data Dashboard Experiences

Blue bar graph with five bars on a scale from 0 to 600 # Clients Helped Today. They include Mail & Fax 5, Lobby 89, Phones 157, Text / Chat 236 and App 522.

The Department of Benefits and Family Services in the San Francisco government provides access to health care and money to pay for food, among other services. The Data Warehouse Team for the department worked with managers to develop a data dashboard. The purpose was to help frontline teams better manage their daily tasks and track their performance over time. As an internal research and design member the department, I was the lead individual contributor for the project managing and doing (1) research strategy, (2) operations, (3) data gathering, (4) analysis, (5) the creation of deliverables (aided by a visual designer) and (6) presenting the work.

Role: Project Manager & Lead Researcher

Year: 2022

Challenge: Supervisors and managers asked their 400+ frontline teammates to use the dashboards. About 40% did not. The Data Warehouse Team wanted to why and what could be improved.

Project Scope: I worked with the team to align on the project’s (1) goals, (2) questions to be answered, (3) timeline, (4) deliverables, and (5) success metrics.

Why is [there two] “Refresh” [links] at the top of this page? ~Research Participant

Strategy: The Data Warehouse Team (DWT) were partners throughout the research. I interviewed two of them about their experiences then the three managers overseeing the 400+ frontline teams, a supervisor and and two frontline employees (one who uses it a lot and another who rarely uses it). I asked them to show me how they use it or what they use instead, taking notes and screenshots with their permission. People felt uncomfortable being interviewed with the DWT joining our video call, so I conducted the 30-minute interviews independently. The DWT asked for my notes, which I provided, along with analysis in a report, removing interviewees’ personally identifiable information in all the documents.

Why are there two places on the dashboard for how many people are helped in the lobby? ~Research Participant

Equity Considerations: My sampling strategy included asking employees to participate from different demographic backgrounds and tenure in the agency. I asked if they felt comfortable doing the interview and informed them they could stop the interview at anytime without penalty. Their personally identifiable information was not shared with the Data Warehouse Team. I then shared my research findings with all research participants, asking for any feedback or edits.

Outputs: (1) Three profiles of research participants who were non-managers, (2) a summary of the three managers’ diverging wants and uses of the dashboard, (3) visuals showing alternative presentations of data (thanks to a visual designer on the project team), and (4) a concise report with recommendations.

Frontline employee who uses the newer dashboard 90%+ more often than everyone else. I have been...In one agency program for decades. I like the newer dashboard because...
I can prioritize working on my overdue tasks by the date, which I can't do the older dashboard.
It tells me how many phone calls I answered last month and how long it took me. I don't have to ask my supervisor, "How am I doing?" I feel...
The older dashboard is better for showing Expedited 
     Services (ES) right away by just clicking on the date. 
The dashboard does not populate ES until 8:30am-10am so I hit both refresh links at the top all morning.
The newer dashboard is a week behind in my performance metrics. 
Today is Monday. The dashboard does NOT show my work on the phones from last Friday. It won't populate this info until this coming Friday.  
No one really explained to me and others how to use the dashboard.

Impact: Based on the insights, (1) the Data Warehouse Team agreed to ongoing feedback loops to keep improving the dashboard, (2) data is automatically refreshed earlier in the morning before employees start their workday, (3) managers are creating a communication strategy and an onboarding document to guide their frontline teams in using the dashboard, (4) edits have been made to discrete pages (e.g., one refresh link is at the top of a page instead of two), and (5) the team rechecked the number of people using the dashboard after all the changes were made and found an increase in uptake by more than 10%.