Data Dashboard Experiences

Blue bar graph with five bars on a scale from 0 to 600 # Clients Helped Today. They include Mail & Fax 5, Lobby 89, Phones 157, Text / Chat 236 and App 522.

The Department of Benefits and Family Services in the San Francisco government provides access to health care and money to pay for food, among other services. The Data Warehouse Team for the department worked with managers to develop a data dashboard. The purpose was to help frontline teams better manage their daily tasks and track their performance over time. As an internal research and design member the department, I was the lead individual contributor for the project managing and doing (1) research strategy, (2) operations, (3) data gathering, (4) analysis, (5) the creation of deliverables (aided by a visual designer) and (6) presenting the work.

Role: Project Manager & Lead Researcher

Year: 2022

Challenge: Supervisors and managers asked their 400+ frontline teammates to use the dashboards. About 40% did not. The Data Warehouse Team wanted to why and what could be improved.

Project Scope: I worked with the team to align on the project’s (1) goals, (2) questions to be answered, (3) timeline, (4) deliverables, and (5) success metrics.


Data dashboard showing only text. In the top left corner it says, "San Francisco Human Services Agency." In the middle of the page, near the top, the word "Refresh" is typed twice with an inch space between them. A black circle is drawn around them with a question mark?
There is date on the right that says 4/25/2022 at 9:32am.
QUESTION ASKED BY AN INTERVIEWEE: WHY IS “REFRESH” TWICE AT THE TOP OF tHis dashboard PAGE?

Strategy: The Data Warehouse Team (DWT) were partners throughout the research. I interviewed two of them about their experiences then the three managers overseeing the 400+ frontline teams, a supervisor and and two frontline employees (one who uses it a lot and another who rarely uses it). I asked them to show me how they use it or what they use instead, taking notes and screenshots with their permission. People felt uncomfortable being interviewed with the DWT joining our video call, so I conducted the 30-minute interviews independently. The DWT asked for my notes, which I provided, along with analysis in a report, removing interviewees’ personally identifiable information in all the documents.

This screenshot of the dashboard shows a green rectangle with the words inside, "Ave # of Lobby Cases Served" with "69" in large black font. Beneath that is five columns with four rows. The columns are: Service Name, # of Lobby Cases Served, # of Lobby Cases Abandoned, # Cases No Show, Sum of Agent Service Time (in Min). # of Lobby Cases Served is circled along with its four rows totaling 32. The interviewee was confused between the 32 total in this column and the 69 in large, black font above.
QUESTION ASKED BY AN INTERVIEWEE: WHY ARE THERE TWO PLACES ON THE DASHBOARD FOR HOW MANY PEOPLE ARE HELPED IN THE LOBBY?

Equity Considerations: My sampling strategy included asking employees to participate from different demographic backgrounds and tenure in the agency. I asked if they felt comfortable doing the interview and informed them they could stop the interview at anytime without penalty. Their personally identifiable information was not shared with the Data Warehouse Team. I then shared my research findings with all research participants, asking for any feedback or edits.

Outputs: (1) Three profiles of research participants who were non-managers, (2) a summary of the three managers’ diverging wants and uses of the dashboard, (3) visuals showing alternative presentations of data (thanks to a visual designer on the project team), and (4) a concise report with recommendations.

Frontline employee who uses the newer dashboard 90%+ more often than everyone else. I have been...In one agency program for decades. I like the newer dashboard because...
I can prioritize working on my overdue tasks by the date, which I can't do the older dashboard.
It tells me how many phone calls I answered last month and how long it took me. I don't have to ask my supervisor, "How am I doing?" I feel...
The older dashboard is better for showing Expedited 
     Services (ES) right away by just clicking on the date. 
The dashboard does not populate ES until 8:30am-10am so I hit both refresh links at the top all morning.
The newer dashboard is a week behind in my performance metrics. 
Today is Monday. The dashboard does NOT show my work on the phones from last Friday. It won't populate this info until this coming Friday.  
No one really explained to me and others how to use the dashboard.
PROFILE OF AN INTERVIEWEE WHO USES THE DASHBOARD A LOT

Impact: Based on the insights, (1) the Data Warehouse Team agreed to ongoing feedback loops to keep improving the dashboard, (2) data is automatically refreshed earlier in the morning before employees start their workday, (3) managers are creating a communication strategy and an onboarding document to guide their frontline teams in using the dashboard, (4) edits have been made to discrete pages (e.g., one refresh link is at the top of a page instead of two), and (5) the team is planning to recheck the number of people using the dashboard a month after all the changes.

Three images. The first is: This screenshot of the dashboard shows a green rectangle with the words inside, "Ave # of Lobby Cases Served" with "69" in large black font. Beneath that is five columns with four rows. The columns are: Service Name, # of Lobby Cases Served, # of Lobby Cases Abandoned, # Cases No Show, Sum of Agent Service Time (in Min). # of Lobby Cases Served is circled along with its four rows totaling 32. The interviewee was confused between the 32 total in this column and the 69 in large, black font above. 
The second image is of four, light, greenish circles with the following text in them: (1) Define that "lobby" means "everyone" (2) Larger font size of Individual data (3)Emphasis on Your Lobby data (4) Different colors
The third and final column in this image says, "Tomorrow" at the top whereas the other two said, "Today" and "Possible Changes." Under this Tomorrow column is a orange square on top of a blueish rectangle. Inside the orange square it says, "Your monthly lobby average, March 2022: 32. Average for everyone 69." Inside the blueish square it says, "Calls I answered last month 128. An upward arrow then "15 more than last month."
comparing the current and possible future state of the dashboard