Building Data Culture through

Dashboards in SFUSD
Written by Ben Glazer (ben.glazer@gmail.com) for The Broad Residency.
Last updated April 26, 2010.

Abstract and Introduction

School districts face a number of challenges in using evidence to inform

decisions despite increasing pressure from government and private

organizations to do so. At the San Francisco Unified School District (SFUSD),

data collection efforts are usually driven by compliance requirements rather

than strategic considerations, data systems are often unlinked to one another,

and district staff frequently lack the time and sometimes the training they need

to effectively compile and interpret data. In this paper, I will present my efforts

at SFUSD to create an information dashboard for high school principals to

quickly communicate big-picture trends and specific aberrations. I will describe

our initial conditions, process, recommendations, and progress, focusing on

information dashboards as a catalyst for reform in other systems supporting

data inquiry cycles.

District Background

SFUSD comprises about 56,000 students and 7,000 staff, and covers the entire

city of San Francisco, roughly seven-by-seven square miles in area. The district
has an Information Technology Department with about 55 full-time employees,

and a Department of Research, Planning, and Accountability with around 50.

Goals

The objective—or means of achieving the goal—was to create information

dashboards that high school principals could review for 30 seconds each day to

get both a big picture overview of the trends in their schools and a detailed

understanding of critical issues that may demand an urgent intervention.

Principals weren’t using data much at all. Our goal was to change that by:

• Providing them with extremely easy access to relevant data.

• Collecting data strategically, not just to comply with regulations.

• Ensuring all data is linked to maximize its value to principals.

Ultimately, I planned to develop dashboards to support the needs of other

district staff, all of whom make decisions in their daily jobs that ultimately

affect students.

Outcomes

The project is far from complete, but present outcomes include:

• Dashboard designs. With the able assistance of a professional design

firm, I developed a dashboard interface mock-up that’s about 70%

 complete. Stephen Few, an dashboard design expert, provided a detailed

review of the interface and suggested several changes.

• Data governance. The process of reviewing data on dashboards has

helped to catalyze reforms in the area of data governance, which I

believe will kick off with a new position and department next year.

Situation

When I started my work as Director of Knowledge Management for SFUSD in

July 2009, my charge was to promote kaizen, or continuous organizational

improvement, by way of a “data culture” in which staff defined goals, collected

and analyzed data, and made decisions based on these analyses. After learning

about SFUSD’s context, other district and CMO approaches to the same

problem, and industry best practice, I developed and implemented several

proof-of-concept approaches to knowledge management (KM). These included

wikis, best practice catalogs, video capture and sharing systems for teachers,

and community-driven FAQs. However, I failed to build much traction with

these KM-related efforts due to a lack of political will and funding.

I subsequently began to examine the district’s approach to data entry, reporting,

and analysis—the systems and structures that supply staff with the data they

then synthesize to create knowledge. While there were a significant number of

independent systems (about 15), I found significant gaps in these systems

relative to what would be useful to district staff:

 • Ease-of-use. Data consumers—especially those empowered to make the

greatest impact, such as principals—lacked the time and sometimes the

expertise to make effective use of existing tools.

• Data quality. Due to a complete lack of any data governance system,

data entry was sporadic and rarely verified, leading to inconsistent and

often inaccurate data.

• Power. Data systems were completely unlinked in most cases, because

they had been adopted on an opportunistic, ad hoc basis driven by

compliance needs. As such, combining displays of data from different

systems, for example financial and student achievement data, was

nearly impossible. Additionally, sophisticated questions that require

data sets to be linked across systems, such as evaluating the cost/benefit

of an instructional program, were unanswerable.

In speaking with colleagues and IT managers in other districts, these gaps are

hardly unique to SFUSD. Rather, there are a multitude of efforts across the

nation to reform data systems along these three dimensions. Compared to an

initiative in private industry, there is less accountability and fewer incentives to

work with data in SFUSD, which poses a more significant change management

challenge.
Impact

The immediate scope of the project—information dashboards for high school

principals—would affect 15 principals out of a total district head count of 8,000

staff. However, those principals oversee about 580 teachers who teach nearly

20,000 students, and thus have significant influence over student achievement.

Funding

We are currently applying for additional funding to finalize the design and

complete a product implementation. To date, the only money spent has been the

$10,000 planning grant from the Carnegie Foundation.

Contribution

I began my work at SFUSD by studying the way knowledge is created,

exchanged, and preserved. I found little in the way of formal systems or

structures; rather, there existed a hodgepodge of uncoordinated, ad hoc

approaches. For example, SFUSD had established formal PLCs for principals

but offered nothing for teachers; as such, classroom expertise was largely

siloed. Across the organization, many policy and strategy decisions were made

mostly on the basis of consensus and opinion, loosely informed by academic

research.
In June, 2009, Natasha Hoehn, Executive Director of the Silver Giving

Foundation, led four sessions to discuss data that brought together about 50

leaders from across the district. I led a group discussion on data dashboard

interface design during one session. Following the four data sessions around the

end of the summer, I was asked to lead an effort to develop information

dashboards for the district.

I gladly accepted and decided to focus on high school principals as an initial

target user because:

• They work with the most politicized and visible data in the district:

standardized test scores.

• They are extremely time-constrained due to the sheer number of staff

and students they oversee.

• More data is available about high schools than other divisions, making

it more challenging for high school principals to use data effectively.

After identifying an initial “customer,” I conducted one-on-one interviews with

as many current and previous high school principals as would talk to me (about

15). The interviews focused on identifying the questions principals ask on a

daily or weekly basis to evaluate the "health" of their school.

With a $10,000 planning grant from the Carnegie Foundation, I hired an

experienced, professional graphic design and programming firm to develop

graphical mockups of the dashboard user interface. The firm also helped me

frame questions in a way that would make it easiest for them to find answers. In

my review of the literature on dashboard design and implementation, one

person stood out as the authoritative expert on the subject. I decided to contact

him to see whether he could help with the project. Serendipitously, he was a

Bay Area local with a passion for education. He donated his time to helping me

figure out how to frame the interviews and better design the interface.

Shortly thereafter, when the designs were about 70% complete, we had spent

all of the money from the planning grant. I submitted a report for inclusion in a

follow-on grant proposal to Carnegie and waited. In the meantime, I turned my

attention to two foundational systems on top of which dashboards are built:

• Data governance, which ensures data collection efforts are prioritized

high-quality.

• Data warehousing, which makes the fruits of data governance available

to tools (such as dashboards) by federating data across disparate

systems into a single interface.

After significant research on both topics, I produced a recommendation for

creating data governance structures in the district, and in collaboration with our

IT Department, I developed a simple proof-of-concept of a data warehouse.

Key Actions and Decisions

Several best practices aided in achieving the outcomes we have realized so far:
 • Be relentlessly user-centric. Solicit frequent feedback from users at

various stages of implementation. My approach was adapted from the

excellent text on usability, Don’t Make Me Think by Steve Krug.

• Find an expert (or become one). My information design guru was

Stephen Few, author of the authoritative Information Dashboard

Design. Based on a passion for education, Steve donated his time to the

project pro bono, and his book taught me everything I know about

dashboard design.

• Inspect the competition. I reviewed dashboards from a number of other

districts and CMOs (including Dallas, Fresno, and Achievement First),

as well as industry approaches (e.g. from enterprise-dashboard.com).

• Focus. Keep the scope of work as tightly focused as possible, and define

the target audience as specifically as possible. This helps you quickly

build a useful, working product so you can get feedback on it and

improve it.

• Choose powerful stakeholders. I chose to focus on high school

principals as my target end-users in part because they wield enormous

influence in the district. Demands from principals are much more likely

to be met than, say, demands from custodians.

Timeline

The timeline was regrettably loose, as it largely revolved around foundation

grant reporting and applications that seemed to be a moving target. However, I

moved as quickly as I could while funding was available, and when it ran out, I

shifted gears to “free” reforms of related systems. A comprehensive project

plan made it possible to communicate urgency to the various stakeholders,

including the executive sponsor, the IT Department, the Research Department,

and end-users.

Barriers

• User time. Principals, teachers, and most other district staff are

overworked as it is. Adding additional responsibilities to their plates

without removing others, will generally engender resentment.

• Funding. The current context for funding has been particularly bleak,

but in general, funding is always a challenge. I needed about $100k to

complete a solid working prototype, but that money never materialized.

• Technology integration. IT systems are often plentiful and independent.

The integration effort required can be significant, especially if the IT

Department staff are resistant to the initiative.

• Data fluency/training. Users won’t necessarily know how to read and

interpret data displays, complicating the design requirements of data

systems and support services (such as training).

Key Success Factors

• Stakeholder buy-in. Data system reforms require system-wide changes

in many cases; even a minor update may significantly affect staff in IT,
 Research, and schools. Buy-in across the system—from a committed

executive sponsor, though relevant departments, to end-users—is a

necessary prerequisite to any significant data system reforms.

• Manageable scope. Focus on a small scope of work and specific target

audience to simplify and expedite the implementation of a working

product. This relates back to the iterative approach; in general, iterative

development cycles enable frequent feedback that inevitably results in a

better product sooner.

Outcomes

The outcomes intended from this work include:

• Bootstrap a data culture by creating an extremely useful, relevant data

tool that works within users’ time and knowledge constraints.

• Drive a strategic wedge into underlying data systems. User-friendly

tools catalyze demand for higher data quality and linkages.

I had a year to make this happen but received less than half a year’s worth of

project funding. As such, the dashboard project is currently stalled at the design

phase, but some of the more aspirational outcomes around data system reform

are already happening. Principals are eager to have dashboards, though, and

I’m optimistic that we’ll be able to generate the additional political will and

funding needed to complete the project.

Appendices

Please see the attached files, which contain the following contents:

• Appendix A. Dashboard mockup.jpg: The latest work product, about

70% done by my estimation. This is a result of many hours of

requirements gathering followed by many hours of iterations on

designs.

• Appendix B. SFUSD High School Principal Dashboard Design

Recommendations from Perceptual Edge.docx: A review of the

dashboard mockup provided in Appendix A conducted by the foremost

experts in dashboard design.

• Appendix C. Dashboard Philosophy and Dev Process.doc: A detailed

description of the dashboard’s philosophical underpinnings and

development process.

• Appendix D. Dashboards implementation plan.png: A detailed timeline

describing the various steps needed to implement dashboards, including

initial plans for underlying systems (such as data governance

structures).