Building DC's First P20W+ Longitudinal Data System

Building a P20W+ longitudinal data system requires solving problems and navigating data engineering, privacy, governance, and multi-agency relationships simultaneously. This paper documents how the District of Columbia's Office of Education Through Employment Pathways (ETEP), in partnership with Resultant, designed and implemented one of the first P20W+ systems built on Azure Databricks, and what that process revealed about doing this kind of work well.

The effort was supported by a strong governance framework that included ETEP's data governance and research committees, which help guide data-sharing decisions, research priorities, and responsible use of integrated data. The project also benefited from close collaboration with the District's Office of the Chief Technology Officer (OCTO), which serves as a key infrastructure partner supporting the District's enterprise data environment.

ETEP prioritized building the system in a way that maximized existing DC infrastructure and investments. The resulting cloud-native architecture, built on Azure Databricks, aligned with the District's technology strategy while providing the security, governance, and scalability needed to support the P20W+ system as it grew.

Azure Databricks serves as the foundation for the full data lifecycle: acquisition, ingestion, transformation, record linkage, governed access, and researcher access to approved datasets. Consolidating those capabilities within a single environment reduces integration complexity, simplifies governance, and makes the system easier for a lean team like ETEP to own and evolve.

The architecture was intentionally designed to support the District's privacy and security obligations. Data segregation, controlled movement of data between environments, role-based access controls, encryption, audit logging, and de-identification processes were implemented to align with federal privacy requirements and District security policies. These controls help ensure that sensitive information remains protected throughout the data lifecycle while still enabling approved research and longitudinal analysis.

Within the Development workspace, a medallion architecture organizes data through three progressive quality layers before curated datasets are promoted downstream:

• Bronze: raw ingested data
• Silver: cleaned and standardized data with cross-agency identity linkage
• Gold: de-identified, aggregated, research-ready datasets; the only layer researchers access

This structure ensures data quality is incremental and auditable. Each layer’s transformations are documented and version controlled.

In its first two years, the Education Through Employment (ETE) Data System has ingested 30 datasets from six partner agencies, covering early learning, K–12 enrollment, K–12 program participation, postsecondary enrollment, workforce training, social service program participation, and wage records. The system enables longitudinal analysis of more than a decade of education and workforce history.

Each dataset required agency-specific ingestion logic, field mapping, and standardization before it could enter the pipeline. Inter-agency data sharing agreements (DSAs) governed what data could be shared and under what conditions.

ETEP led the inter-agency DSAs across partner agencies, a significant coordination effort that ran parallel to technical development. Resultant provided advisory support and documentation to help the team prepare.

In addition to establishing DSAs, ETEP's governance and research committees provide ongoing oversight for data access, project approvals, and research priorities. These structures help ensure that data use remains aligned with agency objectives, community interests, and established privacy protections.