From Data, To Policy, To Practice: What Does This Look Like in an Open Recognition World?

Abstract

Data talks—it is used for policy making and for developing and evaluating practices—but current data structures in higher education and industry are founded on classical structures such as degrees, licenses, and certifications. Even when data are collected on newer credential structures (e.g., microcredentials), existing data structures and mindsets are employed to determine effectiveness and return-on-investments (ROI). ROI measures typically do not address goals, usability, long-term impact, or uptake. ROI needs to be redefined. 

The presentation argued that traditional research methods do not work; we need to redesign our approaches to collecting, storing, and using new forms of data. Most often, the questions being asked are based on old paradigms. For example, in the Credential As You Go research, a comparative interrupted time series (CITS) was employed to determine differences between newly developed incremental credentials and traditional credential counterparts; however, the analysis requires like data structures to compare. Thus, the data set dropped from 183 new credentials to 16 credentials with four years of baseline and two years of treatment data and matched comparisons. In other words, the restrictive nature of the CITS analysis prevented us from using most of the available (and valuable) data. Why? Because the newer credentials vary greatly in length, structure, and what they represent. As a result, matching to comparison credentials do not work. 

Another example presented focused on assessing learning. Projects exploring assessing prior knowledge from an Indigenous lens show we are asking the wrong questions of knowledge; the framing of our traditional approaches doesn't capture the way people learn and construct their knowledge. When we recognize learning by placing traditional Western frameworks onto that learning, much of the richness and diversity of learning remains Unseen. 

The goal of all of this work is to rethink the data that needs to be collected and ways in which that data can be evaluated. Interpretations of data need to be redesigned to capture the stories currently hidden. New frameworks need to be created, and technology solutions need to be rethought. This discussion went beyond the work being done on data inoperability and explored fundamental structures and assumptions being made about data resulting from recognition and credentialing spaces. 

This clinic began by sharing data issues arising from trying to do research on, developing policies for, and implementing practices in open recognition and credentialing. Following a brief contextual overview, the group discussed how data are collected, structured, analyzed, and shared for policy and practice development. Results from the group discussions will be brought together, in a forthcoming paper. 

Keywords: assessing learning, CITS analysis, evaluating credentials, Indigenous knowledge, alternative data collection