Quality measurement has been a vital component in monitoring the U.S. health care system’s quality of care—assessing the effectiveness of quality improvement initiatives and serving as the basis for pay-for-performance programs and public accountability reports [2]. However, current quality measure strategies are limited as they usually (1) focus on processes and treatment goals instead of health outcomes, (2) have a tendency to address a single risk factor or biomarker when patients often have multiple conditions, (3) focus on population thresholds that may not be relevant to individual patient risk [5], and (4) fail to take advantage of opportunities to engage and motivate patients [3].

Atherosclerotic cardiovascular disease (ASCVD) is the leading cause of death in the United States [4], and expenditures continue to be higher for it than for any other diagnostic group, with combined direct and indirect costs estimated at $320 billion in 2011 alone [5]. A personalized approach to quality measurement that accounts for patient preference could be particularly advantageous in minimizing the overall burden of cardiovascular-related death and disability in the United States. To tackle a problem of this magnitude, we propose to shift the current quality measurement focus from population assessments of individual indicators such as smoking status, hypertension management, and A1C control to one of patient-centric assessment using a model that conveys the likelihood of future adverse events. These patient-driven predicted outcomes can help clinicians recognize the specific characteristics of their patients—optimizing treatment of existing ASCVD—while concurrently improving the overall cardiovascular health of their patients by supporting decision-making in both prevention- and risk reduction discussions. The quality measurement approach proposed here continuously supports population health and incentivizes health care teams to engage patients and individualize care management plans based on goal-oriented, probability-weighted outcomes.

While person-centric risk models are desirable, their application to quality measurement presents a number of challenges. First, quality measure specifications that rely on electronic clinical systems as a primary data source require standard extraction protocols referencing structured data elements within the clinical databases. Current risk prediction tools in the United States are yet to be adopted broadly because they rely on manual searches of nonstandardized and unstructured patient data requiring significant resources [6]. Second, any new quality approach must not add to the existing measurement burden experienced by individual clinicians [7, 8]. Finally, most predictive modelling tools have been developed for specific patient populations or unique data sets, making effective quality assessment on a larger scale difficult and very expensive [9].

To meet these challenges, a patient-specific approach is needed that highlights those particular factors most relevant to a clinician’s treatment of ASCVD on a patient-by-patient basis [10]. Our measure of GCVR scores quantifies how well clinicians manage risk and health outcomes across their entire patient population with respect to individual clinical biomarkers and patient characteristics [11]. Linking contributing variables, derived from multiple sources, into a global, predicted outcome measure that accommodates the nuanced interactions between the ensemble of data points informing a person’s risk also provides relevant information about health care quality. The results from these calculations can be specified to assess effectiveness of care at many levels and could stimulate positive change in preventing avoidable events and lowering overall cardiovascular risk. The broader adoption of electronic clinical data systems and quality reporting standards makes it possible to systematically and consistently extract, transform and load (ETL) patient data needed for calculating GCVR, and reduces the burden of manual data extraction and measure calculation. Automating a structured process for clinical data query also introduces the possibility of returning quality information to clinicians in time to actually support decisionmaking. While much recent progress has been made in terms of electronic clinical data standardization and systems interoperability, many clinical practices still do not use standardized data, oftentimes documenting data within free-text fields that are not easily accessible for electronic-based quality measure reporting [6]. Therefore, a necessary first step in changing the quality model for cardiovascular care was systematically evaluating the consistency and completeness of structured electronic data. This paper reports on a pilot effort that investigated whether a standard technical specification could sufficiently define a process that would reliably generate outcome scores from heterogeneous data collected through ECDS.


NCQA recruited four sites to test the feasibility of generating clinician GCVR scores based on data from each organization’s electronic clinical systems. Participating organizations included one nonprofit health plan, one large nonprofit academic medical center, and two integrated health service organizations. Test sites were required to have a fully operational electronic health record system during the measurement period, the ability to warehouse data elements related to cardiovascular risk, and prior experience in reporting quality measure results. In order to obtain patient-level data from external organizations, NCQA first crafted a series of explicit instructions and technical requirements for reporting valid and reliable data. These specifications took the form of a GCVR Data Model (Appendix A) that describes the format of the necessary structured data elements, and a step-by-step process for submission through NCQA’s secure portal. A value set directory was also provided, specifically defining each individual data element in the measure using standard terminologies. This field test consisted of retrospective, electronic database research and a small number of key informant interviews assessing clinician views on the topic. A consent form, a semistructured protocol for our key informant interviews, and request for a waiver of Health Insurance Portability and Accountability Act (HIPAA) authorization on behalf of the participating organizations for this study—as allowed by 45 C.F.R. § 164.512(i) of the Privacy Regulations—were reviewed by the Institutional Review Board (IRB). As these data were not created prospectively for research purposes and were therefore determined not to be human subjects research, a waiver of HIPAA authorization for a limited data set was granted by the IRB.

At the start of the process, each component of the technical specification and protocol was first reviewed by independent subject matter experts and then by pertinent experts from each participating organization. Comments and feedback were incorporated as needed prior to going live with data collection. Once data collection was complete, all sites provided detailed feedback on their experience with the technical documentation as well as their personal experiences in programming the queries to extract the requested data elements from their local systems. In order for a new quality measurement approach to be considered feasible, all participating sites must have been able to follow the protocol as instructed and to generate the requested data in sufficient quantity and of sufficient quality for consistent generation of GCVR scores.

Our next step was to select a risk calculator. There are many risk calculator options available—each with its own set of data requirements, a unique algorithm for determining future outcomes, and a variable level of evidence supporting its risk estimations [12, 13, 14, 15, 16]. For this research, we selected an evidence-based, predictive-risk calculator provided by Archimedes Inc. ( that utilizes a large number of clinical variables to generate relevant risk predictions for the population under study [17, 18]. For each individual patient, the Archimedes calculator uses three scalable treatment-outcome scenarios, which are assembled to produce a GCVR score for each provider [19]. Scenario 1 starts by determining a “no care” risk profile by reversing out any current treatments, eliminating the potential benefits of current care in order to estimate risk as if the patient were not under any treatment. Scenario 2 determines a profile for each patient corresponding to actual current performance levels. Scenario 3 determines a “target” by which each patient will have achieved an optimal level of care. Each scenario has a corresponding rate of predicted outcomes. Using each patient’s data to determine these three points enables the GCVR score to reflect the proportion of preventable adverse events that are being prevented at current levels of care of the total number of preventable adverse events estimated under optimal care.

Each organization identified an eligible population of patients ages 18 to 85 years whose records included retrospective data collected during the 24-month period between January 1, 2011 and December 31, 2012 and produced a .csv data file populated with patient encounters, health risk assessments, and other electronically available information. One of the integrated delivery systems (IDS 1) and the academic medical center limited their eligibility to patients with a prior diagnosis of hypertension, diabetes, or cardiovascular disease (high-risk population), while the second integrated delivery system (IDS 2) and the health plan included any patient meeting the encounter timing and age criteria. This information was then sent to NCQA via a secure file transfer service for verification and for cataloguing of any of the data elements that were missing or misclassified. Next, the files were securely uploaded to the Archimedes server to generate GCVR scores (Figure 1 and 2). Finally NCQA matched GCVR scores to clinician National Provider Identifiers (NPIs) and organization IDs in the research database to analyze the calculated GCVR performance results.

Figure 1 

Data Flow Process for Feasibility Testing

Figure 2 

Clinician GCVR Scores by Test Site


The four organizations successfully returned in excess of 480,000 individual patient records to NCQA for analysis. From this data set, GCVR scores were generated for 2,251 clinicians using 277,780 of those patient records. Records were excluded from the GCVR scoring process due either to data incompleteness for the period under study or to inaccurate data (i.e., not exactly matching the criteria as specified in the technical documentation). We intentionally did not enable any imputation functions available within the Archimedes calculator for this study as we wished to assess the proportion of raw data that could be produced by each of the organizations, following the provided specification. Table 1 presents the aggregated organizational results, illustrating the potential scale of the GCVR score for the total patient population. The (n) is the number of patients at each site with sufficient data to generate a GCVR score. Current Averted Events represents the predicted number of events that, under current levels of treatment, will be prevented in that patient sample, and Optimal Averted Events represent the total number of potentially avoidable events in the population if all patients were receiving optimal treatment based on their individual risk profiles. The GCVR score represents the variance between these two estimates for the sample population.

Table 1

Organizational GCVR Score by Test Site


IDS 1** 69.7 45,198 1,418 2,036
IDS 2 76.5 145,277 1,420 1,856
AMC** 59.7 40,155 896 1,499
Health Plan 59.1 47,150 346 585

Notes: *A higher score is better. ** IDS = Integrated Delivery System; AMC = Academic Medical Center.

Table 2 presents the numeric details of the scatter plots for each organization. The (n) in Table 2 represents NPIs to which the clinician GCVR scores were attributed within each organization.

Table 2

Clinician GCVR Scores by Test Site


IDS 1 1083 70.3 11.2 58.0 66.0 72.2 77.3 81.0
IDS 2 24 76.0 5.4 74.2 75.2 76.4 77.9 78.3
AMC 1052 60.1 18.5 32.8 57.1 64.7 72.2 75.6
Health Plan 92 48.8 21.4 10.6 45.5 55.7 62.4 69.4

In testing the feasibility of GCVR, clinician workflow and decision-making about what is transcribed into the patient record was of foremost interest. It became evident that while it is relatively easy to execute electronic queries in many EHRs, altering clinical workflow patterns to accommodate new measure data requirements are difficult if members of the health care team do not immediately see how the change benefits the patient. For this reason, the GCVR team used a process of continuous feedback throughout the testing process as solutions to each of the various technological challenges were considered.


Almost 60 percent of the patients in our system had sufficient structured data for us to calculate a GCVR score, demonstrating the feasibility of calculating predicted outcome scores using a large number of patient variables. Despite the increased use of national standards, longitudinal clinical data are still represented in many different formats.

Even relatively common clinical data elements are not always stored in standardized, structured fields; they contain repeating data elements across irregular time frames (e.g., outpatient blood pressure readings) and may include patient assessment information from several settings—posing a difficulty in constructing measures like the GCVR. Because a GCVR-based measurement framework requires more data than does typical performance reporting, our pilot was critical to understanding whether complex technical specifications could be efficiently implemented. Understanding how to communicate necessary data requirements such that they are uniformly interpreted by multiple organizations that can then report valid quality results is crucial in developing a new quality measure. Since our main focus was on feasibility, we needed to deliver a highly specific request in a common format to overcome the lack of standardization and to establish a process by which disparate data sets could be transformed (manually at first, but electronically in the future) to our GCVR data model that would then permit consistent and reliable calculation of outcome scores.

Now that feasibility has been successfully demonstrated, a measure like GCVR can become a cornerstone for a new quality framework that assesses the influence of modifying any one of a patient’s factors to see if it affects an individual’s likelihood of an adverse event. GCVR offers a unique opportunity to assess progress by relying on advanced technology to identify patients’ optimal treatment targets, encouraging clinical preventive strategies that set risk reduction goals, then measuring the attainment of those goal [20, 21]. As the complexity of electronic clinical data increases and variables indicative of treatment multiply measure developers must consider more efficient ways of assessing clinical care to produce timely and actionable information. An analytic model that accurately predicts cardiovascular outcomes in heterogeneous populations would be useful, despite the additional complexity because the new quality framework prioritizes a patient-centric approach. Our research findings demonstrate that well-defined technical specifications and a protocol requiring structured data can provide such solutions for quality measurement, regardless of site-specific limitations.


The limited scope of this study prevented us from performing a multidimensional quality analysis of the data files, or an evaluation of the semantic and syntactic accuracy of the data extracted [22]. However, our success in producing GCVR scores from heterogeneous clinical data encompassing more than 270,000 patient records demonstrates the feasibility of the process from a data completeness perspective [23]. In the present study certain data types—such as medications and patient history—presented particular challenges that we worked around by requiring that separate files be submitted by participants, which could be individually validated and integrated into the patient-level file. As both measure developers and health care organizations obtain more experience with the level of technical specification required to ensure consistent ETL processes, we expect many of these steps to become unnecessary in future iterations.


To our knowledge, this is the first time that predictive models have been proposed for quality measurement. This pilot successfully demonstrated that a predicted outcome measure is feasible using electronic patient data. However, new specification standards are required before this approach is fully scalable to the level of a national quality reporting program. As experience with reporting measures using ECDS increases, improvements in quality reliability and standardization will follow–facilitating improved cardiovascular risk predictions and advancing new measurement concepts in health care quality assessment. GCVR’s transformative strategy offers a valuable opportunity to evaluate quality improvement by assessing patient-specific health outcomes on a national scale.