Post a cohesive response based on your analysis of the Learning Resources and your professional experience. Be sure to discuss

6-1

Section 6: Examples of Evaluation Measures

Section 6 includes tables that list sample measures you might use to evaluate your HIE project. Each table includes possible measures, suggested data sources for each measure, practical notes, considerations, and, when available, links to suggested resources. The tables are not exhaustive, but rather highlight measures that have been commonly used to evaluate HIE projects. You should not try to incorporate all or a large number of measures into your evaluation; it is likely that only a small subset of the measures is directly applicable and relevant to your project. Your evaluation team should carefully consider whether a measure is important and applicable to your HIE project, based in part on having an understanding of the resources required to develop the study design and collect and analyze the data for the measures. For example, some data sources may be difficult to access or costly, and patient data may require informed consent. Based on the information provided, do your best to determine whether the value of a given measure outweighs the corresponding cost to your project’s resources.

Section 6 is divided into three subsections:

1. “Measures to evaluate the process of creating an HIE organization” provides a set of mostly process measures to evaluate progress in planning and implementing an HIE system.

2. “Measures for specific types of data exchange” provides details about specific measures, based on the kind of data that are being exchanged among the health care providers participating in the HIE project.

3. “Measures for clinical outcome and clinical process evaluation” provides suggested outcome, impact, and financial measures to use in analyzing the process and quality of clinical care.

Measures To Evaluate the Process of Creating an HIE Organization The tables in this subsection provide examples of measures that are based on the structure and function of an HIE organization. Most of these measures are “Yes/No” measures, and can be ascertained from strategic planning, operations planning, legal, technical, and other documents (e.g., meeting minutes, Gantt charts, and organizational charts). The tables list measures in the following categories:

z Table 6-1. Measures of the infrastructure development effort;

z Table 6-2. Measures of process.

Guide to Evaluating Health Information Exchange Projects

6-2

Table 6-1. measures of The infrasTruCTure developmenT efforT

Measure Data Source(s) Practical Notes Considerations

• Has a strategic plan been developed for the HIE organization?

• The governing board for the HIE organization may have an executive or management team responsible for planning. In the absence of a governing board, the HIE organization executives will assign planning responsibilities.

A strategic plan is a document that describes the mission, vision, and goals of the HIE organization.

The strategic plan is a persistent document that drives the development and direction of the HIE organization.

• Have the appropriate stakeholders been identified (i.e., the institutions and individuals who will participate or be impacted by the evaluation)?

• Strategic or business plan documents

• Minutes from governance meetings

• Memorandums of understanding

• Business agreements

• Standards of participation

An HIE organization’s stakeholders typically include both institutions and individuals. Institutions may include laboratories, pharmacies, hospitals, clinics, long-term care facilities, radiology offices, and payers. Individuals may include providers, pharmacists, allied health care workers, and patients.

It is important to include patients as stakeholders

• Has the legal climate for data sharing been ascertained?

• Is there a document establishing the legal entity (e.g., articles of incorporation or State legislation)?

• Have data-sharing agreements been executed among the partners?

• Have State privacy laws been mapped to the HIE project?

• Strategic plan

• Operations plan

• Stakeholders

• Data-sharing agreements

Many factors impact stakeholders’ willingness and ability to share data.

Be aware of issues arising when the data are to be shared across State boundaries, as the legal environment may be different from State to State.

6-3

Measure Data Source(s) Practical Notes Considerations

• Has a technical plan for data sharing been developed?

• Has there been an assessment of all systems that are to become part of the exchange?

• Will each site have an HIE-provided server?

• Strategic plan

• Technical architecture documents

A technical plan typically specifies the architecture, the hardware and software to be used, and the required technical standards to be implemented.

The technical plan will likely change as the HIE project and system evolve.

• Has an implementation team been identified?

• What resources have been assigned by each participating organization?

• Committee meeting minutes

• Planning documents

• Operations plan

An implementation committee is typically responsible for overseeing the implementation effort, organizational processes, and costs.

• Has an HIE project plan been developed?

• HIE project plan

• Gantt charts

• Strategic plan

• Operations plan

A project plan is necessary to allocate tasks to individuals and teams that will be responsible for conducting them, monitoring task completion and the project schedule, and monitoring project costs.

The project plan may be revised, and it should be updated over time as the HIE project evolves.

• What specific data elements are to be shared, and why?

• Have standards for data exchange been identified?

• Is there a testing plan to validate data being sent?

• Strategic plan

• Operations plan

Selection of data to be exchanged will help determine the necessary technical components of the HIE system and the type of data-sharing agreements that will be needed.

Be aware that different partner organizations may define these data elements differently.

• Have sources of data elements been identified?

• Minutes from technical architecture discussions

• Strategic plan

• Operations plan

The source of the data elements could include EHR systems and other databases and systems (e.g., registration system, billing system, pharmacy system).

Sources for needed data elements will vary across organizations, and the data may need to be reorganized or relabeled so it can be easily understood across institutions.

Table 6-1. measures of The infrasTruCTure developmenT efforT (ConTinued)

6-4

Measure Data Source(s) Practical Notes Considerations

• Is there a procedure in place to obtain patient consent to share their data? to opt-out of sharing their data?

• Does that consent include any use of de-identified data for research purposes?

• Who is responsible for obtaining consent?

• Has the HIE project operationally defined any data that will have special protection, such as behavioral health, Federal alcohol and drug treatment, adolescent reproductive health, or other sensitive data?

• Has the technical implementation team been educated about these definitions?

• Operations plan

• Legal documents

• State law

• Patient consent forms

• Trust Principles/ Framework

HIE organizations may use different consent models. For example, some may use an “opt-in” model, in which patients are explicitly asked for consent to participate in the HIE system. In those cases, some institutions may prefer to ask patients for a single agreement to share all pertinent patient data, while others may prefer to request patient consent to share each data element to be shared. Other HIE organizations may use an “opt-out” consent model, in which patient data will be shared unless they decline participation.

State law may stipulate consent provisions. The Health Insurance Portability and Accountability Act (HIPAA) has special requirements for consent in a research context. Be sure to consult with an institutional review board regarding your evaluation plan design.

Table 6-1. measures of The infrasTruCTure developmenT efforT (ConTinued)

6-5

Measure Data Source(s) Practical Notes Considerations

• Are security and privacy policies in place for all HIE partners?

• Risk assessment process

• Internal reviews and monitoring, including reactive and preventive controls

• User authentication and access controls

• Competence of personnel; privacy and security training

• Physical and environmental security

• Personal health information collection and use limits

• Notice of data practices

• Personal health information integrity and correction processes

• Third-party transfer restrictions

Before determining which legal and information-sharing agreements should be applicable to those they contract with, an HIE project should consider their current internal policies and practices for maintaining the privacy and security of personal health information.

• Have governance structures been established?

• Have meetings of the governance group been held?

• Operations plan

• Articles of incorporation

• State legislation establishing an HIE organization

• Business principles

• Is an evaluation planned as a part of the HIE project?

• Operations plan Evaluation is necessary to assess the impact of the HIE project.

The evaluation may evolve as the HIE organization and system develop.

Table 6-1. measures of The infrasTruCTure developmenT efforT (ConTinued)

6-6

Table 6-2. measures of proCess

Measure Data Source(s) Practical Notes Considerations

• Are participating organizations ready to share the specific data elements?

• Committee meeting minutes and other documents

• Signed data-sharing agreements

It is important to understand whether stakeholders understand what data elements will be shared, how these data elements are represented in their databases, and whether they have plans in place to share data electronically.

It is important to understand each stakeholder’s knowledge regarding these specific data elements.

• Do stakeholders know their roles and responsibilities on the HIE project?

• Governance diagrams

• Charter documents

• Legal documents

Stakeholders’ understanding of their roles is important, as roles and responsibilities are instrumental for building trust and settling disputes.

• Has the technical architecture been finalized?

• Meeting minutes and documents

A technical architecture typically specifies the data-sharing model, the standards and interfaces to be used between systems, the patient matching scheme, the data aggregation scheme, and security.

• Is the implementation progressing according to the project timeline?

• Project plan

• Implementation plan

• Is the implementation proceeding within budget?

• Budget

• Implementation plan

• Actual costs

• Project plan

It is important to monitor implementation costs for each deliverable. The organization may need to shift or reallocate efforts if costs are higher than anticipated.

Cost-related data may be difficult to obtain and analyze for large-scale projects.

6-7

Measures for Specific Types of Data Exchange The tables in this subsection provide examples of measures based on five types of data exchange. Each table provides measures regarding the value of one particular type of data exchange. Some measures are “exchange capability questions,” which are simple “Yes/No” questions as to whether the exchange has achieved certain capabilities. These capability questions do not require every participating organization to have achieved the functionalities, but it is necessary to demonstrate that the exchange organization and technical infrastructure can support the functionalities. The tables list measures in the following categories:

z Table 6-3. Data exchange between providers and laboratories

z Table 6-4. Data exchange between providers and pharmacies

z Table 6-5. Data exchange between providers

z Table 6-6. Data exchange between providers and radiology centers

z Table 6-7. Data exchange between providers and public health departments

6-8

Table 6-3. daTa exChange beTween providers and laboraTories

Measure Data Source(s) Practical Notes Considerations Resources

• Was electronic ordering of laboratory tests between outpatient providers and laboratories achieved?

• Implementation team

Exchange requires an interface between the ambulatory EHR system and the laboratory data system. This is an exchange capability question as to whether this has been demonstrated anywhere within the exchange.

Is this a standards- based bidirectional interface?

• Are providers using data exchange capability with laboratories?

• Usage statistics from system’s audit logs (e.g., order logs, result view logs, system log-on tracking)

You could measure this in several ways. One would be to divide the number of providers using the system (numerator) by the number of total providers (denominator). A second approach might measure how often individual providers are accessing the system, with access hit rates as the numerator and the number of individual providers as the denominator. A third approach might be to get an overall average rate by dividing the number of access hits by the total number of providers. Providers might be defined as nurses and/ or physicians. Tracking this information over time and presenting it visually would give stakeholders an understanding of adoption trends for your project. You could also track the number of paper transactions still being used (i.e., clinical staff putting laboratory results into records).

Finding baseline provider rates might be difficult. For example, what is your sample of physicians who could be using the system? You could consider getting this information from local medical societies or boards of medicine.

See Canada Health Infoway’s Benefits Evaluation Indicators Technical Report, p. 133, for a detailed definition and evaluation method for this measure.

35 This

resource is freely available.

6-9

Measure Data Source(s) Practical Notes Considerations Resources

• What percentage of laboratory orders is sent electronically?

• Usage statistics from system’s audit logs

The denominator is all orders (electronic and paper). The numerator is electronic orders only. This can be done on both the laboratory and provider side.

This measure can be costly if it requires counting paper orders.

See Canada Health Infoway’s Benefits Evaluation Indicators Technical Report, p. 136 for a detailed definition and evaluation method for this measure.

35

• Was there a reduction in calls to providers to clarify an order?

• How much of a reduction?

• Call logs This measure requires tracking call volume before and after the intervention.

Calls may not be for order clarification but to report other issues (e.g., improper specimen collection, unavailability of test, or new test version).

See Canada Health Infoway’s Benefits Evaluation Indicators Technical Report, p. 64 for a detailed definition and evaluation method for this measure.

35

• What was the reduction in costs to send orders to laboratory?

• Pre- and post- implementation review of financial logs, time and motion studies, and workflow analysis in a sample of various settings

First, estimate what these costs are per order (labor costs to prepare forms, costs to send forms) and then multiply by the number of orders sent out. Using time and motion studies compare paper and electronic methods on how much time individuals spend searching for results, writing orders, and transcribing; multiply time by mean staff hourly wage.

Make sure to track orders electronically. The cost of an “electronic transfer” is not zero; it includes the cost of developing and maintaining the infrastructure to send the information electronically.

See AHRQ’s Time and Motion Studies Database for a detailed definition of this measure and additional resources. This resource is freely available.

34

Table 6-3. daTa exChange beTween providers and laboraTories (ConTinued)

6-10

Measure Data Source(s) Practical Notes Considerations Resources

• Impact on duplicate laboratory tests

• Pre- and post- implementation review of claims data

If you are rolling out your project in stages, you could use those organizations or providers who have not gone live yet as your control group, thereby avoiding the need for a retrospective medical record review. You may also be able to use billing data to help focus the search for redundant tests.

Need to define “duplicate” for each type of test. For example, the definition of duplicate would differ by type of blood test, and would differ based on whether the initial test were normal vs. abnormal. This measure might be costly if you have to do a medical record review.

• Was electronic exchange of laboratory results between outpatient providers and laboratories achieved?

• Implementation team

This exchange requires an interface between the ambulatory EHR system and the laboratory data system. The measure is whether exchange capability is in place.

Is this a standards- based bidirectional interface?

• Impact on the number of calls to the laboratory for results

• Laboratory call logs

A reduction in the number of calls to the laboratory for results suggests that providers can find results in a timelier fashion.

Measurements need to be adjusted for the volume of tests conducted by each of the participating laboratories. Also, changes in market share by laboratories need to be considered.

Table 6-3. daTa exChange beTween providers and laboraTories (ConTinued)

6-11

Measure Data Source(s) Practical Notes Considerations Resources

• Decrease in time to report critical results by the laboratory

• Call logs pre- and post- implementation

This is a great measure to consider, given the Joint Commission’s interest in this topic.

If call log information is not already being collected, it will be hard to collect.

See Canada Health Infoway’s Benefits Evaluation Indicators Technical Report, p. 57, for a detailed definition and evaluation method for this measure.

35

• Costs saved for sending and receiving results

• Financial logs Estimate the costs associated with receiving a single result (labor to open mail, sort, distribute to clinicians, and post on patient medical record) and multiply by the number of laboratory results received.

If users are still printing out electronic results to put in paper medical records, this cost must be considered as well.

• Impact on the satisfaction of clinicians

• Surveys or focus groups examining the perception of usability, the ease of learning to use the system, and efficiency as a result of the data exchange

You might consider sampling both your users as well as clinicians who could be involved in the project but who have chosen not to participate. Going to State- or region- wide provider databases from local medical societies or boards of registrations may be ways to determine your target survey group. Consider questions such as asking clinicians how often they were able to find the result they were looking for in a timely manner. You could compare responses before and after implementation. It may be helpful to conduct satisfaction surveys multiple times at different stages of the project to monitor trends and potential unintended consequences (positive and negative).

Table 6-3. daTa exChange beTween providers and laboraTories (ConTinued)

6-12

Measure Data Source(s) Practical Notes Considerations Resources

• Satisfaction of laboratory personnel

• Survey or focus groups

Your survey could sample the laboratory technicians, or the administrative personnel, including those who are responsible for taking phone calls. The survey would need to be designed to be distributed to all involved laboratories. It could be helpful to conduct the survey multiple times at different stages of the project to monitor trends and potential unintended consequences (positive and negative).

Be careful to survey only the personnel affected by data exchange, which may be invisible to some staff. That is, they may not know to whom the data are being sent or who is accessing it. For example, if a laboratory result is viewed by a provider outside the laboratory’s traditional service base, the laboratory technician may not know that, and thus may not be aware of the data exchange.

Consider using or amending an existing satisfaction survey. Review existing surveys using the Health IT Survey Compendium on the AHRQ Health IT Web site.

32

• How much data were able to be exchanged?

• Implementation team

• Data exchange logs

• Number of messages sent or received

Look at the number of discrete elements that were exchanged.

Note that just because a message was sent properly, it does not mean that it was received and processed properly. For example, if an abnormal result is placed in an exception queue, it might stay in that state for months before the “correct” individual has access to those results.

Table 6-3. daTa exChange beTween providers and laboraTories (ConTinued)

6-13

Table 6-4. daTa exChange beTween providers and pharmaCies

Measure Data Source(s) Practical Notes Considerations Resources

• Is e-prescribing available in your HIE region?

• What percentage of prescribers use EHR technology to e-prescribe?

• What percentage of prescribers use a standalone system for e-prescribing?

This could be accomplished through an e-prescribing system (i.e., via RxHub or SureScripts) or through an existing HIE system. This measures whether this type of exchange capability is available through the HIE system.

Is this a standards-based bidirectional interface? What does the pharmacy communicate to the provider? Is that communication done using electronic exchange of information?

• Are providers using data exchange capability with pharmacies?

• How many new prescriptions vs. renewals were ordered electronically?

• How are providers performing on meaningful use measure 4, which (for stage 1) requires 40 percent of all prescriptions to be sent electronically?

• Usage statistics from system’s audit logs

• Implementation team

• Regional extension centers, which track the number of providers who have reached milestone 3, (attesting to meaningful use)

Electronic information collection is possible in several ways. First, you could look at the number of electronic prescriptions received as the numerator and the total number of prescriptions received (both electronic and printed) as the denominator. A second approach would be to divide the number of physicians submitting prescriptions electronically (numerator) by the total number of users of the system (denominator). A third approach would be to divide the number of physicians submitting prescriptions electronically (numerator) by the total number of physicians in the service area (denominator). In addition to providers who have reached milestone 3, providers who have reached milestone 2 have implemented an EHR system and may have operationalized e-prescribing. Any authorized testing and certification bodies (ATCB)-certified complete EHR system must be able to e-prescribe. Some States have found that initial orders for prescriptions are being ordered electronically, while renewals are not ordered electronically.

Measure Data Source(s) Practical Notes Considerations Resources

Table 6-4. daTa exChange beTween providers and pharmaCies (ConTinued)

6-14

6-15

• How much data were able to be exchanged?

• What type of data were exchanged (formulary, eligibility, medication history), and by whom?

• How many electronic drug orders were transmitted as a percentage of total drugs ordered?

• Usage statistics from system’s audit logs for e-prescribing orders

Use the number of e-prescribing orders sent as the numerator and the total number of prescriptions filled (both electronic and printed) as the denominator. The total number of prescriptions may need to be estimated by surveying a sample of provider practices, or by reviewing e-prescribing system audit logs.

Be sure that the messages were correctly received and processed on the receiving end. Evaluators may need to contact the pharmacy to verify the numerator.

• Impact on calls to pharmacies

• Provider call logs with protected health information removed

The logs should also capture the nature of the call.

This is primary data collection from the provider office.

• Impact on calls to providers to clarify a prescription

• Pharmacy call logs with protected health information removed

Make sure the pharmacy call log has the requisite level of detail to capture the nature of the call.

This is primary data collection.

See Canada Health Infoway’s Benefits Evaluation Indicators Technical Report, p. 54, for a detailed definition and evaluation method for this measure.

35

• Impact on costs due to improved formulary compliance or use of generic drugs

• IT team

• Medical record reviews

• Health plan utilization review databases

If the new system has decision support, the system may have the data to show how often a switch is made from a nonformulary choice to a formulary alternative. Evaluating formulary patterns may be more feasible if you focus on a single drug class or narrow down to a subset of patients.

It could be difficult to find the pre- implementation compliance rate. The measure may be costly if medical record reviews are required.

Measure Data Source(s) Practical Notes Considerations Resources

Table 6-4. daTa exChange beTween providers and pharmaCies (ConTinued)

6-16

• Impact on costs by switching to generics

• Health plan utilization review databases

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.