RWT Results 2022 Q3

General Information

Plan Report ID Number
Developer Name	Medical Informatics Engineering
Product Name(s)	WebChart EHR
Version Number(s)	7.4
Certified Health IT Product List ID(s)	15.04.04.1932.WebC.74.00.0.181219
Developer Real World Testing Page URL	https://docs.webchartnow.com/resources/system-specifications/ehr-certification/real-world-testing/
Plan Submission Date	11/15/2021
Results Version	Q3, completed 10/15/2022

Certification Criteria to be Tested

• ‘‘Transitions of care" (§ 170.315(b)(1) Cures Update);
• “Clinical information reconciliation and incorporation” (§ 170.315(b)(2));
• “Electronic prescribing” (§ 170.315(b)(3) Cures Update);
• “Data export” (§ 170.315(b)(6));
• “CQMs – record and export” (§ 170.315(c)(1));
• “CQMs – import and calculate” (§ 170.315(c)(2));
• “CQMs – report” (§ 170.315(c)(3));
• “View, download, and transmit to 3rd party” (§ 170.315(e)(1));
• “Transmission to immunization registries” (§ 170.315(f)(1));
• “Application access – patient selection” (§ 170.315(g)(7));
• “Application access – data category request” (§ 170.315(g)(8));
• “Application access—all data request” (§ 170.315(g)(9) Cures Update);
• “Direct Project” (§ 170.315(h)(1))

Criteria-Measure Matrix

Criteria	Requirement	Measure
"Transitions of care" (§170.315(b)(1))	(b)(1)(i)(A)(Alternative) - Send Using Edge Protocol for SMTP/IXE XDR	17
(b)(1)(i)(B)(Alternative) - Receive Using Edge Protocol for SMTP/IXE XDR	17
(b)(1)(i)(C)(Conditional) - XDM Processing	17
(b)(1)(ii)(A) - Receive, Parse, and Process	7, 19
(b)(1)(ii)(B) - View	7
(b)(1)(ii)(C) - Section Display	7
(b)(1)(iii) - Create	7
(b)(1)(iii)(A) - Assessment, Plan, Goals, Health Concerns	7
(b)(1)(iii)(B) - Diagnoses	7
(b)(1)(iii)(C) - Cognitive Status	7
(b)(1)(iii)(D) - Functional Status	7
(b)(1)(iii)(E) - Ambulatory Referral Summary	7
(b)(1)(iii)(F) - Inpatient Discharge Instructions	7
(b)(1)(iii)(G) - Patient Matching	7
"Clinical information reconciliation and incorporation" (§170.315(b)(2))	(b)(2)(ii) - Correct Patient	7
(b)(2)(iii)(A) - Simultaneous Display	9
(b)(2)(iii)(B) - Reconciled List	9
(b)(2)(iii)(C) - User Review	9
(b)(2)(iii)(D) - List Acceptance	9
(b)(2)(iv) - CCD Creation	9
"Electronic prescribing" (§170.315(b)(3))	(b)(3)(ii)(A)(1) - NewRx	3
(b)(3)(ii)(A)(2) - RxChangeRequest, RxChangeResponse	3
(b)(3)(ii)(A)(3) - CancelRx, CancelRxResponse	3
(b)(3)(ii)(A)(4) - RxRenewalRequest, RxRenewalResponse	3
(b)(3)(ii)(A)(5) - RxFill	3
(b)(3)(ii)(A)(6) - RxHistoryRequest, RxHistoryResponse	3
(b)(3)(ii)(A)(7) - Status	3
(b)(3)(ii)(A)(8) - Error	3
(b)(3)(ii)(A)(9) - Verify	3
(b)(3)(ii)(C)(1) - Primary/Secondary Diagnosis	4
(b)(3)(ii)(E) - Metric Units	5
(b)(3)(ii)(F) - Decimal Format	6
"Data export" (§170.315(b)(6))	(b)(6)(i) - Configure and export	18
(b)(6)(ii) - Set Export	18, 19
(b)(6)(ii)(A) - CCDS	18, 19
(b)(6)(ii)(B) - Diagnoses	18, 19
(b)(6)(ii)(C) - Cognitive Status	18, 19
(b)(6)(ii)(D) - Functional Status	18, 19
(b)(6)(ii)(E) - Ambulatory Reason for Referral	18, 19
(b)(6)(ii)(F) - Inpatient Discharge Instructions	18, 19
(b)(6)(iii)(A) - Timeframe configuration	18
(b)(6)(iii)(B) - Export summary	18
(b)(6)(iv) - Save location	18
"CQMs – record and export" (§170.315(c)(1))	(c )(1)(i) - Report	1
(c )(1)(ii) - Export	1
"CQMs – import and calculate" (§170.315(c)(2))	(c )(2)(i) - Import	2
(c )(2)(ii) - Calculate	1, 2
"CQMs – report" (§170.315(c)(3))	(c )(3)(i) - Report	1, 2
"View, download, and transmit to 3rd party" (§170.315(e)(1))	(e)(1)(i) - Web Content Accessibility	21
(e)(1)(i)(A) - View	14
(e)(1)(i)(A)(1) - CCDS	14
(e)(1)(i)(A)(2) - Ambulatory Contact Info	14
(e)(1)(i)(A)(3) - Inpatient Discharge Instructions	14
(e)(1)(i)(A)(4) - Lab Report	14
(e)(1)(i)(A)(5) - Diagnostic Imaging Report	14
(e)(1)(i)(B)(1)(i) - Download Human Readable	15
(e)(1)(i)(B)(1)(ii) - Download CCD	15
(e)(1)(i)(B)(2) - CCD Human Readable	15
(e)(1)(i)(B)(3) - Inpatient Summary	15
(e)(1)(i)(C)(1)(i) - Email	16
(e)(1)(i)(C)(1)(ii) - Encrypted Transmission	16
(e)(1)(i)(C)(2) - Inpatient Email	16
(e)(1)(i)(D)(1) - Specific Date	14, 15, 16
(e)(1)(i)(D)(2) - Date Range	14, 15, 16
(e)(1)(ii)(A) - Activity Log	14, 15, 16
"Transmission to immunization registries" (§170.315(f)(1))	(f)(1)(i) - Create Content	10
(f)(1)(ii) - Query Records	11
"Application access – patient selection" (§170.315(g)(7))	(g)(7)(i) - Query processing and response	20
(g)(7)(ii)(A)(1) - Functional Documentation	8
(g)(7)(ii)(A)(2) - Implementation Requirements	8
(g)(7)(ii)(A)(3) - Terms of Use	8
(g)(7)(ii)(B) - Public Link	8
"Application access – data category request" (§170.315(g)(8))	(g)(8)(i)(A) - Return CCDS data	20
(g)(8)(i)(B) - Request response	20
(g)(8)(ii)(A)(1) - Documentation	8
(g)(8)(ii)(A)(2) - Implementation Requirements	8
(g)(8)(ii)(A)(3) - Terms of Use	8
(g)(8)(ii)(B) - Public URL	8
"Application access—all data request" (§170.315(g)(9))	(g)(9)(i)(A)(1) - Demonstrate API	20
(g)(9)(i)(A)(3) - Data Classes	20
(g)(9)(i)(B) - Data Return	20
(g)(9)(ii)(A)(i) - Documentation	8
(g)(9)(ii)(A)(ii) - Implementation Requirements	8
(g)(9)(ii)(B) - Public URL	8
"Direct Project" (§170.315(h)(1))	(h)(1)(i) - Send	12
(h)(1)(i) - Receive	13
(h)(1)(ii) - Message Disposition Notification: Processed	12
(h)(1)(ii) - Message Disposition Notification: Failed	12

Justification for Real World Testing Approach

WebChart EHR is a cloud-based, fully-inclusive EHR solution. All certified functionality is delivered in all instances of the product regardless of the care setting, size of practice, or required use cases for a given practice. Each production client is maintained in a separate database; however, the implementation of the environment is identical with the exception of optional increased security protocols that a client may choose to add for enhanced data protection. Additionally, the only differences between the client-facing portion of each system are a result of configuration settings that can be selected at go-live or updated at any time during a client’s contract. Due to this philosophy of product delivery, all certified capabilities may not be actively used in all marketed care settings or may not be actively used in any current client production system. To address the Real World Testing requirements, MIE will be using a hybrid approach. Testing will primarily be conducted using de-identified real patient data from production systems as recorded in database tables and log files. For those criteria for which this live production recording is not available or minimal due to lack of client usage, client reported issues will be tracked and reported in addition to enacting automated tests of the certified functionality in a test system in a production environment. The automated tests will be run daily or weekly as appropriate in a system that is identical in substance and delivery to a client production system with the only exception being live real patient data. This blended approach will allow MIE to prove ongoing maintenance of WebChart EHR’s certified technology regardless of the level of implementation by current clients.

Measures Used in Overall Approach

The following measures outline and justify how each requirement of all criteria to which WebChart EHR is certified will be tested during the 2022 Real World Testing year. Please review the Criteria-Measure Matrix above to review which measure(s) will cover a specific requirement.

Measure 1: Clinical Quality Measures Outgoing

Description

This measure will review WebChart EHR’s ability to measure clinical quality and export the required information. Compliance will be tested both manually by developers and clients as well as automatically by reporting bodies and the Cypress CUV+ test system.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"CQMs – record and export" (§170.315(c)(1))	(c )(1)(i) - Record
(c )(1)(ii) - Export
"CQMs – import and calculate" (§170.315(c)(2))	(c )(2)(ii) - Calculate
"CQMs – report" (§170.315(c)(3))	(c )(3)(i) - Report

Test Methodology

First, MIE will install an instance of Cypress 6+ on our production servers following all of our protocols for maintaining the security of PHI. Cypress CUV+ supports the validation of QRDA reports containing PHI and will be used monthly to validate a random selection of QRDAs from the care settings identified. Any errors identified by Cypress CUV+ will be tracked, reported, and addressed, then followed with testing of a larger sample of files.
Additionally, WebChart EHR has two customers that participate in quarterly attestations using both QRDA I and QRDA III reports. These customers regularly inspect their CQM compliance numbers and will alert MIE to any perceived errors. MIE will then collect and track the attestation results from the reporting bodies including any errors so as to report a success/failure rate.

Results

	Calculations	QRDA I	QRDA III
Client Reported Issues	0	0	0
Submitted Files	N/A	2421	0
Submission Errors	N/A	0	0
Tested Files	N/A	687	12
Testing Errors	N/A	0	0

Discussion

As expected, no errors were found in formatting or coding of the certified measures.

Measure 2: Clinical Quality Measures Incoming

Description

This measure will review WebChart EHR’s ability to measure clinical quality and export the required information. Compliance will be tested both manually by developers and clients as well as automatically by reporting bodies and the Cypress CUV+ test system.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"CQMs – import and calculate" (§170.315(c)(2))	(c )(2)(i) - Import
(c )(2)(ii) - Calculate
"CQMs – report" (§170.315(c)(3))	(c )(3)(i) - Report

Test Methodology

MIE will install an instance of Cypress 6+ on our production servers following all of our protocols for maintaining the security of PHI. Automated testing will download QRDA I files from Cypress for each certified CQM, import the files to WebChart EHR, calculate the CQMs, and export the QRDA files for Cypress validation of both the content and calculations to verify that the import was successful. Any errors identified by Cypress will be tracked, reported, and addressed.

Results

	QRDA I	QRDA III
Tested Files	702	12
Testing Errors	0	0

Discussion

As expected, no errors were found in formatting or coding of the certified measures.

Measure 3: E-Prescribing Messages Sent and Received

Description

This measure will verify that all supported e-prescribing message types are in use in WebChart EHR, including inbound and outbound message types.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Electronic prescribing" (§170.315(b)(3))	(b)(3)(ii)(A)(1) - NewRx
(b)(3)(ii)(A)(2) - RxChangeRequest, RxChangeResponse
(b)(3)(ii)(A)(3) - CancelRx, CancelRxResponse
(b)(3)(ii)(A)(4) - RxRenewalRequest, RxRenewalResponse
(b)(3)(ii)(A)(5) - RxFill
(b)(3)(ii)(A)(6) - RxHistoryRequest, RxHistoryResponse
(b)(3)(ii)(A)(7) - Status
(b)(3)(ii)(A)(8) - Error
(b)(3)(ii)(A)(9) - Verify

Test Methodology

MIE will report a count of messages for each supported message type:

• NewRx
• RxChangeRequest
• RxChangeResponse
• CancelRx
• CancelRxResponse
• RxRenewalRequest
• RxRenewalResponse
• RxFill
• RxHistoryRequest
• RxHistoryResponse
• Status
• Error
• Verify The report will also include a count of outbound messages unable to be transmitted due to connectivity issues or other errors, for each message type. This report will be based on the contents of each client’s local database table of stored messages. MIE will run the report for each client under consideration and aggregate the results.

Results

Message Type	Client Message Counts	Total Count
NewRx	12347	56042	29975	8466	2408	109238
RxChangeRequest	6	114	702	188	0	1010
RxChangeResponse	0	113	665	188	0	966
CancelRx	58	61	0	6	0	125
CancelRxReponse	49	56	0	5	0	110
RxRenewalRequest	3154	2959	14384	1575	0	22072
RxRenewalResponse	3163	2967	13587	1576	0	21293
RxFill	0	0	0	361	0	361
RxHistoryRequest	3	0	6		0	9
RxHistoryResponse	2	0	3		0	5
Status	31644	108405	95660	20594	4466	260769
Error	211	193	875	82	2	1363
Verify	12808	46168	36182	8270	2060	105488

Error Text	Total Count
(Error Code: 601) (Desc. Code: 008)	90
(Error Code: 600)	1
(Error Code: 601) (Desc. Code: 008)	369
(Error Code: 900)	28
1 errors validating against transport.xsd	129
Cannot get endpoint	5
Cannot get portal (id = 422345), please contact support (Error Code: 602)	1
Controlled Substance Cannot Be Faxed (Error Code: 900)	1
Controlled substance must have a DEASchedule populated in Medication Prescribed (Error Code: 900)	10
DEA or SSN not found in the message (Error Code: 900)	24
Duplicate Prescription, original message has been processed (Error Code: 900) (Desc. Code: 220)	1
ERROR FORWARDING XML MESSAGE (Error Code: 900)	13
Fax receipt could not be confirmed (Error Code: 600)	2
Internal error.Contact pharmacy by other means. (Error Code: 600) (Desc. Code: 008)	7
Message is a duplicate.	110
NewRx MedicationPrescribed Quantity must be a non-zero value (Error Code: 900)	40
No NDC or compound or supply code in message for MedicationPrescribed (Error Code: 900)	8
Partner rejected transaction with an error (Error Code: 900) (Desc. Code: 220)	229
Pharmacy does not dispense class I or class II controlled substances (Error Code: 900)	3
Pharmacy is Temporarily Unavailable (Error Code: 900)	3
Pharmacy is temporarily down. Please retry. (Error Code: 601)	1
Pharmacy record inactive, may have new active location (Error Code: 900)	1
Prescriber address required for controlled substance (Error Code: 900)	1
PrescriberOrderNumber must be present and populated (Error Code: 900)	12
Prescription no longer active (Error Code: 601)	115
Receiving partner does not support this type of message (Error Code: 900)	17
Refill request has been canceled (Error Code: 900) (Desc. Code: 4020)	12
Request has been canceled prior to response. Response not processed. (Error Code: 601) (Desc. Code: 008)	80
This pharmacy dispenses oxybate products only. Please select different pharmacy (Error Code: 900)	2
TrackingID service error (Error Code: 900)	1
Transaction is a duplicate. (Error Code: 601) (Desc. Code: 220)	1
UIB Trace Number is Invalid (Error Code: 900)	1
Unable To Process (Error Code: 601) (Desc. Code: 210)	14
Unexpected Exception occurred while processing the request. (Error Code: 900)	19

Total error rate: 1351/522809*100 = 0.26%

Discussion

As expected, all supported message types have a greater than zero total message count, and the total number of messages far exceeds the number of errored messages with a total error rate of 0.26%. This represents an increase in both the total number or errored messages and overall error rate from Q2.

Measure 4: E-Prescribing Diagnosis Codes

Description

This measure will verify that all diagnosis elements are present in some e-prescribing messages as required by §170.315(b)(3), including inbound and outbound message types.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Electronic prescribing" (§170.315(b)(3))	(b)(3)(ii)(C)(1) - Primary/Secondary Diagnosis

Test Methodology

MIE will report the contents of each stored message in a client’s local database table of stored messages, and counts the inbound and outbound messages that include Diagnosis elements. MIE will run the report for each client under consideration and aggregate the results.

Results

Total NewRx Messages	NewRx Messages with Diagnosis Included
109238	32390

Discussion

Since the Diagnosis elements are not a required component of a NewRx message, as anticipated, only a subset (29.65%) of the NewRx messages included a diagnosis; however, this is an increase in diagnosis usage over the 28.61% of Q2.

Measure 5: E-Prescribing Oral Liquid Units

Description

This measure will verify that prescriptions for medications with an oral liquid form will have a quantity unit of measurement of mL, not cc or English units as outlined in §170.315(b)(3).

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Electronic prescribing" (§170.315(b)(3))	(b)(3)(ii)(E) - Metric Units

Test Methodology

MIE will create a system report that examines the contents of each stored NewRx message in a client’s local database table of stored messages, limiting to oral liquid medications, and provides a count of each distinct quantity unit of measure used. MIE will run the report for each client under consideration and aggregate the results.

Results

Unit Code	Unit Description	Quantity
C28254	Milliliter (ml)	3439
C64933	Each	0
C48477	Bottle	1
C48521	Packet	0
C48542	Tablet	0

[Units of Presentation](https://www.fda.gov/industry/structured-product-labeling-resources/units-presentation)

Discussion

As expected, C28254 (milliliters) is the most commonly sent unit of measure for oral liquid medications. Non-C28254 units were only sent in 0.03% of oral liquid medication messages. This represents a decrease in incorrect code usage from Q2.

Measure 6: E-Prescribing Decimal Format

Description

This measure will verify that numeric amounts in prescriptions include leading zeros before decimal points and do not allow trailing zeros after a decimal point.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Electronic prescribing" (§170.315(b)(3))	(b)(3)(ii)(F) - Decimal Format

Test Methodology

MIE will create a system report that examines the contents of each stored NewRx message in a client’s local database table of stored messages, and provides a count of prescription messages that include inappropriate trailing zeros, and a count of those missing leading zeros. MIE will run the report for each client under consideration and aggregate the results.

Results

Total NewRx Messages	NewRx Messages with Improper Decimal Format
109238	530

Discussion

As expected, the number of NewRx messages sent with inappropriate trailing zeros, or missing leading zeros, occurs rarely in only 0.49% of messages; however, this represents an increase from Q2 which indicates the need for either client education or coded validation in the software.

Measure 7: CDA Download

Description

This measure will verify that the system can accept a CDA document uploaded into the system, assign it to the appropriate chart in the system as appropriate, and display the document with a standard stylesheet with all sections being accepted and visible.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Clinical information reconciliation and incorporation" (§ 170.315(b)(2))	(b)(2)(ii) - Correct patient.
‘‘Transitions of care" (§ 170.315(b)(1))	(b)(1)(ii) - All paragraphs
(b)(1)(iii) - All paragraphs

Test Methodology

MIE will report on the number of CDA formatted documents uploaded into tracked Webchart systems and the number of upload attempts that failed as stored in client databases and error log files.
MIE will report on the number of requests to view a CDA document within the system, and the number of times it displayed correctly, and when there were errors in display.
Any errors reported by customers or the recipients of their quarterly attestations will be tracked and reported as a baseline. These test assumptions for customer reporting align with the “visual inspection” aspects of the test lab tests.

Results

Total CDA Documents Uploaded	14,506
Total CDA Document Upload Failures	0
Total CDA Document Views	1,904
Total CDA Document View Reported Errors	0
Total Successful CDA Views	1,904

Discussion

As anticipated, there were no errors in being able to upload and view valid CCDA documents. The number of CDA documents has increased dramatically from Q2 to Q3, mainly from one client. The number of views more than doubled, because of that client’s upload increase, although most CDA documents generated were not being subsequently viewed.

Measure 8: Application Access Documentation

Description

This measure will verify that WebChart EHR’s API documentation is publicly and perpetually available. Compliance will be recorded by an external uptime monitor and reported quarterly. Upon request, or in the event of downtime, data can additionally be reported in daily, weekly, or monthly increments.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Application access – patient selection" (§ 170.315(g)(7))	(g)(7)(ii)(A)(1) - Functional Documentation
(g)(7)(ii)(A)(2) - Implementation Requirements
(g)(7)(ii)(A)(3) - Terms of Use
(g)(7)(ii)(B) - Public Link
"Application access – data category request" (§ 170.315(g)(8))	(g)(8)(ii)(A)(1) - Documentation
(g)(8)(ii)(A)(2) - Implementation Requirements
(g)(8)(ii)(A)(3) - Terms of Use
(g)(8)(ii)(B) - Public URL
"Application access—all data request" (§ 170.315(g)(9))	(g)(9)(ii)(A)(i) - Documentation
(g)(9)(ii)(A)(ii) - Implementation Requirements
(g)(9)(ii)(B) - Public URL

Test Methodology

An external uptime monitor will check the availability of all documentation available at https://docs.webchartnow.com/resources/system-specifications/application-programming-interface-api.html . Both up- and downtime will be logged to be reported quarterly. The cause of any downtime and the duration will also be logged In the event of any downtime, the amount of downtime can be reported at daily, weekly, or monthly intervals in addition to the quarterly reports, and the cause of each downtime occurrence will be reported.

Results

The MIE API documentation was available 99.902% of Q3. There were several short down-time events during July averaging 17.29 minutes per event all due to Cloudflare timeouts.

Discussion

As expected, the documentation maintained an uptime of greater than 99.9% at 99.902% for the quarter. Three of the seven down-time events were longer than the acceptable 10 minute duration at 81 minutes on 07/12/2022, 12 minutes on 07/15/2022, and 11 minutes on 07/16/2022. Since all down-time events were caused by Cloudflare timeouts, the root cause of this will be investigated to mitigate future extended down-time events.

Measure 9: Clinical Information Reconciliation and Incorporation

Description

This measure will verify that the system can take a CCDA transition of care/referral summary formatted according to the standards adopted §170.205(a)(3) and §170.205(a)(4) and read the data for medications, allergies, and conditions from the document, reconcile those into the chart, and that the data is fully incorporated into the chart.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Clinical information reconciliation and incorporation" (§ 170.315(b)(2))	(b)(2)(iii)(A), (B), (C), (D)
(b)(2)(iv) - System Verification

Test Methodology

MIE will report on the number of CDA formatted documents reconciled via the “Froozhie” tool.
Following each reconcile, a temporary CDA for the chart will be created and validated to ensure the reconciled data can be incorporated into a CDA created free of schematic errors (the CDA document will NOT be kept, only the result of the validation). Additionally, any client complaints that data is not being imported correctly from the tool will be tracked, investigated, and reported

Results

CDA Documents Reconciled	368
CDA Documents Generated on a Chart after a Reconcile that had an invalid Validation	23
Client reported Issues with CDA reconciliation	0

Discussion

The use of CDA reconciliation in systems was relatively low, and no issues with the reconciliation of the documents were reported.

The number of invalid validations were higher than anticipated, but lower than Q2 (6.2% from 11.1%). Work is being done to rectify the remaining issues with the CDA generation in these systems, and to put in place a more immediate notification process when invalid CDAs are generated.

Measure 10: Transmission to Immunization Registry: Create Content

Description

This measure will verify that the system can generate a VXU conforming to the HL7 v2.5.1 standard, CDC guidance for communication to Immunization Registries and state/local guidance. The VXU messages shall contain information related to the demographics and vaccination administration record.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Transmission to immunization registries" (§ 170.315(f)(1))	(f)(1)(i) - Create Content

Test Methodology

MIE will report from the database the number of successfully sent VXU messages acknowledged as received by the state immunization registry. MIE will also report from the database on the number of records rejected by the state registry due to error, whether the failure was due to registry internal errors, clinical data entry issues or a not well-formed message. Finally, MIE will report from the database the number of messages which declined to be generated due to data entry issues failing message pre-validation.

Results

Successful transmission to registry	2250
Rejected by the registry	68
Declined to generate	5

Discussion

As expected, very few messages were rejected by the state registry. There is a small portion of the entered injections which were not able to be transmitted to the state registry due to user data entry issues.

Measure 11: Transmission to Immunization Registries: Query Records

Description

This measure will verify that the system can generate a QBP conforming to the HL7 v2.5.1 standard, CDC guidance for communication to Immunization Registries and state/local guidance. Furthermore, the system shall be able to retrieve, consume and display to the end user the results of any such query.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Transmission to immunization registries" (§ 170.315(f)(1))	f)(1)(ii) - Query Records

Test Methodology

MIE will report the number of successful retrievals of evaluated history and forecasting operations from the database. MIE will report the number of failed retrievals, including those resulting from an internal error in the registry resulting in an inability to consume a response from the database. MIE will manually track, resolve and report issues resulting from WebChart EHR application errors as reported by end users.

Results

Successful retrievals	8008
Failed retrievals	113

Discussion

The overall error rate increased slightly between Q2 and Q3, increasing from 1% to 1.4%; however, this still represents a decrease in overall error rate from 4.2% in Q1.

Measure 12: Direct Project: Send

Description

This measure will verify that the system can transmit a Direct project conforming S/MIME to a HISP. The measure will also verify the receipt of those transmissions by verifying the status of the resultant MDN messages.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Direct Project" (§ 170.315(h)(1))	(h)(1)(i) - Send
(h)(1)(ii) - Message Disposition Notification: Processed
(h)(1)(ii) - Message Disposition Notification: Failed

Test Methodology

MIE will report from log files the number of messages transmitted. MIE will report from logs the number of messages which failed to be transmitted whether due to internal error, external failures or inability to verify trust of the recipient. MIE will report from logs the number of Processed MDN messages received. MIE will report from logs the number of Failed MDN messages received.

Results

Messages Transmitted	7
Failed to transmit	0
Failed MDNs received	0

Discussion

While very few outgoing messages were processed, all messages were successful.

Measure 13: Direct Project: Receive

Description

This measure will verify that the system conforms to Direct Project message receipt requirements for validation.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Direct Project" (§ 170.315(h)(1))	(h)(1)(i) - Receive

Test Methodology

MIE will report from logs the number of messages transmitted to the HISP. MIE will report from logs the number of messages failing to conform to Direct Project specifications. MIE will report from logs the number of messages which are successfully delivered to recipients.

Results

Messages received by HISP	4
Messages failing to conform	0
Messages successfully delivered	4

Discussion

As expected, the received messages were successfully processed and delivered.

Measure 14: Patient Portal View

Description

This measure will verify that a patient can view various document types within the patient portal.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"View, download, and transmit to 3rd party" (§ 170.315(e)(1))	(e)(1)(i)(A)(1),(2),(3),(4),(5)
(e)(1)(i)(D)(1), (2)
(e)(1)(ii)(A)

Test Methodology

MIE will report a number of measurements surrounding documents, including:

• Number of documents sent successfully to patients in the patient portal
• Number of documents unsuccessfully sent to the patient portal
• Number of documents read by patients in the patient portal
• Number of failures in the ability to read messages in the patient portal Results will be retrieved from database tables and aggregated for reporting. Any failures will be reported from the information found in log files as well as any client reported issues tracked during the testing period.

Results

CDA Documents Sent to the patient portal	0
CDA Documents Unsuccessfully Sent to the patient portal	0
CDA Documents Read by Patients in the Patient Portal	0
CDA Documents Failed to be read in the Patient Portal	0

Discussion

During the third quarter, no tracked clients were sending CDA documents into the patient portal. Additional testing as well as client education will be investigated to provide more robust data in future quarters.

Measure 15: Patient Portal Download

Description

This measure will verify that a patient can download various document types within the patient portal.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"View, download, and transmit to 3rd party" (§ 170.315(e)(1))	(e)(1)(i)(B)(1), (2), (3)
(e)(1)(i)(D)(1), (2)
(e)(1)(ii)(A)

Test Methodology

MIE will report a number of measurements surrounding documents, including:

• Number of documents sent successfully to patients in the patient portal
• Number of documents unsuccessfully sent to the patient portal
• Number of documents successfully downloaded from the patient portal
• Number of documents unsuccessful in being downloaded from the patient portal. Results will be retrieved from database tables and aggregated for reporting. Any failures will be reported from the information found in log files and third party reports as well as any client reported issues tracked during the testing period.

Results

CDA Documents Sent to the patient portal	0
CDA Documents Unsuccessfully Sent to the patient portal	0
CDA Documents Downloaded by Patients in the Patient Portal	0
CDA Documents Failed to be downloaded in the Patient Portal	0

Discussion

During the third quarter, no tracked clients were sending CDA documents into the patient portal. Additional testing as well as client education will be investigated to provide more robust data in future quarters.

Measure 16: Patient Portal CCDA Transmit

Description

This measure will verify that a patient can transmit various document types within the patient portal to other entities.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"View, download, and transmit to 3rd party" (§ 170.315(e)(1))	(e)(1)(i)(C)(1), (2)
(e)(1)(i)(D)(1), (2)
(e)(1)(ii)(A)

Test Methodology

MIE will report a number of measurements surrounding documents, including:

• Number of documents sent successfully to patients in the patient portal
• Number of documents unsuccessfully sent to the patient portal
• Number of documents successfully transmitted from the patient portal
• Number of documents unsuccessful in being transmitted from the patient portal. Results will be retrieved from database tables and aggregated for reporting. Any failures will be reported from the information found in log files and third party reports as well as any client reported issues tracked during the testing period.

Results

CDA Documents Sent to the patient portal	0
CDA Documents Unsuccessfully Sent to the patient portal	0
CDA Documents Sent by Patients in the Patient Portal	0
CDA Documents Failed to be sent in the Patient Portal	0

Discussion

During the third quarter, no tracked clients were sending CDA documents into the patient portal. Additional testing as well as client education will be investigated to provide more robust data in future quarters.

Measure 17: Send Using Edge Protocol for SMTP / XDM

Description

This measure will verify that the system is able to utilize a SMTP edge protocol for sending and receiving Direct Project messages. As part of receiving messages, XDM shall be handled when applicable.

Associated Certification Criteria

Certification Criteria	Requirement(s)
‘‘Transitions of care" (§ 170.315(b)(1))	(b)(1)(i)(A)(Alternative) - Send Using Edge Protocol for SMTP/IXE XDR
(b)(1)(i)(B)(Alternative) - Receive Using Edge Protocol for SMTP/IXE XDR
(b)(1)(i)(C)(Conditional) - XDM Processing

Test Methodology

MIE will report from logs the number of messages transmitted to the HISP by SMTP. MIE will report from logs the number of messages received from the HISP by SMTP. MIE will report from logs the number of XDM packages processed. In the case where insufficient real-world data is available, data resulting from regular testing with DirectTrust shall be included in the reporting.

Results

Messages submitted to HISP via SMTP	7
Number of XDM packages processed	54

Discussion

As expected, there was very low usage. XDM packages that were processed included packages from standard regular testing.

Measure 18: Data Export

Description

This measure will verify that a user can use WebChart EHR’s Data Export Tool to pull down groups of patient data from a Webchart EHR system.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Data export" (§ 170.315(b)(6))	(b)(6)(i)
(b)(6)(ii)(A)-(F)
(b)(6)(iii)(A)-(B)
(b)(6)(iv)

Test Methodology

MIE will report from the event log database tables a series of occurrences that indicates use of the WebChart EHR Data Export Tool:

• Event logs of the report to find all patients for Document Export being called.
• Event logs of CDA documents being generated within a certain short time period following the report. MIE will track customer reports of data expected to be in mass data export downloads that did not download as failures.

Results

Number of times EHR Data Export Tool was used: 0

Discussion

During the third quarter, no clients used the EHR Data Export Tool. Additional testing will be investigated to provide more robust data in future quarters.

Measure 19: CDA Validation

Description

This measure will verify that CDAs both created by and received by a Webchart EHR system pass basic CDA validation.

Associated Certification Criteria

Certification Criteria	Requirement(s)
‘‘Transitions of care" (§ 170.315(b)(1) Cures Update)	(b)(1)(ii)(A)
"Data export" (§ 170.315(b)(6))	(b)(6)(ii), (A)-(F)

Test Methodology

All CDAs stored within a Webchart EHR will be run through schema validation regardless of the document’s origin. Documents may originate within the WebChart EHR system or be imported from a third party application of manual upload. The schema validator will be installed within the MIE production environment to ensure the security of all PHI contained in the documents. Only results of the validation will be made available, document content will not be revealed to developers during testing.

The number of valid vs. invalid CDAs and their sources will be reported.

Results

All CDAs

CDA Documents Marked Valid in Period	171
CDA Documents Marked Invalid in Period	540
% Marked Invalid In Period	76

Webchart Generated CDAs Only

CDA Documents Marked Valid in Period	4
CDA Documents Marked Invalid in Period	19
% Marked Invalid In Period	82

Discussion

The percentage of CCDAs marked invalid was extremely high this period. As stated in Measure 9, “Work is being done to rectify the issues with the CDA generation in these systems, and to put in place a more immediate notification process when invalid CDAs are generated.”

Most of these documents were also NOT created by WebChart EHR systems, but came in from third party providers. Where possible, these third parties will be informed of issues with their CCDAs.

WebChart EHR generated documents with errors were determined to have an issue with a performer element getting generated in the wrong spot, this is currently being rectified. Overall these issues would not inhibit usage of the CCDA by another system.

WebChart EHR Generated CDAs were not evaluated directly in Q1 or Q2, but it suspected that the majority of CDAs were again coming from third parties.

Measure 20: Patient Data requests VIA API

Description

This measure will verify that the API as outlined in WebChart EHR’s documentation is functional. A valid request for patient information must provide that information.

Associated Certification Criteria

Certification Criteria	Requirement(s)
"Application access – patient selection" (§ 170.315(g)(7))	(g)(7)(i) - Query processing and response
"Application access – data category request" (§ 170.315(g)(8))	(g)(8)(i)(A) - Return CCDS data
(g)(8)(i)(B) - Request response
"Application access—all data request" (§ 170.315(g)(9))	(g)(9)(i)(A)(1) - Demonstrate API
(g)(9)(i)(A)(3) - Data Classes
(g)(9)(i)(B) - Data Return

Test Methodology

To address the overall automated testing, the following test requests will be made daily against a test system in a production environment.

• Issue a request in the browser to search for a patient (patient selection)
• Issue a request in the browser to request demographics of a patient (data category request)
• Issue a request using the export tool described in the documentation. All API requests made in production systems are recorded in log files. The number of requests logged will be reported against the number of issues with API functionality that are reported.

Results

Production Exports	1
Total Charts Exported	43436
Total Export Errors	5

Discussion

As expected, the production use of the API was rare, but the total number of export errors was minimal. Patient chart exports were successful in 99.99% of cases.

Measure 21: Web Content Accessibility

Description

This measure will verify that all certified content in the patient portal will maintain accessibility conformance as outlined in the Web Content Accessibility Guidelines (WCAG) 2.0.

Associated Certification Criteria

Certification Criteria	Requirement(s)
	(e)(1)(i) - Web Content Accessibility

Test Methodology

MIE will conduct monthly third-party production accessibility scanning as well as automated nightly internal accessibility scanning of a test system in a production environment.

Results

The internal accessibility scanning of a pre-production test system identified 0 urgent and 0 secondary non-conformance issues in 100% of nightly and ad-hoc scans. Production accessibility scanning identified 0 urgent and 0 secondary non-conformance issues for the entire quarter.

Discussion

As expected no urgent non-conformance issues were identified in either live production or pre-production test systems. Also, this quarter, no secondary non-conformance issues were identified.

Schedule of Key Milestones

Key Milestone	Care Setting	Date/Timeframe	Status
Release of documentation for the Real World Testing to be provided to ACB and providers	All settings	Nov 15, 2021	Complete
Begin collection of information as laid out by the plan	All settings	Jan 1, 2022	Complete and ongoing
New certification of b.7, b.8, b.9	All settings	Q1, 2022	Complete
Certification of additional CQMs to c.1, c.2, c.3 and QRDA SVAP	All settings	Q1, 2022	Scheduled October
Attest to Cures update version of b.2, e.1	All settings	Q1, 2022	Complete
Follow-up with providers and authorized representatives to understand any issues arising with the data collection.	All settings	Quarterly, 2022	Q1 and Q2 complete
Data collection and review.	All settings	Quarterly, 2022	Q1 and Q2 complete
New certification of b.10, g.10	All settings	Q3, 2022	G.10 complete B.10 planned for 2023
End of Real World Testing period/final collection of all data for analysis	All settings	Dec 31, 2022
Data analysis and report creation	All settings	January, 2023
Submission of Real World Testing Results to ACB	All settings	Feb 1, 2023

Attestation

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.

Authorized Representative Name	Doug Horner
Authorized Representative Email	horner@mieweb.com
Authorized Representative Phone	260-459-6270
Authorized Representative Signature
Date	10/15/2022