Radiotherapy and Oncology 103 (2012) 279–286
Contents lists available at SciVerse ScienceDirect
Radiotherapy and Oncology journal homepage: www.thegreenjournal.com
Quality assurance
EORTC Radiation Oncology Group quality assurance platform: Establishment of a digital central review facility Alysa Fairchild a,b,⇑, Edwin Aird c, Paul A. Fenton d, Vincent Gregoire e, Akos Gulyban f, Denis Lacombe a, Oscar Matzinger g, Philip Poortmans h, Pascal Ruyskart a, Damien C. Weber i, Coen W. Hurkmans j a
EORTC Headquarters, Brussels, Belgium; b Cross Cancer Institute, Edmonton, Canada; c Mount Vernon Hospital, Northwood Middlesex; d Southampton University Hospital NHS Trust, UK; e Université catholique de Louvain,Brussels; f University Hospital Ghent, Belgium; g Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland; h Institute Verbeeten, Tilburg, The Netherlands; i Geneva University Hospital, Switzerland; j Catharina Hospital, Eindhoven, The Netherlands
a r t i c l e
i n f o
Article history: Received 23 October 2011 Received in revised form 10 April 2012 Accepted 24 April 2012 Available online 23 May 2012 Keywords: Quality assurance Radiotherapy Clinical trial Platform Central review
a b s t r a c t Objective: Quality assurance (QA) in clinical trials is essential to ensure treatment is safely and effectively delivered. As QA requirements have increased in complexity in parallel with evolution of radiation therapy (RT) delivery, a need to facilitate digital data exchange emerged. Our objective is to present the platform developed for the integration and standardization of QART activities across all EORTC trials involving RT. Methods: The following essential requirements were identified: secure and easy access without on-site software installation; integration within the existing EORTC clinical remote data capture system; and the ability to both customize the platform to specific studies and adapt to future needs. After retrospective testing within several clinical trials, the platform was introduced in phases to participating sites and QART study reviewers. Results: The resulting QA platform, integrating RT analysis software installed at EORTC Headquarters, permits timely, secure, and fully digital central DICOM-RT based data review. Participating sites submit data through a standard secure upload webpage. Supplemental information is submitted in parallel through web-based forms. An internal quality check by the QART office verifies data consistency, formatting, and anonymization. QART reviewers have remote access through a terminal server. Reviewers evaluate submissions for protocol compliance through an online evaluation matrix. Comments are collected by the coordinating centre and institutions are informed of the results. Conclusions: This web-based central review platform facilitates rapid, extensive, and prospective QART review. This reduces the risk that trial outcomes are compromised through inadequate radiotherapy and facilitates correlation of results with clinical outcomes. Ó 2012 Elsevier Ireland Ltd. All rights reserved. Radiotherapy and Oncology 103 (2012) 279–286
Quality assurance (QA) in multicentre international clinical trials is essential to ensure that radiation therapy (RT) is safely and effectively delivered. The QA strategy of the EORTC Radiation Oncology Group (ROG) has been in place since 1982 [1]. The pioneers of QA within the ROG established the culture and basic principles of clinical trial QART in Europe. Over the past 30 years, attempts to ensure quality RT delivery have included site visits, evaluation of institutions’ staff and infrastructure, dosimetric checks of treatment units, cross-check of patient charts and portal images, radiobiological modelling of inter-institutional differences, and mailed thermoluminescent dosimetry audits [2]. Efforts have consistently reflected what is considered state-of-the-art given available human and financial resources.
⇑ Corresponding author. E-mail address:
[email protected] (A. Fairchild). 0167-8140/$ - see front matter Ó 2012 Elsevier Ireland Ltd. All rights reserved. http://dx.doi.org/10.1016/j.radonc.2012.04.015
In 2006, five levels of QA were defined at the EORTC, including specific QART requirements for participation in ROG trials [2]. In the dummy run (DR), investigators are provided with relevant clinical and radiologic information on an index patient, and are asked to treat the case as if it were entered into that trial protocol. The ability to appropriately contour target volumes and produce a compliant treatment plan is evaluated by trial QA reviewers. The individual case review (ICR) confirms protocol compliance with RT parameters for specific patients including target delineation, beam configuration and dose verification. To maximize the validity of the review, the full RT dataset should be evaluated, often including diagnostic images on which the RT plan is based. ICR datasets are evaluated by members of the trial QA committee, either retrospectively or prospectively (in relation to timing of RT delivery). DR and ICR datasets are evaluated as acceptable, minor or major deviation, and a customized report is delivered to the submitting institution. Major deviations trigger compulsory DR resubmission and in the case of
280
EORTC ROG QART platform
the ICR, another submission from that institution may be requested. Ideally, reviewer feedback from these procedures is applied to subsequent protocol patients’ RT treatment. As recently as 5 years ago, DR and ICR evaluation relied mainly on paper documents and hard copy films; consequentially, limited data were available [3]. In the absence of a full RT plan, sites were asked to complete an additional case report form (CRF) reporting a limited number of dose levels or points per volume of interest. Documentation was either mailed between EORTC Headquarters (HQ), the submitting institution and the central reviewers, or evaluated twice per year at ROG meetings. However, as QA requirements have increased in complexity in parallel with evolution of RT delivery techniques [4], a pressing need to facilitate digital data exchange and timely review has emerged [5]. Over the past decade, efforts have been made to address this by the EORTC HQ and ROG. The transition from hardcopy to digital data transfer carries significant advantages, allowing comprehensive prospective collection of RT data and real-time approaches to digital evaluation of ICRs. Electronic review can also display inconsistencies that would be difficult to identify with paper or screenshots [6]. Short timelines required by rapid case reviews, in which a patient’s RT is not permitted to begin until central approval is obtained, are increasingly achievable with digital data transfer. Along with the increase in efficiency, financial savings are realized in terms of mailing and storage costs, by largely obviating site visits and in-person ROG meeting attendance. Digital review is faster than assessment of equivalent data extracted from hard copies, minimizing what would otherwise be considerable costs for reviewers’ time. Our objective is to present the platform developed for the integration and standardization of QART activities across the EORTC as a result of a joint effort between the EORTC HQ and ROG. Methods General requirements for the acquisition of a platform were enumerated by the ROG QART Strategic Committee and HQ QART office based on the need to integrate it into the existing EORTC HQ clinical remote data capture (RDC) system and the requirement to store data for future outcome comparisons, using lessons learned from collaboration with American QA offices and accumulated experience with case review in the paper era [3,7–16]. The essential tasks to be enabled include: data submission from participating institutions to the HQ QART office; HQ QART office performance of digital data integrity QA (DDIQA); and evaluation by QA reviewers in geographically separate locations. Submission from centres to QART office Since the EORTC ROG has member sites in over 20 countries, local installation of QA-related software for data submission is not possible at each. Even if restricted to home institutions of central reviewers, this is not feasible as up to six different reviewers participate in each clinical trial. The platform must therefore be accessible from anywhere, with no dedicated local hardware or software needed. A malware check should be built into the method of transfer to ensure data are clean before being accepted by HQ. A record of date and time of data receipt is required. For troubleshooting, the contact details of the person submitting the dataset should be included, as this may not be the local principal investigator, or the radiation oncologist or clinical physicist responsible for the given case. QART office DDIQA It is an advantage that site investigators perform QART procedures with their own treatment planning software (TPS). This requires, however, that the central QART office is prepared to
receive data from diverse imaging and planning systems [5]. TPSspecific guidelines on how to select and upload correct data formats have been developed for the ROG website. DDIQA should correct remaining issues regarding readability of data and transformation of image coordinates [17]. A check of consistency, formatting, and anonymization is performed to confirm the dataset is complete before releasing it for QA review. A consistent method of de-identification is needed to ensure RT plans can be matched with the correct patient in the clinical RDC database. Clinical CRFs are usually submitted at a different time by different personnel who may not be familiar with RT details, so a match on the basis of sequential ID number, date of birth and patient code is required. This also ensures an ability to query a growing RT plan database.
Central evaluation The RT dataset needs to be accessed by the reviewers, who also receive appropriate clinical information and diagnostic images in the case of an ICR. They then complete a formal, standardized evaluation matrix, which is communicated to EORTC HQ. If central review is performed by more than one clinician (commonly one or two radiation oncologists and one clinical physicist per case), consensus must be reached before feedback can be sent to participating institutions. Dedicated central coordination is required to monitor timelines and produce summary reports for trial committees, the ROG and EORTC HQ. Results The essential utility of an integrated modular central review platform is the exchange and review of the entire digital RT dataset (DICOM images, RT-Structure, RT-Dose and RT-plan) (Table 1). Defining different levels of access limits the availability of system contents and functionality depending on anticipated tasks of each user. Work flow within the platform is summarized in Fig. 1. Participating institutions upload requested data through a javabased standard secure upload webpage, which is operating system-independent (Fig. 2). The same technology is used by the EORTC central randomization system, so technical requirements are easily met by sites presently registering patients. While institutional firewalls may hamper data transfer, this is usually rapidly resolved by local IT support. The dataset is time-stamped and transferred to a designated folder. Once transferred, a trial-specific automatic notification email is sent to the QART office. An onscreen notice confirms successful data transfer and the clinician submitting the dataset is redirected to a webform requesting supplemental information. This webform is constructed not to allow submission unless all mandatory fields are completed. Although central QA reviewers require a minimum amount of clinical information to interpret RT volumes and plans correctly, the submission webform should not duplicate the clinical CRF. Information that can be extracted from the clinical database or directly from the digital RT dataset is not requested from the institution to avoid redundancy, minimize the risk of introducing inconsistencies and decrease workload for participants. The submission webform (Fig. 3) only requests information that cannot be found elsewhere. Webform contents are transferred directly into a Microsoft Access database and both the participating site and QART office receive an email notification upon completion. While user names and passwords were previously instituted to grant site personnel access to the uploader, this required significant time on the part of the central QART office to manage, especially since RT dataset submission may be performed by multiple different personnel at each institution. Therefore, this type of credentialing is no longer required.
A. Fairchild et al. / Radiotherapy and Oncology 103 (2012) 279–286
281
Table 1 Main system requirements. Task
Parameters
General
Network reliability in terms of up-time and fidelity of data transfer Avoidance of local hardware and software installation by use of local computers ensures costs are minimized Web-based accessibility from any geographic location Easy performance of software upgrades and updates Intuitive functionality e.g. evaluation tools similar to those in treatment planning software Automatic data deposition into existing databases Uniformity of data submission procedures for all trials Ability to customize to specific studies, use over the lifetime of a clinical trial, and adapt to future needs Clear procedures for and levels of access Adequate firewall security and audit trails of electronic transactions Avoidance of duplication of data in clinical case report forms Central storage and archiving in a queriable database Fulfilment of regulations regarding data confidentiality Opportunity for collaboration on future development Affordable
Submission from centres to QART office Ability to upload operating-system independent Built in malware scan prior to acceptance by HQ terminal server Reads in digital data (DICOM-RT, DICOM, RTOG formats) exported from multiple TPS Sufficient information submitted for QART office to allow troubleshooting (e.g. copy of source data) QART office DDIQA Utilize VODCA software to ensure consistency, completeness, formatting, lack of corruption, and recalculation of DVHs Consistent method of de-identification between RDC clinical database, pharmacovigilance information, imaging platform and RT data Central evaluation Document assessment of RT plan via a web-based evaluation matrix, standardized to facilitate consensus-building First-line support provided by QART office in case of technical difficulties and if required, HQ IT department and VODCA vendor are contacted HQ QART office active in central coordination, follow-up on missing data and communication of final reports to submitting institutions
Fig. 1. Summary of work flow within the platform.
282
EORTC ROG QART platform
Fig. 2. Uploader interface.
Fig. 3. Site submission webform example.
A terminal server solution was employed, which makes use of the full potential of an EORTC HQ workstation via remote access.
This allows regular software updates and upgrades to be performed easily by the HQ IT department. The EORTC is in a unique
A. Fairchild et al. / Radiotherapy and Oncology 103 (2012) 279–286
position as a sponsor in many trials involving RT. Since it manages both treatment and clinical data, a submitted RT dataset can immediately be cross-linked with other clinical data on the same patient, making it easier to detect errors. However, it is critical that anonymization is performed in a consistent way for all different types of information (RT, clinical, pharmacovigilance, translational) pertaining to the same patient. DDIQA by the HQ QART office uses the VODCA (Visualisation and Organization of Data for Cancer Analysis; [MSS GmbH, Zurich]) program, a software package which allows digital RT-related imaging, treatment planning and verification data to be collected and reviewed [18,19]. VODCA is an updated version of the GUINESS software which was originally developed for complications modelling within the UK MRC RT01 trial [20,21]. VODCA performs TPS file format translation, storage in a unified internal DICOM format, visualization, and editing, with internal database links. Since site dose-volume histograms (DVHs) lack consistency as a result of the various TPS algorithms in use, recalculation of DVHs is also performed [22]. In the current version of VODCA, diagnostic images such as MRI and PET can also be uploaded to permit evaluation of contours. If issues are found with the dataset, the institution is contacted directly to investigate and may be asked to re-export. Completion of the submission process often requires iterative communication with site personnel: the Advanced Technology Consortium (ATC) reports a 26–29% intervention rate to correct data integrity problems [22]. With increasing duration of participation, the TROG central QA office rate of intervention decreased to 15% from approximately 80% at trial commencement [6]. At EORTC, we have observed a widely varying rate of intervention required depending on the trial. Central QART reviewers access VODCA remotely through a secure virtual private network. Evaluation of treatment plans cannot be performed automatically at this time. Reviewers, assigned by the QART Strategic Committee co-chairs, assess submissions for protocol compliance through an online matrix under five headings: planning procedure; volume definition; treatment plan and target volume dose parameters; organ at risk (OAR) dose parameters; and RT delivery and verification (Fig. 4). Each section has specific items to be checked against protocol requirements, with space to provide a grade and additional comments. An overall summary grade is assigned based on a priori definitions of deviations. Reviewers receive a copy of their evaluation by email. A second Microsoft Access database is automatically populated with grades and text feedback. Once consensus reviews are obtained, participating institutions are informed of the results. The EORTC QART Manager, with the assistance of successive Emmanuel van der Schueren Fellows in Radiation Oncology, established the foundation for the platform with the ROG Executive and QART Strategic Committee. Defining the RT data analysis software to be acquired was performed in 2008 after completion of a market survey. After retrospective testing within several clinical trials, the platform was introduced in phases starting in January 2009. The complete QA platform, integrating the VODCA software installed at EORTC HQ, permits timely, secure, and fully digital central DICOM-RT review [2]. It was clinically implemented in November 2010 and is currently being utilized to complete QART procedures of ongoing EORTC trials involving RT. The first trial to use this digital platform was the randomized phase III EORTC 22033-26033 protocol, comparing standard conformal RT versus primary systemic therapy for low grade glioma (ClinicalTrials.gov identifier NCT00182819). The platform was employed for the central evaluation of tumour and OAR volumes from 41 DR cases submitted by 40 institutions [23]. Participating institutions downloaded CT slices from the EORTC server, delineated protocol volumes and uploaded resulting structure files.
283
After import into VODCA, the trial QART team compared the DR dataset against expert contours and generated a customized report. The average interval from site submission of volumes to first QART team decision was 39.5 days (N = 29 sites), including troubleshooting. For the ICR of the same trial, 72 plans from 58 institutions were to be uploaded for central assessment. Of these, 57/72 (79.2%) from 48 sites and 14 countries were submitted and technically usable; the remaining 15 were never uploaded in an evaluable format or not submitted at all. In 10 of the reviewable cases, technical issues on the part of the sending institution were encountered which delayed case evaluation by an average of two months. Overall grades of minor and major deviation were assigned to 36.8% and 31.6% of cases, respectively. The ICR results have been accepted for publication separately.
Discussion Platforms for digital data exchange such as this one streamline RT plan collection and analysis, improve the quality and standardization of RT delivered within multicentre clinical trials, and reduce the time and effort required by participating sites and coordinating offices to complete QA procedures [24]. Through central RT plan evaluation, the current use of advanced technologies by sites participating in ROG trials can be assessed. Evaluation of film and hard copy data twice per year or via transfer by mail is simply no longer adequate [22]. Digital data transfer is less burdensome than paper submission since the RT plan can be exported as one package; ideally this is performed by a radiation oncologist, clinical physicist or radiation technologist involved in the patient’s care. Large volumes of images and image-based RT data can be transferred, reviewed and stored in a cost-effective manner. Deidentification is much easier with digital data, and review is more efficient. Opportunities for secondary analysis of centrally archived RT plans are almost unlimited, such as correlation of dosimetric or volumetric parameters with clinical outcomes [25]. Although cost can be a factor when considering the acquisition and construction of a digital review platform, this must be balanced against the significant increase in data availability, savings compared to paperbased systems, and decreased risk of breaches of confidentiality or loss of documentation during electronic transmission between facilities, compared to mailing [26]. Other advantages of the EORTC ROG platform in comparison to others include: pre-registration of users is not required; automatic notification of upload to system administrators; minimization of the need for IT support for both senders and receivers of data; and avoidance of on-site installation. The platform is simple to use while still flexible enough to be tailored to the requirements of current and future trials. The ROG and EORTC HQ are actively pursuing the QART ideal of prospective RT plan review on a real-time basis [5]. Several platforms have been developed by inter-centre networks and cooperative trial groups. The levels of sophistication and complexity vary with the main distinguishing feature being whether the systems are used for central review or not [6]. Two European centres built a Matlab-based data-sharing infrastructure with free or open-source software to collect both clinical and RT data [27]. MISTIR is another functional data management and analysis framework used for data exchange and manual analysis [17]. It is an all-in-one centrally hosted database used primarily for RT planning studies. After upload to the central database and DDIQA, RT plans are visualized using Matlab, CERR or VODCA. Recalculated DVHs are then used to derive relevant dose matrices [17]. As MISTIR was originally built for a 12-centre in silico trial, it is a closed, centrally-managed network. A third platform developed by investigators from Michigan provides a framework specifically for importing and visualizing IMRT plans [28]. Like MISTIR and
284
EORTC ROG QART platform
Fig. 4. Example evaluation matrix. Each field has a drop-down menu containing pre-specified choices based on degree of compliance with the protocol which are either quantitative (e.g.
