Accuracy of SIAscopy for pigmented skin lesions encountered in primary care: development and validation of a new diagnostic algorithm
© Emery et al; licensee BioMed Central Ltd. 2010
Received: 1 December 2009
Accepted: 25 September 2010
Published: 25 September 2010
Diagnosing pigmented skin lesions in general practice is challenging. SIAscopy has been shown to increase diagnostic accuracy for melanoma in referred populations. We aimed to develop and validate a scoring system for SIAscopic diagnosis of pigmented lesions in primary care.
This study was conducted in two consecutive settings in the UK and Australia, and occurred in three stages: 1) Development of the primary care scoring algorithm (PCSA) on a sub-set of lesions from the UK sample; 2) Validation of the PCSA on a different sub-set of lesions from the same UK sample; 3) Validation of the PCSA on a new set of lesions from an Australian primary care population. Patients presenting with a pigmented lesion were recruited from 6 general practices in the UK and 2 primary care skin cancer clinics in Australia. The following data were obtained for each lesion: clinical history; SIAscan; digital photograph; and digital dermoscopy. SIAscans were interpreted by an expert and validated against histopathology where possible, or expert clinical review of all available data for each lesion.
A total of 858 patients with 1,211 lesions were recruited. Most lesions were benign naevi (64.8%) or seborrhoeic keratoses (22.1%); 1.2% were melanoma. The original SIAscopic diagnostic algorithm did not perform well because of the higher prevalence of seborrhoeic keratoses and haemangiomas seen in primary care. A primary care scoring algorithm (PCSA) was developed to account for this. In the UK sample the PCSA had the following characteristics for the diagnosis of 'suspicious': sensitivity 0.50 (0.18-0.81); specificity 0.84 (0.78-0.88); PPV 0.09 (0.03-0.22); NPV 0.98 (0.95-0.99). In the Australian sample the PCSA had the following characteristics for the diagnosis of 'suspicious': sensitivity 0.44 (0.32-0.58); specificity 0.95 (0.93-0.97); PPV 0.52 (0.38-0.66); NPV 0.95 (0.92-0.96). In an analysis of lesions for which histological diagnosis was available (n = 111), the PCSA had a significantly greater Area Under the Curve than the 7-point checklist for the diagnosis of melanoma (0.83; 95% CI 0.71-0.95 versus 0.61; 95% CI 0.44-0.78; p = 0.02 for difference).
The PCSA could have a useful role in improving primary care management of pigmented skin lesions. Further work is needed to develop and validate the PCSA in other primary care populations and to evaluate the cost-effectiveness of GP management of pigmented lesions using SIAscopy.
Pigmented skin lesions are a common presenting problem in general practice and, while the majority are benign naevi or non-melanocytic lesions (seborrhoeic keratoses, haemangiomas), a small minority are malignant melanomas. Melanoma is a serious skin cancer, responsible for 2% of all cancers and 1% of all cancer deaths in the UK, with about 8,000 new cases and 1,800 deaths a year . Worldwide, the incidence of melanoma is increasing faster than any other solid cancer with an approximate doubling of rates every 10-20 years in countries with Caucasian populations [2, 3].
Pigmented lesions and melanoma pose particular diagnostic and management challenges for general practitioners (GPs) . GPs are less able than dermatologists to differentiate melanomas from other pigmented lesions [5, 6], probably because an individual GP will encounter melanoma infrequently . British data following the establishment of urgent referral pathways for all suspected skin cancers  showed that only 12% of referred lesions were diagnosed as skin cancer and only 42% of skin cancers were referred via this route . There have been conflicting findings about the performance of GPs who have been trained in melanoma diagnosis either face-to-face  or via the internet . In a primary care setting the ability to distinguish benign from suspicious lesions is as important as a clinical diagnosis of melanoma in making the decision either to reassure or to refer urgently for dermatological review. Studies of diagnostic accuracy and decision aids for use in primary care need to reflect the diagnostic distinction between suspicious and benign lesions as well as the identification of melanomas.
New approaches are required to improve GPs' assessment of pigmented skin lesions. Dermoscopy has been shown to improve the diagnostic accuracy for melanoma in the specialist setting  and in two randomised controlled trials in general practice [13, 14]. However, dermoscopy is a relatively time-consuming technique to learn; in a recent trial of dermoscopy and digital monitoring Australian GPs required up to 30 hours of internet-based learning to acquire adequate skills and only 63% of those trained actually recruited patients into the trial. There is also current interest in teledermatology but, for suspicious pigmented lesions, it is unlikely to dramatically reduce the need for conventional clinical consultations with experts whilst maintaining clinical safety .
Development of the primary care scoring algorithm (PCSA) on a sub-set of lesions from the UK sample (UK Development lesion dataset);
Validation of the PCSA on a different sub-set of lesions from the same UK sample (UK Validation lesion dataset);
Validation of the PCSA on a new set of lesions from an Australian primary care population (Australian Validation lesion dataset).
Ethical approval for the UK component of the study was obtained from the Cambridge Research Ethics Committee (REC Ref. 04/079) and research governance approval from Cambridge City and South Cambridgeshire Primary Care Trusts (Project number L00569). Ethical approval for the Australian component of the study was obtained from the University of Western Australia's Human Research Ethics Committee (HREC Ref. RA/4/1/1739).
Six general practices were recruited from Cambridge city and the surrounding suburban and rural areas covering a registered population of 52,913. Adult patients aged over 18 years were recruited into the study by their general practitioner (GP) if they had presented with concerns about a pigmented skin lesion: as these were lesions presented by the patients to their GP, they included lesions that were ultimately considered not clinically suspicious. Participants were formally consented and data collected about their lesion by JH within two weeks of initial presentation to their GP. Data collection occurred between January 2005 and January 2006.
Three primary care skin cancer clinics operated by GPs were recruited from the metropolitan area of Perth, Western Australia. Adult patients aged over 18 years were recruited into the study by their GP if they presented with concerns about a pigmented skin lesion: again, these included lesions that ultimately were considered not clinically suspicious. Additional lesions were also included when a pigmented skin lesion was identified as potentially suspicious during their clinical examination. Participants were formally consented and data collected by AJW on the same day as they presented to their GP. Data collection occurred between April 2008 and January 2009.
7-point melanoma checklist11;
Macroscopic digital photograph (Canon EOS 400 D camera, Canon EF-S60 macro lens, Canon MR-14EX Macro Ring Lite flash, JPEG picture format: 3888 × 2592 pixels);
Dermoscopic digital photograph (Canon EOS 400 D camera, Canon EF-S60 macro lens, Heine SLR Photadaptor, Heine Delta 20 dermatoscope, JPEG picture format: 3888 × 2592 pixels);
SIAscan (MoleMate™SIAscope V and Microsoft Windows™application).
SIAscan images and data (including the location of the lesion and the age group and sex of the patient) were assessed by a SIAscopy expert, who was blinded to the 7-point melanoma checklist results and clinical photographs. The SIAscopy expert scored the presence or absence of each specific SIAscopic feature including those previously associated with melanoma : size of lesion, age of patient, dermal melanin, collagen holes and blood displacement with erythematous blush. Additional features that were also scored were: blood vessels, white dots on the collagen view, blood lacunes and a cerebriform melanin pattern (see Figure 1).
Diagnostic reference standards
In-person clinical review of the lesion by one expert, including 7-point checklist and digital dermoscopy.
Clinical diagnosis made on the basis of the 7-point checklist, photographic and dermoscopy images.
The expert reviewers were blinded to the SIAscan images. For the reference standard diagnosis we categorised lesions in two complementary ways relevant to primary care decision-making: (1) 'suspicious' or benign and (2) melanoma or other pigmented lesion. The definition of 'suspicious' was a lesion that, if seen in general practice, would warrant referral, excision or short-term monitoring.
Development stage: a 66% sub-sample of lesions (UK Development lesion dataset) was scored using the Moncrieff scoring system. In order to account for the different prevalence of certain pigmented skin lesions seen in primary care, a Primary Care Scoring Algorithm (PCSA) was developed.
Validation stage one: the PCSA was validated against the remaining 33% sub-sample of lesions (UK Validation lesion dataset).
Validation stage two: the PCSA was validated against the lesions recruited in Australia (Australian Validation lesion dataset).
Data were recorded on a Microsoft Access database and analysed with Microsoft Excel and SPSS version 11.5 for Windows. Sensitivity, specificity, positive and negative predictive values and their associated 95% confidence intervals were calculated using standard approaches including the Wilson method to account for small sample sizes in some of the cells . Receiver operating characteristic (ROC) curves and associated area-under-the-curve (AUC) were created using standard functions within SPSS 11.5 to explore different cut-off scores for the PCSA. We compared the area under the curve for the PCSA and the 7-point checklist for all lesions for which we had obtained histology, using the manual method described by Hanley and McNeil .
Distribution of lesions in Development and Validation datasets, based on expert clinical diagnosis or histology where available
UK validation dataset
Australian validation dataset
Basal cell carcinoma
(a) Development stage
Performance characteristics of SIAscopy for the diagnosis of 'suspicious' in the different datasets of lesions.
(Moncrieff scoring system)
UK validation dataset
Australian validation dataset (PCSA)
0.54 (0.35 - 0.72)
0.77 (0.73 - 0.81)
Positive predictive value
0.12 (0.075 - 0.20)
Negative predictive value
0.96 (0.93 - 0.98)
(b) UK Validation stage
The new PCSA was tested against the 208 lesions in the Validation Lesion dataset, which included 6 suspicious lesions and two histopathologically confirmed melanomas. The performance of the PCSA is presented in Table 2.
(c) Australian Validation stage
The PCSA was tested against the 581 lesions recruited in Australia. There were 52 suspicious lesions including 5 histopathologically confirmed melanomas and 2 lentigo malignas. The performance of the PCSA is presented in Table 2. In this second validation stage, the sensitivity for the diagnosis of suspicious was similar to the UK findings (0.44; 95% CI 0.32-0.58). However, specificity was significantly better (0.95; 95% CI 0.93-0.97). Furthermore, due to the higher prevalence of suspicious lesions in the Australian dataset, the positive predictive value for the diagnosis of suspicious was 0.52 (95% CI 0.38-0.66) while maintaining an acceptably high negative predictive value (0.95; 95% CI 0.92-0.96).
Distribution of lesions for which histological diagnosis was available.
Basal cell carcinoma
This is the first study to test the use of SIAscopy for lesions encountered in a primary care setting and also outside the UK. We have applied a systematic approach in which we tested the initial Moncrieff scoring system to see how it would function on lesions presented in a primary care setting. This is an important step in studies of new diagnostic techniques to reduce the effects of spectrum bias. In addition to developing a new diagnostic algorithm we have conducted a second validation study on a different primary care population. This second validation study was conducted in an Australian primary care setting which, except for a higher prevalence of solar lentigos, had a similar prevalence of lesions to the UK dataset. We accept that a limitation of this study is our inability to obtain histopathological diagnoses on all the lesions recruited, but this would have been ethically unacceptable. To inform the clinical-expert reference-standard diagnosis we deliberately chose to obtain maximum clinical data, including the 7-point checklist and dermoscopy. We chose to do this so we could be as accurate as possible with our reference diagnosis where histology was not available. It is also theoretically possible that some amelanotic melanomas were not recruited into the study on the basis of our inclusion criteria. Furthermore, we did not follow-up any lesions determined as benign so it is theoretically possible that some clinically significant lesions may have been missed by our reference standard diagnoses.
The Moncrieff scoring system was found to be less accurate than in the secondary care setting due to the different prevalence of lesions among the primary care population. In order to account for the higher prevalence of non-melanocytic lesions, such as seborrhoeic keratoses and haemangiomas, we developed a new Primary Care Scoring Algorithm which was more specific than the Moncrieff scoring system for 'suspicious' but no more sensitive. Higher specificity was particularly identified in the Australian dataset suggesting that sun-related skin damage does not adversely affect the diagnostic accuracy of SIAscopy. The current algorithm accounts for size of lesion (> 6 mm) and age of patient. The mean age of participants from each studied population was 45 and 50 years respectively. It is not possible therefore to comment on the performance of the PCSA in elderly populations in which the algorithm may become less specific. It was reassuring that the PCSA's moderate sensitivity for 'suspicious' does not appear to be reflected in its sensitivity for melanoma, but inevitably there were too few melanomas in this study to provide robust estimates of diagnostic accuracy for melanoma in primary care. In subsequent research, simulation modelling of the PCSA in which a higher prevalence of melanomas was entered into the dataset suggests high sensitivity and specificity to detect melanoma . Ultimately, a primary care algorithm should be good at identifying 'suspicious' pigmented lesions, including melanoma, as well as accurately ruling out lesions which are unlikely to be clinically significant. It is interesting that the PCSA appears to be more accurate than the 7-point checklist in diagnosing melanoma. The 7-point checklist was completed by the two medically qualified and trained researchers including one who was a plastic surgeon (JH). The relatively poor performance of the 7-point checklist cannot be explained by inconsistent application of the items. There were several non-melanocytic lesions which were thought clinically to be pigmented. The 7-point checklist, and Siascopy, are intended for use only with melanocytic lesions, although in clinical primary care practice this distinction can sometimes be difficult. The inclusion of non-melanocytic lesions may partially explain the poorer performance of the 7-point checklist compared to previously published data.
The analysis conducted assumes that there was no selection bias in the sampling of lesions chosen for biopsy. This is theoretically possible, for example if the 7-point score were used to inform the clinical decision to excise, and so our finding should be interpreted with some caution.
In this study we used experts in SIAscopy to interpret the SIAscans. This therefore reflects the best performance of SIAscopy in primary care and not how it would perform in the hands of general practitioners. As this is the first study of SIAscopy on lesions from primary care, we needed to determine the best possible performance of the technique in experienced hands. We are now conducting a randomised controlled trial of training English general practitioners in SIAscopy, including the application of the PCSA, to determine its effects on clinical practice (the MoleMate UK Trial). This trial will provide further evidence on the diagnostic accuracy of the PCSA when used by general practitioners as well as the cost-effectiveness of SIAscopy in English primary care.
We believe that the features of SIAscopy may be a great deal easier to learn than those of dermoscopy which can take a long period of training in which to become proficient. A recent study we have conducted suggests that SIAscopy features can be learnt by general practitioners using a CD-rom based tutorial in approximately two hours . In a recent trial in general practice, dermoscopy had a sensitivity of 55% and specificity of 89% for malignant lesions which is comparable with our findings for SIAscopy, albeit in expert hands . The MoleMate UK Trial will provide more comparable data in due course . While there is no doubt that dermoscopy and digital monitoring can significantly improve the management of pigmented lesions in primary care, we believe that SIAscopy could be simpler to learn and may therefore have greater utility for a wider group of primary care practitioners than dermoscopy.
The PCSA for SIAscopy could have an important role in improving the management of pigmented skin lesions in primary care. This study has confirmed the key diagnostic features of lesions commonly encountered in primary care. Further work is required to determine the impact of training GPs in SIAscopy on their clinical management of pigmented skin lesions, and the quality of their referrals to the secondary care skin cancer clinics.
We thank the BUPA Foundation who funded the UK component of this study, Tom Fanshawe, formerly of the Centre for Applied Medical Statistics, Department of Public Health & Primary Care, University of Cambridge, and Chris Brown from the Clinical Trials Centre, University of Sydney for their statistical input. We also thank the general practices, skin cancer clinics and patients who participated in this study.
- UK skin cancer incidence statistics. CRUK[http://info.cancerresearchuk.org/cancerstats/types/skin/incidence]
- Lens MB, Dawes M: Global perspectives of contemporary epidemiological trends of cutaneous malignant melanoma. Br J Dermatol 2004,150(2):179–185. 10.1111/j.1365-2133.2004.05708.xView ArticlePubMedGoogle Scholar
- Diffey BL: The future incidence of cutaneous melanoma within the UK. Br J Dermatol 2004,151(4):868–872. 10.1111/j.1365-2133.2004.06216.xView ArticlePubMedGoogle Scholar
- Murchie P, Campbell NC: Pigmented lesions, cutaneous melanoma, and future challenges for primary care. Eur J Gen Pract 2007,13(3):151–154. 10.1080/13814780701627354View ArticlePubMedGoogle Scholar
- Chen SC, Pennie ML, Kolm P, Warshaw EM, Weisberg EL, Brown KM, et al.: Diagnosing and managing cutaneous pigmented lesions: primary care physicians versus dermatologists. J Gen Intern Med 2006,21(7):678–682. 10.1111/j.1525-1497.2006.00462.xView ArticlePubMedPubMed CentralGoogle Scholar
- Morrison A, O'Loughlin S, Powell FC: Suspected skin malignancy: a comparison of diagnoses of family practitioners and dermatologists in 493 patients. Int J Dermatol 2001,40(2):104–107. 10.1046/j.1365-4362.2001.01159.xView ArticlePubMedGoogle Scholar
- Brochez L, Verhaeghe E, Bleyen L, Naeyaert JM: Diagnostic ability of general practitioners and dermatologists in discriminating pigmented skin lesions. J Am Acad Dermatol 2001,44(6):979–986. 10.1067/mjd.2001.113442View ArticlePubMedGoogle Scholar
- NICE, Cancer service guidance:Improving Outcomes for People with Skin Tumours including Melanoma. 2006. [http://www.nice.org.uk/Guidance/CSGSTIM]Google Scholar
- Cox NH: Evaluation of the U.K. 2-week referral rule for skin cancer. Br J Dermatol 2004,150(2):291–298. 10.1111/j.1365-2133.2004.05793.xView ArticlePubMedGoogle Scholar
- Burton RC, Howe C, Adamson L, Reid AL, Hersey P, Watson A, et al.: General practitioner screening for melanoma: sensitivity, specificity, and effect of training. J Med Screen 1998,5(3):156–161.View ArticlePubMedGoogle Scholar
- Gerbert B, Bronstone A, Maurer T, Berger T, McPhee SJ, Caspers N: The effectiveness of an Internet-based tutorial in improving primary care physicians' skin cancer triage skills. J Cancer Educ 2002,17(1):7–11.PubMedGoogle Scholar
- Vestergaard ME, Macaskill P, Holt PE, Menzies SW: Dermoscopy compared with naked eye examination for the diagnosis of primary melanoma: a meta-analysis of studies performed in a clinical setting. Br J Dermatol 2008,159(3):669–676.PubMedGoogle Scholar
- Argenziano G, Puig S, Zalaudek I, Sera F, Corona R, Alsina M, et al.: Dermoscopy improves accuracy of primary care physicians to triage lesions suggestive of skin cancer. J Clin Oncol 2006,24(12):1877–1882. 10.1200/JCO.2005.05.0864View ArticlePubMedGoogle Scholar
- Menzies SW, Emery J, Staples M, Davies S, McAvoy B, Fletcher J, et al.: Impact of dermoscopy and short-term sequential digital dermoscopy imaging for the management of pigmented lesions in primary care: a sequential intervention trial. Br J Dermatol 2009.Google Scholar
- Bowns IR, Collins K, Walters SJ, McDonagh AJ: Telemedicine in dermatology: a randomised controlled trial. Health Technol Assess 2006,10(43):iii-xi.View ArticlePubMedGoogle Scholar
- Moncrieff M, Cotton S, Claridge E, Hall P: Spectrophotometric intracutaneous analysis: a new technique for imaging pigmented skin lesions. Br J Dermatol 2002,146(3):448–457. 10.1046/j.1365-2133.2002.04569.xView ArticlePubMedGoogle Scholar
- Govindan K, Smith J, Knowles L, Harvey A, Townsend P, Kenealy J: Assessment of nurse-led screening of pigmented lesions using SIAscope. J Plast Reconstr Aesthet Surg 2007,60(6):639–645. 10.1016/j.bjps.2006.10.003View ArticlePubMedGoogle Scholar
- Armitage P, Berry G: Statistical methods in medical research. 3rd edition. Oxford; Boston: Blackwell Scientific Publications; 1994.Google Scholar
- Hanley JA, McNeil BJ: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983,148(3):839–843.View ArticlePubMedGoogle Scholar
- Johr RH: Dermoscopy: alternative melanocytic algorithms-the ABCD rule of dermatoscopy, Menzies scoring method, and 7-point checklist. Clin Dermatol 2002,20(3):240–247. 10.1016/S0738-081X(02)00236-5View ArticlePubMedGoogle Scholar
- Hunter JE: Triaging suspicious pigmented skin lesions in primary care using the SIAscope. MD Thesis, University of Cambridge; 2008.Google Scholar
- Hanley JA, Hajian-Tilaki KO: Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: an update. Acad Radiol 1997,4(1):49–58. 10.1016/S1076-6332(97)80161-4View ArticlePubMedGoogle Scholar
- Wood A, Morris H, Emery J, Hall PN, Cotton S, Prevost AT, et al.: Evaluation of the MoleMate training program for assessment of suspicious pigmented lesions in primary care. Inform Prim Care 2008,16(1):41–50.PubMedGoogle Scholar
- Walter FM, Morris HC, Humphrys E, Hall PN, Kinmonth AL, Prevost AT, et al.: Protocol for the MoleMate UK Trial: a randomised controlled trial of the MoleMate system in the management of pigmented skin lesions in primary care [ISRCTN 79932379]. BMC Fam Pract 2010, 11: 36. 10.1186/1471-2296-11-36View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-5945/10/9/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.