Citation: Vigo, M. & Brajnik, G. (2011). Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers 23: 137-155. Retrieved from http://iwc.oxfordjournals.org/content/23/2/137.short
In this article, the authors study seven quantitative metrics for reviewing web accessibility to determine which are the most reliable for assessing web sites. As the authors note, despite conformance criteria like the Web Content Accessibility Guidelines (WCAG) and numerous automated conformance-checking tools, metrics can provide more detailed quality control when comparing multiple web sites or multiple iterations of the same web site (p. 137).
The authors explored different metrics using the following areas to determine which was of the highest quality:
- Validity – “How well scores produced by a metric predict all and only the effects that real accessibility problems will have” and “how well scores mirror all and only the true violations” of conformance criteria like WCAG 2.0 (p. 138)
- Reliability – Are the metrics consistent?
- Sensitivity – Is the metric too sensitive to minor changes in accessibility level?
- Adequacy – Does the metric report its findings in a consistent manner that can be adequately quantified?
- Complexity – How many variables does the metric need to compute its scores, and/or do external tools exist to create the metric?
Out of a large number of metrics reviewed briefly, the authors analyzed the following automatic metrics:
- Sullivan and Matson (2000) – “FR”
- Olsina and Rossi (2002) – “Web Quality Evaluation Method” or Web-QEM
- Bailey and Burd (2005) – “Page Measure” or PM
- Parmanto and Zeng (2005) – “Web Accessibility Barrier” or WAB
- Bühler et al (2006) – “A3”
- Velleman et al. (2007) – “Unified Web Evaluation Methodology” or UWEM
- Vigo et al. (2007) – “Web Accessibility Quantitative Metric” or WAQM
- Lopes and Carrico (2008) – “Web Interaction Environments” or WIE
Of the metrics analyzed, only WAQM, WAB, and PM fulfilled the validity criteria (p. 151), with the WAQM and WAB scoring slightly better than the PM metric in terms of adequacy (p. 154). The authors note that even these three metrics are less than idea, and suggest that researchers “focus more on quality aspects of accessibility metrics with the long-range goal” of improving them (p. 154).