Manaswi Saha
Jon Froehlich
ABSTRACT
We introduce Project Sidewalk, a new web-based tool that enables online crowdworkers to remotely label pedestrian-related accessibility problems by virtually walking through city streets in Google Street View. To train, engage, and sustain users, we apply basic game design principles such as interactive onboarding, mission-based tasks, and progress dashboards. In an 18-month deployment study, 797 online users contributed 205,385 labels and audited 2,941 miles of Washington DC streets. We compare behavioral and labeling quality differences between paid crowdworkers and volunteers, investigate the effects of label type, label severity, and majority vote on accuracy, and analyze common labeling errors. To complement these findings, we report on an interview study with three key stakeholder groups (N=14) soliciting reactions to our tool and methods. Our findings demonstrate the potential of virtually auditing urban accessibility and highlight tradeoffs between scalability and quality compared to traditional approaches.
Supplemental Material
- Court of Appeals 3rd Circuit. 1993. Kinney v. Yerusalim, 1993 No. 93--1168. Technical Report. Retrieved January 7, 2019 from https: //www.leagle.com/decision/199310769f3d10671900Google Scholar
- Ahmed Ali, Nuttha Sirilertworakul, Alexander Zipf, Amin Mobasheri, Ahmed Loai Ali, Nuttha Sirilertworakul, Alexander Zipf, and Amin Mobasheri. 2016. Guided Classification System for Conceptual Overlapping Classes in OpenStreetMap. ISPRS International Journal of Geo-Information 5, 6 (Jun 2016), 87.Google ScholarCross Ref
- V Antoniou and A Skopeliti. 2015. Measures and Indicators of VGI Quality: An Overview. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences II-3/W5 (2015), 345--351. https://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci. net/II-3-W5/345/2015/Google Scholar
- Michael D.M. Bader, Stephen J. Mooney, Yeon Jin Lee, Daniel Sheehan, Kathryn M. Neckerman, Andrew G. Rundle, and Julien O. Teitler. 2015. Development and deployment of the Computer Assisted Neighborhood Visual Assessment System (CANVAS) to measure healthrelated neighborhood conditions. Health & Place 31 (Jan 2015), 163--172.Google Scholar
- Hannah M Badland, Simon Opit, Karen Witten, Robin A Kearns, and Suzanne Mavoa. 2010. Can virtual streetscape audits reliably replace Project Sidewalk CHI 2019, May 4--9, 2019, Glasgow, Scotland UK physical streetscape audits? Journal of urban health : bulletin of the New York Academy of Medicine 87, 6 (Dec 2010), 1007--16.Google ScholarCross Ref
- Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010. Soylent: A Word Processor with a Crowd Inside. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 313--322. Google ScholarDigital Library
- John R Bethlehem, Joreintje D Mackenbach, Maher Ben-Rebah, Sofie Compernolle, Ketevan Glonti, Helga Bárdos, Harry R Rutter, Hélène Charreire, Jean-Michel Oppert, Johannes Brug, and Jeroen Lakerveld. 2014. The SPOTLIGHT virtual audit tool: a valid and reliable tool to assess obesogenic characteristics of the built environment. International Journal of Health Geographics 13, 1 (Dec 2014), 52.Google ScholarCross Ref
- Carlos Cardonha, Diego Gallo, Priscilla Avegliano, Ricardo Herrmann, Fernando Koch, and Sergio Borger. 2013. A Crowdsourcing Platform for the Construction of Accessibility Maps. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A '13). ACM, New York, NY, USA, 26:1--26:4. Google ScholarDigital Library
- Philippa Clarke, Jennifer Ailshire, Robert Melendez, Michael Bader, and Jeffrey Morenoff. 2010. Using Google Earth to conduct a neighborhood audit: Reliability of a virtual audit instrument. Health & Place 16, 6 (Nov 2010), 1224--1229.Google ScholarCross Ref
- John W. Creswell. 2012. Qualitative Inquiry and Research Design: Choosing Among Five Approaches (third ed.). Sage Publications, Inc.Google Scholar
- Igor Gomes Cruz and Claudio Campelo. 2017. Improving Accessibility Through VGI and Crowdsourcing. In Volunteered Geographic Information and the Future of Geospatial Data, Claudio Campelo, Michela Bertolotto, and Padraig Corcoran (Eds.). 208--226.Google Scholar
- DC.gov. {n. d.}. OpenData DC Quadrants. Retrieved January 7, 2019 from http://opendata.dc.gov/datasets/ 02923e4697804406b9ee3268a160db99Google Scholar
- DC.gov. 2010. OpenData DC Sidewalk Ramp Dataset. Retrieved January 7, 2019 from http://opendata.dc.gov/datasets/sidewalk-ramps-2010Google Scholar
- DC.gov. 2016. OpenData DC Zoning Regulations of 2016. Retrieved January 7, 2019 from http://opendata.dc.gov/datasets/ zoning-regulations-of-2016Google Scholar
- Chaohai Ding, Mike Wald, and Gary Wills. 2014. A Survey of Open Accessibility Data. In Proceedings of the 11th Web for All Conference (W4A '14). ACM, New York, NY, USA, 37:1--37:4. Google ScholarDigital Library
- Steffen Fritz, Linda See, and Maria Brovelli. 2017. Motivating and sustaining participation in VGI. In Mapping and the Citizen Sensor. London: Ubiquity Press, Chapter 5, 93--117.Google Scholar
- Google Inc. 2018. Google Street View Service. Retrieved January 7, 2019 from https://developers.google.com/maps/documentation/ javascript/streetviewGoogle Scholar
- David Gutman. 2017. Seattle may have to spend millions making sidewalks more accessible to people with disabilities | The Seattle Times. Retrieved January 7, 2019 from https://www.seattletimes.com/seattle-news/transportation/ seattle-may-have-to-spend-millions-making-sidewalks-more-accessible/Google Scholar
- Richard Guy and Khai Truong. 2012. CrossingGuard: exploring information content in navigation aids for visually impaired pedestrians. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 405--414. Google ScholarDigital Library
- Mordechai Haklay. 2010. How Good is Volunteered Geographical Information? A Comparative Study of OpenStreetMap and Ordnance Survey Datasets. Environment and Planning B: Planning and Design 37, 4 (Aug 2010), 682--703.Google ScholarCross Ref
- Muki Haklay. 2013. Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation. In Crowdsourcing Geographic Knowledge. Springer Netherlands, Dordrecht, 105--122.Google Scholar
- Mordechai (Muki) Haklay, Sofia Basiouka, Vyron Antoniou, and Aamer Ather. 2010. How Many Volunteers Does it Take to Map an Area Well? The Validity of Linus' Law to Volunteered Geographic Information. The Cartographic Journal 47, 4 (Nov 2010), 315--322.Google ScholarCross Ref
- Kotaro Hara, Shiri Azenkot, Megan Campbell, Cynthia L Bennett, Vicki Le, Sean Pannella, Robert Moore, Kelly Minckler, Rochelle H Ng, and Jon E Froehlich. 2015. Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View: An Extended Analysis. ACM Transactions on Accessible Computing (TACCESS) 6, 2 (Mar 2015), 5:1--5:23. Google ScholarDigital Library
- Kotaro Hara, Christine Chan, and Jon E Froehlich. 2016. The Design of Assistive Location-based Technologies for People with Ambulatory Disabilities: A Formative Study. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1757--1768. Google ScholarDigital Library
- Kotaro Hara, Vicki Le, and Jon Froehlich. 2013. Combining crowdsourcing and google street view to identify street-level accessibility problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13. ACM, 631--640. Google ScholarDigital Library
- Kotaro Hara, Jin Sun, Robert Moore, David Jacobs, and Jon Froehlich. 2014. Tohme: Detecting Curb Ramps in Google Street View Using Crowdsourcing, Computer Vision, and Machine Learning. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 189--204. Google ScholarDigital Library
- Daniel J. Hruschka, Deborah Schwartz, Daphne Cobb St.John, Erin Picone-Decaro, Richard A. Jenkins, and James W. Carey. 2004. Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research. Field Methods 16, 3 (Aug 2004), 307--331.Google ScholarCross Ref
- Winnie Hu. 2017. For the Disabled, New York's Sidewalks Are an Obstacle Course - The New York Times. Retrieved January 7, 2019 from https://www.nytimes.com/2017/10/08/nyregion/ new-york-city-sidewalks-disabled-curb-ramps.htmlGoogle Scholar
- Yusuke Iwasawa, Kouya Nagamine, Ikuko Eguchi Yairi, and Yutaka Matsuo. 2015. Toward an Automatic Road Accessibility Information Collecting and Sharing Based on Human Behavior Sensing Technologies of Wheelchair Users. Procedia Computer Science 63 (Jan 2015), 74--81.Google Scholar
- Levente Juhász and Hartwig H. Hochmair. 2016. User Contribution Patterns and Completeness Evaluation of Mapillary, a Crowdsourced Street Level Photo Service. Transactions in GIS 20, 6 (Dec 2016), 925-- 947.Google ScholarCross Ref
- Reuben Kirkham, Romeo Ebassa, Kyle Montague, Kellie Morrissey, Vasilis Vlachokyriakos, Sebastian Weise, and Patrick Olivier. 2017. WheelieMap: An Exploratory System for Qualitative Reports of Inaccessibility in the Built Environment. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, 38:1--38:12. Google ScholarDigital Library
- Anthony Li, Manaswi Saha, Anupam Gupta, and Jon E. Froehlich. 2018. Interactively Modeling And Visualizing Neighborhood Accessibility CHI 2019, May 4--9, 2019, Glasgow, Scotland UK M. Saha et al. At Scale: An Initial Study Of Washington DC. In Poster Proceedings of ASSETS'18. Google ScholarDigital Library
- Afra Mashhadi, Giovanni Quattrone, and Licia Capra. 2013. Putting Ubiquitous Crowd-sourcing into Context. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 611--622. Google ScholarDigital Library
- Afra Mashhadi, Giovanni Quattrone, Licia Capra, and Peter Mooney. 2012. On the Accuracy of Urban Crowd-sourcing for Maintaining Large-scale Geospatial Databases. In Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration (WikiSym '12). ACM, New York, NY, USA, 15:1--15:10. Google ScholarDigital Library
- Hugh Matthews, Linda Beale, Phil Picton, and David Briggs. 2003. Modelling Access with GIS in Urban Systems (MAGUS): capturing the experiences of wheelchair users. Area 35, 1 (Mar 2003), 34--45.Google ScholarCross Ref
- Andrew May, Christopher J. Parker, Neil Taylor, and Tracy Ross. 2014. Evaluating a concept design of a crowd-sourced 'mashup' providing ease-of-access information for people with limited mobility. Transportation Research Part C: Emerging Technologies 49 (Dec 2014), 103--113.Google Scholar
- Allan R Meyers, Jennifer J Anderson, Donald R Miller, Kathy Shipp, and Helen Hoenig. 2002. Barriers, facilitators, and access for wheelchair users: sbstantive and methodologic lessons from a pilot study of environmental effects. Social Science & Medicine 55, 8 (Oct 2002), 1435--1446.Google ScholarCross Ref
- Amin Mobasheri, Jonas Deister, and Holger Dieterich. 2017. Wheelmap: the wheelchair accessibility crowdsourcing platform. Open Geospatial Data, Software and Standards 2, 1 (Dec 2017), 27.Google ScholarCross Ref
- Ladan Najafizadeh and Jon E. Froehlich. 2018. A Feasibility Study of Using Google Street View and Computer Vision to Track the Evolution of Urban Accessibility. In Poster Proceedings of ASSETS'18. Google ScholarDigital Library
- Andrea Nuernberger. 2008. Presenting Accessibility to Mobility-Impaired Travelers (Ph.D. Thesis). Ph.D. Dissertation. University of California, Santa Barbara.Google Scholar
- OpenSidewalks.com. {n. d.}. OpenSidewalks. Retrieved January 7, 2019 from https://www.opensidewalks.com/Google Scholar
- Katherine Panciera, Reid Priedhorsky, Thomas Erickson, and Loren Terveen. 2010. Lurking? cyclopaths?: a quantitative lifecycle analysis of user behavior in a geowiki. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1917--1926. Google ScholarDigital Library
- Falko Weigert Petersen, Line Ebdrup Thomsen, Pejman Mirza-Babaei, and Anders Drachen. 2017. Evaluating the Onboarding Phase of FreetoPlay Mobile Games: A Mixed-Method Approach. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17). ACM, New York, NY, USA, 377--388. Google ScholarDigital Library
- Catia Prandi, Paola Salomoni, and Silvia Mirri. 2014. mPASS: Integrating people sensing and crowdsourcing to map urban accessibility. In 2014 IEEE 11th Consumer Communications and Networking Conference (CCNC). IEEE, 591--595.Google ScholarCross Ref
- Giovanni Quattrone, Afra Mashhadi, Daniele Quercia, Chris SmithClarke, and Licia Capra. 2014. Modelling Growth of Urban Crowdsourced Information. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM '14). ACM, New York, NY, USA, 563--572. Google ScholarDigital Library
- Alexander J Quinn and Benjamin B Bederson. 2011. Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1403--1412. Google ScholarDigital Library
- Andrew G. Rundle, Michael D.M. Bader, Catherine A. Richards, Kathryn M. Neckerman, and Julien O. Teitler. 2011. Using Google Street View to Audit Neighborhood Environments. American Journal of Preventive Medicine 40, 1 (Jan 2011), 94--100.Google ScholarCross Ref
- Daniel Sinkonde, Leonard Mselle, Nima Shidende, Sara Comai, Matteo Matteucci, Daniel Sinkonde, Leonard Mselle, Nima Shidende, Sara Comai, and Matteo Matteucci. 2018. Developing an Intelligent PostGIS Database to Support Accessibility Tools for Urban Pedestrians. Urban Science 2, 3 (Jun 2018), 52.Google ScholarCross Ref
- Sharon Spall. 1998. Peer Debriefing in Qualitative Research: Emerging Operational Models. Qualitative Inquiry 4, 2 (Jun 1998), 280--292.Google Scholar
- Jin Sun and David W. Jacobs. 2017. Seeing What is Not There: Learning Context to Determine Where Objects are Missing. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1234--1242.Google Scholar
- United States Access Board. {n. d.}. Accessible Guidelines and Standards for Public Rights-of-Way. ({n. d.}). Retrieved January 7, 2019 from https://www. access-board.gov/guidelines-and-standards/streets-sidewalks/ public-rights-of-way/proposed-rights-of-way-guidelines/ chapter-r3-technical-requirementsGoogle Scholar
- United States Department of Justice. 2010. 2010 ADA Standards for Accessible Design. Retrieved January 7, 2019 from https://www.ada. gov/regs2010/2010ADAStandards/2010ADAstandards.htmGoogle Scholar
- United States Department of Justice Civil Rights Division. 1990. Americans with Disabilities Act of 1990, Pub. L. No. 101--336, 104 Stat. 328. Technical Report.Google Scholar
- U.S. Census Bureau. {n. d.}. U.S. Census QuickFacts: District of Columbia. Retrieved January 7, 2019 from https://www.census.gov/ quickfacts/fact/table/dc/PST045217Google Scholar
- U.S. Department of Transportation Federal Highway Administration. {n. d.}. A Guide for Maintaining Pedestrian Facilities for Enhanced Safety. Retrieved January 7, 2019 from https://safety.fhwa.dot.gov/ ped_bike/tools_solve/fhwasa13037/chap3.cfmGoogle Scholar
- Washington.org. {n. d.}. Washington DC Visitor Research. Retrieved January 7, 2019 from https://washington.org/press/dc-information/ washington-dc-visitor-researchGoogle Scholar
- Jeffrey S. Wilson, Cheryl M. Kelly, Mario Schootman, Elizabeth A. Baker, Aniruddha Banerjee, Morgan Clennin, and Douglas K. Miller. 2012. Assessing the Built Environment Using Omnidirectional Imagery. American Journal of Preventive Medicine 42, 2 (Feb 2012), 193--199.Google ScholarCross Ref
- J.O. Wobbrock, S.K. Kane, K.Z. Gajos, S. Harada, and J. Froehlich. 2011. Ability-Based Design: Concept, Principles and Examples. ACM Transactions on Accessible Computing (TACCESS) 3, 3 (2011), 9. http: //portal.acm.org/citation.cfm?id=1952384 Google ScholarDigital Library
Index Terms
- Project Sidewalk: A Web-based Crowdsourcing Tool for Collecting Sidewalk Accessibility Data At Scale
Recommendations
“I never realized sidewalks were a big deal”: A Case Study of a Community-Driven Sidewalk Accessibility Assessment using Project Sidewalk
CHI '24: Proceedings of the CHI Conference on Human Factors in Computing SystemsDespite decades of effort, pedestrian infrastructure in cities continues to be unsafe or inaccessible to people with disabilities. In this paper, we examine the potential of community-driven digital civics to assess sidewalk accessibility through a ...
Multiple Images of the City: Unveiling Group-Specific Urban Perceptions through a Crowdsourcing Game
HT '17: Proceedings of the 28th ACM Conference on Hypertext and Social MediaOur perceptions of public spaces are central for our experience in the city. Understanding which factors shape this perception informs both urban planners, that aim at improving city life, as well as computational models that help us navigate in urban ...
Smart and Fermented Cities: An Approach to Placemaking in Urban Informatics
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsWhat makes a city meaningful to its residents? What attracts people to live in a city and to care for it? Today, we might see such questions as concerns for HCI, given the emerging agendas of smart and connected cities, IoT, and ubiquitous computing: ...
Comments