Kurt Luther
Jari-Lee Tolentino
Björn Hartmann
ABSTRACT
Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.
- 99designs. http://www.99designs.com/.Google Scholar
- Barrett, T. A comparison of the goals of studio professors conducting critiques and art education goals for teaching criticism. Studies in art education (1988), 22--27.Google Scholar
- Bransford, J. D., Brown, A. L., and Cocking, R. R., Eds. How people learn: brain, mind, experience, and school. National Academy Press, Washington, D.C., 2000.Google Scholar
- Dannels, D., Gaffney, A., and Martin, K. Beyond content, deeper than delivery: What critique feedback reveals about communication expectations in design education. Int. J. Schol. Teach. & Learn. 2, 2 (2008).Google Scholar
- Dannels, D. P., and Martin, K. N. Critiquing critiques a genre analysis of feedback across novice to expert design studios. Jo. Bus. & Tech. Comm. 22, 2 (2008), 135--159.Google ScholarCross Ref
- Dondis, D. A. A primer of visual literacy. MIT Press, Cambridge, Mass., 1973.Google Scholar
- Dow, S., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D., and Klemmer, S. Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. In Proc. CHI 2011 (2011), 2807--2816. Google ScholarDigital Library
- Dow, S., Gerber, E., and Wong, A. A pilot study of using crowds in the classroom. In Proc. CHI 2013, CHI '13 (New York, NY, USA, 2013), 227--236. Google ScholarDigital Library
- Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., and Klemmer, S. R. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans. Comput.-Hum. Interact. 17, 4 (Dec. 2010), 18:1--18:24. Google ScholarDigital Library
- Easterday, M. W., Rees Lewis, D., Fitzpatrick, C., and Gerber, E. M. Computer supported novice group critique. In Proc. of DIS '14 (2014). Google ScholarDigital Library
- Feedback army. http://www.feedbackarmy.com/.Google Scholar
- Feldman, E. Practical Art criticism. Prentice Hall, Englewood Cliffs, NJ, 1994.Google Scholar
- Five second test. http://www.fivesecondtest.com/.Google Scholar
- Forrst. http://www.forrst.com/.Google Scholar
- Klemmer, S. R., Hartmann, B., and Takayama, L. How bodies matter: five themes for interaction design. In Proceedings of DIS, ACM (2006), 140--149. Google ScholarDigital Library
- Kulkarni, C., and Klemmer, S. R. Learning design wisdom by augmenting physical studio critique with online self-assessment. Tech. rep., Stanford U., 2012.Google Scholar
- Lidwell, W., Holden, K., and Butler, J. Universal principles of design. Rockport Pub, 2010.Google Scholar
- Marlow, J., and Dabbish, L. From rookie to all-star: Professional development in a graphic design community of practice. In Proc. CSCW 2014 (2014). Google ScholarDigital Library
- Nichols, A. L., and Maner, J. K. The good-subject effect: investigating participant demand characteristics. The Journal of general psychology 135, 2 (Apr. 2008), 151--165. PMID: 18507315.Google Scholar
- Nielsen, J., and Molich, R. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '90, ACM (New York, NY, USA, 1990), 249--256. Google ScholarDigital Library
- Reynolds, G. Presentation Zen: Simple ideas on presentation design and delivery. New Riders, 2011.Google Scholar
- Sadler, D. R. Formative assessment and the design of instructional systems. Instr. Sci. 18, 2 (1989), 119--144.Google ScholarCross Ref
- Schón, D. Educating the Reflective Practitioner. Jossey-Bass Publishers, San Francisco, 1990.Google Scholar
- Shneiderman, B. The eyes have it: a task by data type taxonomy for information visualizations. In Proc. VL/HCC 1996 (1996), 336--343. Google ScholarDigital Library
- Soloway, E., Guzdial, M., and Hay, K. E. Learner-centered design: the challenge for HCI in the 21st century. interactions 1 (April 1994), 36--48. Google ScholarDigital Library
- Tinapple, D., Olson, L., and Sadauskas, J. Critviz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology 15, 1 (2013), 29.Google Scholar
- Tohidi, M., Buxton, W., Baecker, R., and Sellen, A. Getting the right design and the design right. In Proceedings of CHI, ACM (2006), 1243--1252. Google ScholarDigital Library
- Wattenberg, M., and Kriss, J. Designing for social data analysis. IEEE Transactions on Visualization and Computer Graphics 12, 4 (2006), 549--557. Google ScholarDigital Library
- Willett, W., Heer, J., and Agrawala, M. Strategies for crowdsourcing social data analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '12, ACM (New York, NY, USA, 2012), 227--236. Google ScholarDigital Library
- Willett, W., Heer, J., Hellerstein, J., and Agrawala, M. Commentspace: structured support for collaborative visual analysis. In Proc. CHI 2011 (2011), 3131--3140. Google ScholarDigital Library
- Williams, R. The Non-Designer's design book. Peachpit Press, 2008. Google ScholarDigital Library
- Xu, A., and Bailey, B. What do you think?: a case study of benefit, expectation, and interaction in a large online critique community. In Proc. CSCW 2012 (2012), 295--304. Google ScholarDigital Library
- Xu, A., Bailey, B. P., and Huang, S.-W. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proc. CSCW 2014 (2014). Google ScholarDigital Library
Index Terms
- Structuring, Aggregating, and Evaluating Crowdsourced Design Critique
Recommendations
Voyant: generating structured feedback on visual designs using a crowd of non-experts
CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computingFeedback on designs is critical for helping users iterate toward effective solutions. This paper presents Voyant, a novel system giving users access to a non-expert crowd to receive perception-oriented feedback on their designs from a selected audience. ...
Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques
CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social ComputingExpert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide ...
CrowdCrit: crowdsourcing and aggregating visual design critique
CSCW Companion '14: Proceedings of the companion publication of the 17th ACM conference on Computer supported cooperative work & social computingPeople who create visual designs often struggle to find high-quality critique outside a firm or classroom, and current online feedback solutions are limited. We created a system called CrowdCrit which leverages paid crowdsourcing to generate and ...
Comments