Journal of Theoretical and Applied Electronic Commerce Research

ISSN 0718-1876 Electronic Version VOL 7 / ISSUE 3 / DECEMBER 2012 / 49-62 © 2012 Universidad de Talca - Chile

This paper is available online at www.jtaer.com

 

Smart Ideas for Smart Cities: Investigating Crowdsourcing for Generating and Selecting Ideas for ICT Innovation in a City Context

Dimitri Schuurman¹, Bastiaan Baccarne², Lieven De Marez³ and Peter Mechant⁴

Ghent University, iMinds-MICT, Ghent, Belgium, ¹ dimitri.schuurman@ugent.be, ² bastiaan.baccarne@ugent.be,³ lieven.demarez@ugent.be, ⁴ peter.mechant@ugent.be

---

Abstract

Within this article, the strengths and weaknesses of crowdsourcing for idea generation and idea selection in the context of smart city innovation are investigated. First, smart cities are defined next to similar but different concepts such as digital cities, intelligent cities or ubiquitous cities. It is argued that the smart city-concept is in fact a more user-centered evolution of the other c/fy-concepts which seem to be more technological deterministic in nature. The principles of crowdsourcing are explained and the different manifestations are demonstrated. By means of a case study, the generation of ideas for innovative uses of ICT for city innovation by citizens through an online platform is studied, as well as the selection process. For this selection, a crowdsourcing solution is compared to a selection made by external experts. The comparison of both indicates that using the crowd as gatekeeper and selector of innovative ideas yields a long list with high user benefits. However, the generation of ideas in itself appeared not to deliver extremely innovative ideas. Crowdsourcing thus appears to be a useful and effective tool in the context of smart city innovation, but should be thoughtfully used and combined with other user involvement approaches and within broader frameworks such as Living Labs.

Keywords: Crowdsourcing, Ideation, Idea selection, Smart city, Living labs

---

1 Introduction

From 3.000 BCE cities have been developing as a means for human living and habitation. Cities formed as a natural response for the changed life circumstances, but also have had a profound and lasting impact on the further development of the human species as a whole. However, only very recently an important threshold was passed as for the first time in history research showed that more than half of the total human population lives in areas considered as urban or part of a city [61]. The passing of this threshold can be seen as a landmark example of the so-called process of urbanization. This concept refers to the rate at which cities are growing, both in terms of space and in terms of population. Urbanization is driven by profound technological, social and, last but definitely not least, economic changes caused by globalization processes. This forces cities from all over the globe to adopt innovative and competitive, as well as sustainable and long-term policy strategies [49]. In the context of the emergence of new media and ICTs, performance indicators such as knowledge based social capital have gained ground compared to the physical outlook and infrastructure of cities. Against this background, concepts such as smart cities, digital cities and ubiquitous cities have been developed and put in place in order to attain both a competitive as well as a sustainable impact, as the cities themselves firmly believe that innovative uses of ICT will foster sustainable city innovation that is able to improve the quality of life of its citizens [10]. Especially the smart c/fy-concept has gained a lot of attention and momentum in the last several years in the European Union, with various projects being set up in nearly every European city [49].

A lot of publications try to conceptualize and define the elements and application domains that constitute smart c/f/es, mostly through case studies or comparative case study analysis. However, it is argued that there is a need for research on effective strategies for cities to become smarter and on approaches to mobilize the participation and intelligence of citizens, businesses and societal organizations [54]. Parallel with current developments in the domain of innovation research, the role of the end-user, or citizen in the context of cities, is becoming more and more prominent. Scholars as well as policy makers seem to agree that this citizen involvement is key to the successful development of cities, but empirical research into the nature and characteristics of this involvement remains scarce. This research article will tackle a part of this problem by adding to the understanding of generating, evaluating and selecting innovative ideas for smart city innovation by means of online crowdsourcing. We will do this by means of a case study which took place in the Belgian city of Ghent where citizens could submit and evaluate ideas for smart city innovation, a first step towards the establishment of Ghent Living Lab, a city Living Lab. In parallel, a selection of ideas was made by an independent group of professionals. Finally, representatives of the city itself evaluated all selected ideas on three criteria: innovativeness, feasibility and user benefit. This way, the ideation process could be evaluated as well as the outcomes of the selection of the users versus the independent experts. The case study research presented in this paper therefore generates some empirical evidence with regards to the usage of crowdsourcing through an online platform in the context of trying to establish a smart(er) city through a Living Lab-approach.

 

2 Defining Smart Cities

Within this paragraph, we will briefly review the present literature on smart cities and on related, but differing concepts such as d/g/fal c/f/es, /nfell/genf c/f/es or ub/qu/fous c/f/es. All these concepts are often used to address the current desire to develop sustainable and participatory citizen communities that integrate the mutual shaping perspective between society and communication technologies [38]. However, despite the fact that these concepts are too often used interchangeably, there are some strong arguments to see them as clearly separate from each other.

The oldest of the c/fy-concepts is the d/g/fal c/fy. This term was originally used to refer to all sorts of digital initiatives undertaken by cities, especially digital representations of the city and connecting citizens by providing internet access. Some examples of these early digital city-initiatives could be found in Amsterdam (Digital City Amsterdam), Helsinki (Virtual Helsinki) or Kyoto (Digital City Kyoto), sometimes even including 3D-representations of physical locations of these cities, sometimes referred to as cyber c/f/es [26]. Digital cities are sometimes characterized as information systems that collect the corresponding digital information from the actual physical cities and organize this in a public virtual space where citizens can consult this information but also interact with it, and with each other [40]. This last part of the definition, interaction with each other, is also referred to in other definitions of digital cities which stress the connectivity between the various stakeholders in a city context [16], [43]. The network infrastructure and the platforms to disclose the vast amount of digital information are of central importance within this discourse, which leads us to the conclusion that this concept carries a quite heavy technology-deterministic connotation with it [41]. Digital cities are sometimes also referred to as w/red c/f/es or /nfell/genf c/f/es and can be considered as technology-burdened peers of smart cities, although in practice a lot of cities use the concept smarf c/fy to refer to their digital city-initiatives [49].

Ubiquitous cities, sometimes referred to as U-c/f/es, build further upon the idea of digital cities as this concept represents a convergent form between physical space in the city and online spaces, which is seen as a next generation form of urban space [31]. Some examples are again Helsinki (Helsinki's Virtual Village), Seoul (U-Seoul) and New York (Lower Manhattan project) [59], [31]. U-cities are described as a future city model fostering urban innovations and including urban management, life quality improvement and new industrial development. Finding a match between the needs of the citizens of U-cities and the right ubiquitous services is put forward as a critical success factor, stressing the importance of the end-user [13], [31]. However, in practice a lot of U-city projects focus only on certain groups of citizens (e.g. youngsters), a finding that weakens the user-centric nature of U-cities [13].

The concept of smarf c/f/es can be viewed as a recognition of the growing importance of digital technologies for a competitive position and a sustainable future. Although the smart city-agenda, which grants ICTs with the task to achieve strategic urban development goals such as improving the life quality of its citizens and creating sustainable growth, has gained a lot of momentum in recent years through the programs of the EU or other organizations such as the Economic Co-operation and Development (OECD), the concept itself continues to be used in different ways and remains quite ambiguous [49]. An important differentiating element with the other c/fy-concepts is the collaboraf/ve aspect between the various city stakeholders, including citizens. In smart cities these collaborative digital environments facilitate the development of innovative applications, starting from the human capital of the city, rather than believing that the digitalization /n se can transform and improve cities. Six main areas can be identified in which these digital innovations should make a difference: smart living, smart governance, smart economy, smart environment, smart people and smart mobility [21]. An important aspect within these innovative applications of ICTs for these six dimensions is the collection of all sorts of data and information by sensors and sensor networks. Under the monicker open dafa, this information is made public and put to use in in smarf c/fy applications and technologies that visualize, transform and utilize this data on public and private screens through the web [48].

Smart cities are built upon the involvement of all relevant stakeholders for an interactive, participatory and information based urban environment, whereas d/g/fal c/f/es, w/red c/f/es or ub/qu/fous c/f/es stress the presence of technological infrastructure needed to become a true smarf c/fy. In other words, a city needs to be d/g/fal, w/red and /nfell/genf in order to become smarf, although being d/g/fal, w/red and /nfell/genf does not automatically imply that the city will become smarf by itself.

This also links the smart city-concept with the L/v/ng Labs-concept and with the quadruple helix-model for innovation. Within the quadruple helix-models, a fourth hel/x is added to the traditional triple helix-model: besides industry, universities and public authorities, end-users are also deeply involved as important stakeholders in the innovation process [4]. Of the four identified quadruple helix innovation models, two are characterized as specific forms of Living Labs, whereas another form is seen as citizen-centered. Within EU programs such as i2010 and Europe 2020, the importance of smart cities is highlighted, and the Living Lab-approach is seen as best practice for attaining these objectives as it is able to structure user involvement by keeping the users continuously involved in making better products and services while their expectations are continuously monitored and reflected upon in a systematic process [49]. Consequently, co-creation between all stakeholders within a smart city-context requires user-driven and user-centric research approaches to replace the more technology-based initiatives, related to the digital city and U-city concepts.

Within this article, we will look deeper into ways to facilitate the involvement of citizens, one of the important stakeholders representing the human capital needed to create smart city innovations. Research is still needed into various aspects regarding this digital collaboration with citizens, such as how to facilitate this collaboration and how to optimize citizen participation, essential aspects to building a smart city. There is clearly a lack of empirical evidence in this research area. We will partly tackle this, focusing on the idea generation and idea selection phases of the innovation process, looking at citizen participation through crowdsourcing through an online city platform, a first step in establishing Ghenf L/v/ng Lab, a city Living Lab in Ghent, Belgium. This is in line with the remark that smarf c/f/es require smarf c/f/zens to be truly inclusive, innovative and sustainable [49], which again stresses the need for adequate methodologies to optimally involve and put to use the smarfness of the citizens. In the next paragraph, we will first look into the shift in user involvement in innovation processes from top-down innovation processes to open innovation with customers and then further explore the concept of crowdsourc/ng.

 

3 User Involvement in Innovation Processes

Within this section we will first give an overview regarding the shift in view on innovation processes in general, from closed to open innovation, with specific attention for the role of users. We will discuss the crowdsourcing concept in the next paragraph as a means to structure this user involvement.

3.1 From Top-down Innovation Processes to Open Innovation with Customers

Trad/f/onal /nnovaf/on processes start from the belief that an innovation is best developed using a top-down approach. This implies that innovations are being developed by the internal R&D department, designers and the marketing department. According to the supporters of this approach, the end-users have little or no knowledge about the technical possibilities nor about the market, making them obsolete in this process. Moreover, the creative mindset of end-users is considered too narrow and limited within the traditional view on innovation [35], [55]. Customers and end-users are only subjected to market research methods and techniques in order to abstract apparent user needs for which an internal team of professionals creates solutions [22]. This approach gives an organization much more control on the intellectual property of the products and services, and the solutions fit better into the strategic plans of the organization [52].

However, this classic approach has certain limitations. Some research suggested that the more innovations are being generated internally, the less successful they are [27]. These limitations lead to an increased attention, both academic as in practice, for sources of innovation outside of the company boundaries [12]. At first, this open /nnovaf/on parad/gm did not explicitly include end-users as possible sources of innovation. However, when looking at the works of especially von Hippel on the topic of Lead Users, dating back to the 70's, in the light of this open innovation framework, it did not take long to consider customers and end-users as actively participating stakeholders within the innovation process. This open /nnovaf/on w/fh cusfomers-paradigm believes that end-users can make a relevant contribution to the development process and therefore involves them actively in the ideation, design and development of solutions to their own needs and problems [42], [62]-[64]. By doing so, companies and organizations hope to develop innovative products and services which are better able to fulfill actual user needs and which stand closer to the market [15]. ICT applications have played an important role as enabler and driver for an easy end-user participation and co-creation. The internet has led to the development of so-called web 2.0-applications that for instance facilitate the collaboration between large groups of people [8]. Open source projects such as Linux and the Mozilla software have already proven that users are able to produce highly innovative products without the help of professional organizations [32], [64].

[34] distinguish three different approaches for involving customers in open innovation. The first approach is the classical lead-user mefhod [62]. This method identifies persons with specific characteristics that can be useful during the NPD-process such as having a need that will be general in the market place in a few years, high use expertise, high product knowledge, dissatisfaction with the current offering or having innovated themselves [57]. These Lead Users will then work with the internal R&D department to develop an innovation. A second approach consists of the foolk/fs for user /nnovaf/on [14] or /nfernef foolk/fs [17]. These are online applications and toolkits such as SDK's (software development kits) which stimulate end-users to develop and create their own innovations. The third approach is the so-called /dea-compef/f/ons [14], [51]. An idea competition consists of an organization launching an online challenge that is broadcast to a wide group of people able to respond to this challenge with ideas or solutions. After the competition ends, the best ideas are selected and the /deafors are often rewarded. A key concept within idea-competitions is crowdsourc/ng. This will be discussed in the next paragraph, as the empirical research within this article focuses on this last approach.

3.2 Crowdsourcing

One of the main theoretical frameworks for the concept of crowdsourc/ng is the work of James Surowiecki [60] and earlier studies by Francis Galton [19]. Surowiecki was amazed about the fact that a group of individuals was able to surpass trained problem solvers in finding solutions to problems of a sometimes very sophisticated nature. His research shows that a group, in the right circumstances, can be smarter than its smartest member. He called this phenomenon fhe w/sdom of crowds. However, to be a smart crowd and to make the mechanism of fhe w/sdom of crowds work, several conditions have to be fulfilled. Each individual must be able to formulate the answer to the question individually, without knowing the responses of the others, starting solely from the knowledge he or she has at that given time, a so-called state of decenfral/zed /nformaf/on [60]. When interaction takes place, the individual answers will be biased by social processes. This lowers the global intelligence of the group [39]. In a world where everything and everyone is interconnected, it is not obvious to have this state of decentralized information. From this perspective, the internet and new ICT applications can be a threat to the wisdom of crowds. Furthermore, the group of problem solvers must be sufficiently diverse and there must be a clever mechanism to transform the individual ideas to a collective decision [60]. So, despite the frequent references to the wisdom of crowds in the literature on open innovation, this principle is not always applicable. On top of that, wisdom of crowds is only proven for objective, measurable information which allows calculating an average. This is not the case in the field of innovation and ideas.

Another concept in this context is Pierre Levy's collecf/ve /nfell/gence [37], [45]. Collective intelligence refers to the intelligent decisions which are made when individuals, contrary to the wisdom of crowds, actually do combine their knowledge. Through social interaction, individual knowledge is shared, corrected, opened, processed, enriched and evaluated. This leads to results which are better than the results of a single individual [46]. In the networked society in which we live today [11], the concept of collective intelligence seems to be more applicable than the wisdom of crowds. The internet and web 2.0 applications allow us to collectively think in fhe global bra/n [24]. A beautiful illustration of this phenomenon is Wikipedia.

In the context of the open /nnovaf/on-framework, these principles can be put into practice in order to give end-users or citizens, as external sources of innovation, a voice and an active role in the NPD-process. Open innovation with users or customers goes beyond the mere identification of the needs of the users, something which can be associated with classical market research. It also involves them in the development of the solutions to these needs. The combination of the open innovation-concept of oufsourc/ng with the concept w/sdom of crowds leads to the so-called process of crowdsourc/ng, a term first used by Jeff Howe [25]. His article The r/se of crowdsourc/ng connects the w/sdom of crowds from Surowiecki and the concept collecf/ve /nfell/gence from Levy. Howe describes crowdsourcing as the phenomenon in which everyday people use their free time to help solving problems. The crowd is described as a big and unknown group of people. Using an online crowdsourcing platform, organizations are looking for creative ideas, answers or solutions, or they delegate certain tasks to a broad, diverse and decentralized network of individuals. In contrast to the lead-user method, crowdsourcing relies on a system of self-selection as end-users decide for themselves whether they participate or not [51].

A concept showing some similarities with crowdsourcing is open source. Biggest differentiator is the absence of any commercial goal for open source and the collaborative aspect, which is absent in most crowdsourcing practices. According to [9], most crowdsourcing initiatives solve problems and/or design products or services to the benefit of companies, this by means of non-experts and amateurs from fhe crowd, whereas open source relies on collections of experts that collaborate towards shared goals in an open manner. It can be argued that open source facilitates a crowdsourcing process in which users with certain expertise or degrees of Lead Userness are attracted to participate through the nature of the tasks to be performed.

Crowdsourc/ng covers a broad range of initiatives which makes it a difficult concept. [20] created a faxonomy of methods by analyzing several crowdsourcing initiatives. A first way of using crowdsourcing is what they call /nfegraf/ve sourc/ng w/fhouf remuneraf/on. This includes free user generated content (e.g. Youtube), wiki's, comments, tagging, etc. A second form of crowdsourcing is selecf/ve sourc/ng crowd w/fhouf assessmenf. These are idea competitions in which the submitted ideas are not visible to the other participants. Often a prize is awarded to one or more winning ideas. Examples of this approach are Innocenf/ve and the Belgian Bra/nspof. When the entries are visible to the other participants and these ideas can be discussed and voted on, [20] call this form of crowdsourcing selecf/ve sourc/ng w/fh crowd assessmenf. The online platform M/jn d/g/faal /dee voor Genf, the case studied in this research, belongs to this category. Websites like Threadless where the entries (designs, pictures,...) of the users are being sold on a central platform are called /nfegraf/ve sourc/ng w/fh success-based remuneraf/on. The so-called m/cro-fasks, finally, use /nfegraf/ve sourc/ng w/fh f/xed remuneraf/on. This does not only outsource creative tasks to the crowd, but also very practical small tasks. It is a kind of freelancing at micro-level. The most cited example of a micro-task platform is Amazon's mechanical turk.

Table 1: Overview types of crowdsourcing with examples

 

 

The research in this article focuses on the /dea generaf/on as well as on the /dea selecf/on by the crowd. The latter form of crowdsourcing occurs less frequently than the former and can be a part of /dea compef/f/ons. The most common form of an /dea compef/f/on consists of a challenge which runs for a certain amount of time (usually around 4 to 26 weeks) in which participants can submit solutions or ideas. Idea competitions are mainly conducted online, which means that the effort of both the organizer and the participant is relatively low and that a lot of people, virtually from around the globe, have access to the initiative. The intellectual property of the submitted ideas and solutions usually becomes property of the organizing organization and is used for their own benefit. The entries are usually invisible to other participants. The evaluation and selection of the entries is mostly performed by an internal team of professionals, but this can also be done by the crowd itself, in most instances by voting. Often, one or more winning entries receive a financial reward or credits on the crowdsourcing platform. This competitive element intends to be a stimulus for the participants to try as hard as they can and to come up with an idea that is as unique as possible [14], [25], [34]. The other form of idea competitions is the already discussed selecf/ve sourc/ng w/fh crowd assessmenf which lets the participants see all the other entries [20]. This allows the users to comment on these ideas, to interact with each other and to evaluate the ideas. This is impossible in a closed idea competition.

Idea competitions usually generate a large amount of ideas. These ideas must be processed and evaluated. This is a very intensive and time consuming labor. Therefore it might be interesting to know the properties of a selection made by the crowd. A competitive model increases the incentive to submit an idea which is as innovative as possible, but it does not allow collaboration between the participants. This is good for the w/sdom of crowds mechanism (decentralized knowledge), but does not allow collecf/ve /nfell/gence [28], [50]. Another challenge for crowdsourcing projects is to keep the motivation of the participants high enough. Some challenges require a considerable amount of time and effort from the participants [1]. In highly competitive market situations it is often difficult to combine a closed company culture with an open innovation strategy. In order to release a challenge, a company must also release a certain amount of internal information if they wish to generate useful entries. When an organization chooses to work with an open feedback system, it needs to release even more information and the submitted ideas become freely available to the competition that way.

One of the reasons why crowdsourcing is a popular innovation method is its ability to bypass the weaknesses of internal innovation methods. Internal teams of professionals have a certain thinking pattern which makes it hard to create disruptive innovations. They are part of a certain structure which determines the way they think. On top of that they do not need to prove themselves that much because there is no, or little, competition involved. This makes it much harder for them to think outside the box [8], [27]. Crowdsourcing might be a solution to these limitations since it addresses a large amount of people outside the company or organization with very different backgrounds and little knowledge about the production processes and technical possibilities, but it has to be organized in a clever way. Users also tend to think within a familiar framework of things they know. Therefore, it is absolutely necessary to think carefully on the methods and incentives that will be used.

[8] argues that 3 things should be taken into consideration when decisions are made by the crowd. Firstly, the composition of the crowdsourced jury must be as diverse as possible. The more diverse the jury is, both in terms of expertise and in socio-economical terms, the better the result will be. The number of judges also needs to be large enough. As the law of Linus from the open source movement says "with enough eyeballs the bugs are shallow" [53]. Secondly, there must be a clever mechanism which allows an additive aggregation of the different opinions. Finally, a system of self-organization is desirable. To illustrate this, Bonabeau refers to Wikipedia and the way it organizes itself through a combination of control, addition, evaluation and approval. The way this mechanism is designed is crucial for the failure or success of the project. Questions as Do all parf/c/panfs have equal power?, Whaf are fhe /nferacf/on poss/b/l/f/es?, Who can see whaf?, etc. require good thinking because they might make or break the project.

There is only little comparaf/ve emp/r/cal research that examines the differences between crowdsourced tasks and tasks performed by professionals, but the few results do indicate that crowdsourcing might have certain advantages [7], [52] for example, examined if there is a difference between ideas which are generated by users (using an idea competition) and ideas from an internal brainstorm by the R&D department. The ideas of both groups were evaluated by the head of the R&D department and the CEO of the company. Their results show that the ideas of the professionals are less innovative and have less benefit for the end-user, but their ideas are more feasible. This is probably because professionals have a better knowledge about the existing production processes. Besides these three quality dimensions, an overall quality index was calculated. The ideas of the users scored significantly higher on this matter. Other comparative research by [30] compared the performance of students to the performance of professionals in generating ideas for new mobile services. This research showed that students seem to generate ideas which are more creative than professionals. [30] argue that this is due to the fact that professionals think too much about the technical possibilities. Finally, [47] showed that ideas generated by the crowd can also be more commercially successful compared to ideas generated internally. Summarizing, crowdsourced ideas seem to be more innovative [30], [52] offer more benefit for the users [52] and are even able to generate a bigger profit [47]. Within our case study, presented in the next chapters, we will assess these preliminary findings in the context of a (smart) city initiative where the crowd consists of the citizens of the particular city. The approach, a first step towards establishing a city Living Lab, takes into account idea generation through crowdsourcing, but also focuses on the stage after the idea generation through selective crowdsourcing. The further stages of the innovation process are not dealt with within this article.

 

4 Methodology

Within the previous chapters, it became apparent that the involvement of citizens into innovation processes through and with regards to ICTs is a key aspect in cities becoming smart(er). This should facilitate sustainable city development though innovation with Living Labs as a structuring concept. Theory and some empirical evidence in the domain of user involvement in innovation processes showed the potential of crowdsourcing for early stages in the innovation process. In order to further aid to the development of systemic processes for generating and utilizing citizen input for smart city innovation, we conducted an empirical case study analysis looking at the generation and selection of ideas generated during a crowdsourcing project M/jn d/g/faal /dee voor Genf (translates as my d/g/fal /dea for Ghenf). This project, as a first step in the smarf c/fy-strategy which aims at establishing a city Living Lab (Ghenf L/v/ng Lab), asked citizens of Ghent how ICT could improve their everyday life in the city. This project meets some essential criteria for this research to provide some valuable results. Firstly, the company or organization should have the need to innovate. Being a city in a rapidly changing environment, both at a technical, economic and social level, the city of Ghent definitely has this need. This is also the reason why Ghent is establishing its own Living Lab in order to become a true smarf c/fy. Secondly, it needed to be a project which used crowdsourcing as a method to gather input from the users. This was the case for ideation as well as selection of the ideas. Furthermore, it was necessary that the organization leading this project was willing to share the results and to give an insight in the selection processes. Also, the crowd needed to have the opportunity to vote on the submitted ideas, but the idea selection by internal professionals should happen independently of this user evaluation. Finally, it was necessary that the organization was large enough so it would be able and willing to make an independent evaluation of the ideas from both selections by internal experts who were not involved in the idea selection before.

The EU-funded SMARTiP-project (cf. the acknowledgement-section) used crowdsourcing as a method to actively involve the citizens in the city policy and to gather ideas on how digital innovation could improve the city. This effort was seen as a first step to engage citizens in the process of establishing a city Living Lab as it allowed them to spontaneously suggest ideas and solutions for city innovation. This made it possible to identify the key issues as felt by the citizens and to discover the level of enthusiasm and engagement of citizens towards online involvement in city innovation in an exploratory manner, allowing to learn lessons in order to pave the way for a more systematic approach in the future. The online interaction platform that was used for this project is called UserVoice (UserVoice.com). On this platform participants can easily submit ideas and are also able to comment on the ideas of the others. Both the ideas and the comments can be rated using a voting system. Each IP address is allowed to vote on three ideas or comments. This way, users need to think carefully about what is really important to them. The more votes an idea gets, the higher the idea is positioned in the list.

The project was launched on the first of April, 2011. It was announced in the weekly press conference of the City of Ghent and an e-mail was sent to all students of Ghent University. The project did not get much press coverage, but was picked up by various social networking sites (Facebook, Twitter and LinkedIn) and it was posted on several blogs. About 5.500 people visited the website and 1.410 people registered themselves (which was necessary to be able to vote, post comments and submit ideas). The incentive to submit ideas was an iPad2, which was awarded to a random participant. It is important to note that the winner of this prize was chosen completely at random and this choice was not determined by the number of votes that an idea had gotten. Because the entries were visible to the other participants, and they could interact with each other on the platform, this project is based on collective intelligence as it was defined by [37]. Participation was based on self-selection. As mentioned above, the traditional media gave the project little coverage, but the URL was distributed through social media and blogs. Therefore it can be expected that the participants' profiles were not completely random, something which is nearly impossible in crowdsourcing and idea competitions. The project generated a total of 128 ideas which will be discussed more in detail in the resulfs-section.

The profess/onals - These 128 ideas were evaluated by representatives of the various stakeholders through an evaluation form. This was done using a COCD box. This method requires the evaluators to evaluate the feasibility and originality of the ideas. Based on this evaluation, the ideas are given the label NOW (a feasible, but not very original idea), HOW (the idea is not feasible at this time, but is very original) and WOW (a very original and feasible idea). The evaluators did this analysis independently of the voting by the crowd on the UserVoice platform. Based on this COCD-analysis, they selected 30 ideas to proceed to the next stages in the innovation development process.

The crowd - Because the selection by the professionals was made regardless of the evaluation by the crowd, this project is a perfect case-study for this research. For the sake of this study, a second selection was performed, based on the votes of the users on the UserVoice platform.

A first analysis of the submitted ideas learned that several ideas were very similar. To allow a good analysis, the identical ideas were merged and the votes for these ideas were added. This process reduced the number of ideas from 128 to 97 unique ideas. To create the selection of 30 ideas by the crowd, there had to be chosen on which variable this would be based. The UserVoice platform offers data on both the total number of votes that an idea received, the number of unique people that voted on the idea and the average amount of votes given by each person who voted on this idea. This study chose to create the user selection based on the total amount of votes on an idea. In total, more than 4.000 votes were cast. Using these data, a user selection of 30 ideas was created. Both the expert and the crowd selections were generated completely independent of each other. Of the 97 original ideas, only 12 ideas occurred in both selections, which means that the majority of both selections differs from each other.

To find out whether, and on what dimensions, the selection by the crowd is different from the selection by professionals, each idea was evaluated on three scales by four experts. These seven point scales measured the extent to which an idea was perceived as innovative, as having a benefit for the users and as feasible within a city context. These three quality dimensions are chosen based on previous research [52], [2], [18], [44]. The dimension /nnovaf/on was interpreted as the degree to which the idea is new in comparison with other existing products or services in a city context. Benef/f fo fhe users was defined as the extent to which this idea meets the needs or concerns of visitors and inhabitants of the city of Ghent and offer a certain added value for them. Regarding the feasibility, the experts were explicitly asked to consider technical aspects, legal issues, policy and economic aspects of the idea. These three quality dimensions were measured on a seven point rating scale where 1 stood for nof new / no benef/f / nof feas/ble and 7 stood for very /nnovaf/ve / a lof of benef/f / very easy fo /mplemenf. The experts also had the opportunity to mark an idea worthless for fhe c/fy counc/l. If this box was checked, the above three variables could be left blank for that idea.

The expert evaluation was performed by four experfs on d/g/fal urban developmenf within the city council: the coordinator and expert of the cell e-Strategy, the head of the strategy and coordination department, the head of innovation at the department of economy and innovation and a business analyst at Digipolis, the ICT company of the City of Ghent. These people are considered to be experts in the sector, to be well aware of digital urban innovations and to have a good knowledge about the technical possibilities. Therefore they should be able to make a proper assessment of the quality dimensions described above. To evaluate the ideas as objectively as possible, the ideas of both selections were mixed randomly and their origin was made unknown. To make sure every expert had the same definition of the quality dimensions, they received a detailed description of these concepts. This is essential for a good, generalizable evaluation [23]. To make the comparison a little easier, the ideas were grouped thematically if possible. At the end of the evaluation, the experts also got the opportunity to add general observations or remarks, which provided some qualitative data.

However, these quality dimensions are not objectively measurable parameters. This endangers the reliability and generalizability of the evaluation. To determine the extent to which the experts agreed on their evaluation, Krippendorffs alpha was used. This is a rigorous reliability index which measures the agreement between two or more evaluators when a series of parameters is assessed on a certain scale. The more similar their opinion is, the higher this index will be and the higher the chance will be that the same results will be obtained by other experts [23], [29], [52]. To calculate this reliability index, a freely available macro for SPSS was used. The Krippendorffs alpha values for the dimensions /nnovaf/veness, benef/f for fhe user and feas/b/l/fy are respectively .025, .215 and .245. These values are rather low, which can be explained by the different background of the experts. The experts each have their own area of expertise (communications, policy, ICT and innovation) which might have affected the results on all three quality dimensions. Other explanations might be that the ideas were thematically very diverse and that it is not easy to estimate to what extent a new product or new service offers added value for the users or citizens [3], [30]. The reliability values show that the agreement on the feasibility of an idea (KAPLHA = .245) and on the benefit for the users (KAPLHA = .215) is much higher than the agreement on the innovativeness of an idea (KALPHA = .025). To be able to analyze the results, the average of the four expert evaluations on each dimension was calculated for each idea.

A first way to examine the differences between the two selections is to compare the average scores for each dimension for both selections. This is done using a t-test. An important condition to be able to use a t-test is that every group needs to be distributed normally. This was tested using a Kolmogorov-Smirnov test. If the data is not distributed normally, a Mann-Whitney test must be used to test this hypothesis. The null hypothesis for this analysis is that there is no difference between the average scores of both selections, for none of the three quality dimensions. Another way to investigate the possible differences between the selection of the crowd and the selection by users is to compare the very best ideas. To examine this, a dummy variable was created for each dimension. This variable was given a value of 1 if the idea scored higher than 4 and 0 if the idea scored 4 or less. This is in analogy to the research of [52]. In a similar way, a fourth variable was calculated which was given a value of 1 if all three dimensions scored higher than 4. Finally, this study examines whether one selection contains more ideas which are marked useless for fhe c/fy counc/l.

The main research questions that are to be tackled are:

• What kind of ideas did the idea generation phase of this project deliver? How were they valued by both the user and expert groups?

• What are the results of the crowdsourced selection of a short list out of the long list of generated ideas? How does this selection compare to the short list generated by experts?

• In what stage does crowdsourcing have the biggest potential? What are the lessons leant in the context of a Living Lab-approach towards smart city-innovation?

 

5 Results

The idea generation-stage through the online platform yielded 128 crowdsourced ideas which could be reduced to 97 unique ideas. These can be divided into the following pre-defined general categories: e-governmenf, hous/ng, mob/l/fy, secur/fy, sporf & recreaf/on and ofher. Three categories received more than 20 idea submissions and thus appeared to be the most fop of m/nd with the participating citizens: e-government, mobility and the ofher-category. The other three categories yielded far less ideas.

In terms of votes, five ideas scored more than 200 votes out of just over 4.000 votes that were cast in total. The first idea a mulf/funcf/onal c/fy appl/caf/on or webs/fe gained 812 votes and 54 comments. D/g/fal /nformaf/on k/osks placed /n fhe c/fy came in second with 662 votes, but yielded the most comments (91). An aufomaf/c sysfem for un/f/ed mob/l/fy received 397 votes (only 1 comment), a d/g/fal op/n/on and feedback plafform for urban projecfs 224 votes (3 comments) and a mob/le app for fhe Ghenf sfreef fesf/val 222 (15 comments).

At first sight, these fop ideas do not seem that extremely innovative and remain quite vague. This finding also holds true for the total sample of ideas as globally, an average of 3.37 on a seven point scale for innovativeness was attained. This is also reflected by the comments of the experts. One stated: "I did not have a WOW-feeling when I reflected on the innovativeness of the ideas". Another commented "A lot of the ideas have to do with my daily job activities and therefore do not seem that innovative to me". So in this respect, the crowdsourcing project seems to have failed to generate truly ground-breaking ideas. The global average scores of benef/f for fhe users (mean = 5.52) and feas/b/l/fy (mean = 4.71), on the other hand, are rather high. Furthermore, there is a negative correlation between the innovativeness of an idea and its feasibility (r = -.66; p = .00 < .01). This was expected and is related to the trade-off between smaller, quicker, easier and often cheaper implementable incremental innovations on the one hand, and bigger, harder and often more expensive disruptive innovations on the other hand [52]. Besides this, there is a positive correlation between the amount of votes on an idea on the UserVoice platform and the extent to which the idea offers benefit to the users (r = .34; p = .01 < .05). Presumably the citizens evaluated the ideas mostly on their perceived benefit and less on the innovativeness (r = -.06; p = .65 > .05) or feasibility (r = -.19; p = .16 > .05) of the idea. A last correlation worth mentioning is the strong correlation between the amount of votes on an idea and the amount of comments on this idea (r = .97; p = .00 < .01). Popular ideas clearly seem to be discussed the most, as already became apparent when looking at the top 5 ideas.

A t-test is used to compare the selection of the crowd to the selection of the professionals. To check if all compared groups are normally distributed a Kolmogorov-Smirnov test was performed. The table below (table 2) shows the results of this test. All groups have a p-value higher than .05 which means all data is distributed normally. This means it is allowed to use a t-test for all dimensions.

Table 2: Checking the normal distribution of all groups using a Kolmogorov-Smirnov test

The comparison of the average scores on innovativeness of both selections using a t-test shows there is no significant difference between the average innovativeness of both selections (t = .96; p = .34 > .05). The average feasibility of the ideas does not seem to differ significantly between the two selections either (t = -.55; p = .58 > .05). The one that does differ significantly, however, is the average benefit for the users (t = 2.28; p = .03 < .05). The ideas selected by the crowd offer significantly more user benefit compared to the ideas selected by the professionals. This confirms the hypothesis that users primarily choose ideas based on their perceived benefit.

Table 3: Differences between the selection by the crowd and the selection by the professionals on the quality

The crowd and the professionals both selected 30 ideas out of the generated 97 unique ideas. Only 12 of these 97 ideas were chosen by both groups. Despite this small overlap, the innovativeness and the feasibility of both selections do not seem to be very different. Only the user benefit differed significantly. When both selections are compared in terms of the average amount of votes on the ideas on the UserVoice platform, the selection by the crowd obviously scores higher since this selection was based on this amount of votes. The average amount of votes on the ideas in the professional selection is 88.60 while the average of the user selection is 132.73. Surprisingly, this difference does not seem to be significant (t = .61; p = .55 > .05). So, despite the fact that only 12 ideas appeared in both selections, the professionals did not select ideas which got significantly less user votes on the UserVoice platform.

Examining the averages of both groups is one way of analyzing the data. An important limitation to this technique is that it reduces all data to a central value, which has as a consequence that very good ideas are compensated by very bad ideas. In order to analyze whether one of the two long l/sfs of ideas contained more very good ideas, another approach was taken. An idea was marked as a fop /dea if it scored higher than 4 on the seven point scale. This was done for each dimension. On top of that, an idea was marked as an overall fop /dea if it scored higher than 4 on all three dimensions. To compare both groups, a chi-square test was used.

For the quality dimension /nnovaf/veness this analysis resulted in 9 top ideas in the user selection and 6 top ideas in the professional selection. This difference is not significant (χ2 = .80; p = .37 > .05). On the quality dimension user benef/f, the user selection contained only top ideas (30). The selection by the professionals contained 28 top ideas. This difference cannot be tested using Chi2 since 2 of the 4 cells (50%) has an fe < 5. On the quality dimension feas/b/l/fy, the user selection contained 18 top ideas and the professional selection contained 22 top ideas. This difference is not significant as well (χ2 = 1.20; p = .27 > .05). Finally, the user selection contained 6 overall top ideas while the professional selection contained 4 overall top ideas. The Chi2 test shows, however, that this difference is not significant as well (χ2 = 0.48; p = .49 > .05). Despite the fact that none of the above differences is significant, the results suggest that a selection by users is slightly more innovative, offers more user benefit and contains more overall top ideas. The professionals, on the other hand, tend to select ideas which are more feasible. This is also noted in one of the remarks of the experts: A fyp/cal aspecf of crowdsourc/ng /s fhaf fhey do nof feel /nh/b/fed by further developmenf, nor budgef.

On the amount of worthless ideas in each selection, the hypothesis would be that there will be no worthless ideas in the professional selection since this selection is based on an evaluation by people of the city council and relevant stakeholders. Despite this, both selections contained exactly 3 ideas which were marked as useless for fhe c/fy counc/l by at least one of the experts, so there is no significant difference between the amount of worthless ideas in both selections.

 

Table 4: Differences between the amount of top ideas in both selections

 

6 Discussion and Conclusion

Within the philosophy of a true smarf c/fy, the so-called smarf c/f/zen is seen as an important stakeholder and contributor. However, what are the limits of the smarfness of these citizens? How boffom-up can innovation be in the context of smart cities? What roles can the citizen as a stakeholder play within the smart city ecosystem?

Previous research has already demonstrated that specific citizens can be involved successfully within innovation processes: citizens with Lead User-characteristics [62], [36], [6]. However, it was also argued that involving Lead Users is not always a very easy and straight-forward process [56]-[58], and that for example self-assessment is not the optimal method for Lead User-identification [5]. Other research suggested crowdsourcing as a solution [52]. By broadcasting the idea to a larger group of people (the crowd), a process of self-selection is triggered. This method of user involvement can also be seen as more user dr/ven than Lead User-methods, which remain quite company-, organization- or government-driven, dependent on the initiator of the process. Previous research already suggested that in some instances crowdsourcing within the ideation stages generates ideas that are more novel and with a higher user benefit than experf /deas, although the ideas from the experts score higher in terms of feasibility [52], but this kind of crowdsourcing initiative has not been researched in the context of smarf city innovation. Another aspect has also received not much attention in terms of empirical research. As crowdsourcing initiatives such as idea contests sometimes generate a whole bunch of ideas and suggestions, the selection and evaluation of these results from the crowdsourcing process remains a time-consuming task. Therefore, a crowdsourcing approach towards idea selection, briefly selecf/ve sourc/ng, also seems a valid and promising approach for smart city innovation.

To start tackling these issues, a case study analysis was carried out based on a research project set-up within the city of Ghent, Flanders (the Northern part of Belgium). Through various city channels, citizens were asked to submit their ideas with regards to ICTs and how they could benefit the quality of life of the citizens of Ghent through an online platform. This initiative was undertaken as a first step towards establishing a city Living Lab (Ghent Living Lab). The role of the citizens was multifold, as besides submitting ideas, as every citizen could also cast three votes on the preferred ideas and discuss each idea. This way, out of all submitted ideas (128) 30 top ideas could be distinguished. In the meantime, independently of the crowdsourcing exercise, a panel of independent experts also chose their top 30 ideas out of all the generated ideas. A panel of experts from the city of Ghent itself, initiator of the crowdsourcing exercise, rated both the citizen and the expert selection of ideas on three quality dimensions: /nnovaf/veness, user benef/f and feas/b/l/fy. Both the average scores on all dimensions and the amount of fop /deas were compared for each selection.

In general, all selected ideas scored relatively low on innovativeness, which indicates that ideation through crowdsourcing did not yield radical, breakthrough ideas for ICT innovation in a city context. This is in line with previous research results where crowdsourcing also led to mainly incremental ideas [56], [52]. When comparing the selected ideas from the users with those from the external experts, more than half of the selection was different, indicating that users and experts have in the majority of instances different opinions whether an idea is considered fop or not. The results further show that the ideas from a crowdsourced selection offer significantly more user benefit than the ideas of a selection by professionals. Users seem better able to select ideas on their perceived ability to provide solutions to their problems and needs than experts. This also confirms the research comparing idea generation by users with experts where user ideas scored higher in terms of user benefit than expert ideas, whereas expert ideas scored higher in terms of feasibility [30], [52]. This suggests that the dominant dimensions for users and experts are the same when generating ideas as well as when selecting ideas. The differences in terms of feasibility in our research were not significant, but nevertheless hint in the same direction.

One of the weak spots in evaluating ideas is the objectivity and therefore the generalizability, since there is no objectively measurable truth concerning the innovativeness, the user benefit or the feasibility of an idea. The dimensions itself are abstracted from previous research, but can be disputed. For example the estimated user adoption or other measures of success could also be included in future studies, although it is doubtful whether users or citizens themselves are able to assess the market potential of ideas. In some studies, the amount of profit an idea generates when it comes to the market is used as a benchmark. However, in the context of smart cities, this variable seems to be quite useless. Instead, other variables, assessing the impact of user-centered innovation, should be used to evaluate and compare different approaches.

Another remark can be made regarding the openness of the crowdsourcing process and the overlap between /deaf/on crowdsourcing and selecf/ve crowdsourcing. Especially if the number of votes has a relation with the chance to get an incentive, this stimulates idea submitters to mobilize fr/ends, fools and fam/ly to support their idea. This would mean that not the best /deafor, but the best mob/l/zer receives the most votes. This reasoning pleads for a system of closed crowdsourcing, which seems the most pure way to apply Surowiecki's wisdom of crowds-principle [60], since only hiding the submitted ideas to the others does create a situation of decentralized knowledge. A closed system is the only system which allows rewarding the best ideas, which is probably the best way to generate truly innovative ideas. On the other hand, an open crowdsourcing approach allows for Levy's process of collective intelligence to take place [37]. One of the interesting things of an open crowdsourcing approach, which is not taken into account in much detail within this study, is the role of the comments on the user platform. When a decision needs to be made by the crowd, it is essential to have a clever system of additive aggregation of the opinions and ideas [8]. Despite the assumption that the conclusion of the comments is probably reflected in the amount of votes on an idea, it is nevertheless interesting to analyze the role, content and benefit of these comments.

These findings have several implications for citizen involvement in the context of smart cities and in the context of city Living Labs. It appears that citizens are quite capable of selecting ideas that score high in terms of user benefit and that they are willing to engage in this process without a large incentive, as could be deducted from the fact that more than 1.000 citizens made the effort of registering on the platform. However, in terms of idea generating itself, a general crowdsourcing approach does not seem to be the way to identify breakthrough or radical ideas. For this purpose, Lead User-methods, compared with brainstorming techniques, seem to be more appropriate, or for example a more selective form of idea generation through crowdsourcing, where citizens with certain characteristics [57] are selected and invited to submit their ideas for specific topics related to their characteristics and interests. In the case of Ghent Living Lab, this will be possible in the future as a profiled panel of citizens is being recruited, starting from the user base that participated in the crowdsourcing project. The ideas generated by selected members from this citizen panel can in a next stage be selected by means of an open call for selective crowdsourcing.

One important element however still needs to be researched more in depth: user motivation. In the present study, citizens were motivated by a randomized incentive which did not motivate users to try and submit the best idea or the most ideas. It did however trigger users to vote, as in comparison with the submitted ideas (120), far more votes were cast (4.000). It remains an interesting research topic for future studies to investigate the optimal motivation for citizens in order to get them to submit their best ideas, which also implies the r/ghf citizens, i.e. those with the best ideas, are triggered by the crowdsourcing process to participate. Previous research suggested that intrinsic motivation, or more personal, creative motivations, are more associated to open source-like projects, whereas extrinsic motivations, or external, rewarding motivations like prizes or career opportunities, are more dominant for crowdsourc/ng-like assignments [33], [58].

Perhaps the idea generation in itself should be conceptualized more as a collaborative project where the participants are rewarded with the possibility of actually taking part in the realization of the idea into a concrete innovation. The selective crowdsourcing process, incentivized with random prizes or rewards, acts as a gatekeeper and selector of the ideas that effectively pass to the stage of development. Ideally, this development process takes place through a Living Lab-approach, involving the crowdsourced ideators, together with other stakeholders, in order to iteratively co-create an innovation and test it amongst citizens, where the gatekeeping crowdsourcers are first in line to try out and provide feedback on the first prototypes of the innovation in development. This is certainly something which should be investigated in future studies and which will be possible within the Ghent Living Lab initiative. Further research should also assess the concrete implementation stages of the generated and selected ideas, something which is left untouched within this paper.

Finally, one of the downsides to handing this selection/gatekeeping process over to the crowd is that this might result in choices conflicting with the goals and policies of the organization itself. However, in the context of cities and city governments, this seems to provide an opportunity rather than being a problem. When the results of the selective crowdsourcing differ fundamentally from the city goals and policies, this indicates a gap between the city and its citizens. By adopting more user-centric approaches for city innovation, this will help city governments and policy makers to stay in touch with the citizens, one of the fundamental aspects in the process of cities becoming true smarf c/f/es.

Acknowledgments

M/jn d/g/faal /dee voor Genf, the case study within this article, was organized by the city of Ghent, a city in Belgium with nearly 250.000 inhabitants, and the IBBT (Interdisciplinary Institute for Broadband Technology, an independent research organization funded by the Flemish government, which encourages and supports innovation in the ICT sector) and is part of the European project SMARTiP (Smart Metropolitan Areas Realized through Innovation & People) which is funded by the European CIP (Competitiveness & Innovation) program. SMARTiP is a project that aims to develop user-centric innovations for smarf c/f/es. This is mainly about the development of new public services. The project focuses on the empowermenf of citizens and on smarf engagemenf. Besides the project in Ghent, similar projects run in Manchester (England), Cologne (Germany), Oulu (Finland) and Bologna (Italy). The goal is that these cities cooperate to empower their citizens and that this active partnership with the citizens shapes city of the future. The projects within SMARTiP are focusing on the topics smarf engagemenf, smarf env/ronmenf and smarf mob/l/fy.

The authors also wish to thank the three anonymous reviewers for their relevant and useful comments and suggestions. These helped a great deal in improving this article significantly.

References

[I] M. Ahonen and K. Lietsala, Managing service ideas and suggestions - information systems into innovation brokering, in Proceedings of the Tekes Haas Conference of Service Innovation, Berkeley, 2007.

[2] T. M. Amabile, S. G. Barsade, J. S. Mueller, and B. M. Staw, Affect and creativity at work, Administrative science quarterly, vol. 50, no. 3, pp. 367-403, 2005.

[3] T. M. Amabile, R. Conti, H. Coon, J. Lazenby, and M. Herron, Assessing the work environment for creativity, Academy of management journal, vol. 39, no. 5, pp. 1154-1184, 1996.

[4] R. Arnkil, A. Jàrvensivu, P. Koski, and T. Piirainen, Exploring the Quadruple Helix. Report of Quadruple Helix Research for the CLIQ Project, Work Research Centre, University of Tampere, Tampere, Finland, 2010.

[5] F. Belz and W. Baumbach, Netnography as a Method of Lead User Identification, Creativity and Innovation Management, vol. 19, no. 3, pp. 304-313, 2010.

[6] V. Bilgram, A. Brem, and K. Voigt, User-Centric Innovations in New Product Development - Systematic Identification of Lead Users Harnessing Interactive and Collaborative Online Tools, International Journal of Innovation Management, vol. 12, no. 3, pp. 419-458, 2008.

[7] I. Blohm, U. Bretschneider, J. M. Leimeister, and H. Krcmar, Does collaboration among participants lead to better ideas in IT-based idea competitions? An empirical investigation, in Proceedings of the 43rd Hawaii International Conference on System Sciences, 2010, pp. 1-10.

[8] E. Bonabeau, Decisions 2.0: the power of collective intelligence, MIT Sloan management review, vol. 50, no. 2, pp. 44-52, 2009.

[9] D. C. Brabham, Crowdsourcing as a model for problem solving: an introduction and cases, Convergence, vol. 14, no. 1, pp. 75-90, 2008.

[10] A. Caragliu, C. Del Bo, and P. Nijkamp, Smart Cities in Europe, In Proceedings of the 3rd Central European Conference on Regional Science, Kosice, 2009, pp. 1-15.

[II] M. Castells, The Rise of the Network Society: The Information Age: Economy, Society, and Culture. Oxford: Wiley-Blackwell, 2011.

[12] H. Chesbrough, Open Innovation. The New Imperative for Creating and Profiting from Technology. Boston:Harvard Business School Press, 2003.

[13] J. H. J. Choi, The city is connections: Seoul as an urban network, Multimedia Systems, vol. 16, no. 1, pp. 75-84, 2010.

[14] W. Ebner, J. M. Leimeister, and H. Krcmar, Community engineering for innovations: the ideas competition as a method to nurture a virtual community for innovations, R & D management, vol. 39, no. 4, pp. 342-356, 2009.

[15] E. Enkel, J. Perez Freije, and O. Gassmann, Minimizing market risks through customer integration in new product development: learning from bad practice, Creativity and Innovation Management, vol. 14, no. 4, pp. 425-437, 2005.

[16] E. Ergazakis, K. Ergazakis, D. Askounis, and Y. Charalabidis, Digital Cities: Towards an integrated decision support methodology, Telematics and Informatics, vol. 28, no. 3, pp. 148-162, 2011.

[17] N. Franke and F. Piller, Value creation by toolkits for user innovation and design: The case of the watch market, The Journal of product innovation management, vol. 21, no. 6, pp. 401-415, 2004.

[18] N. Franke, E. Von Hippel, and M. Schreier, Finding commercially attractive user innovations: a test of lead-user theory, Journal of product Innovation Management, vol. 23, no. 4, pp. 301-315, 2006.

[19] F. Galton, Vox populi, Nature, vol. 75, pp. 450-451, March 1907.

[20] D. Geiger, S. Seedorf, T. Schulze, R. Nickerson, and M. Schader, Managing the crowd: towards a taxonomy of crowdsourcing processes, in Proceedings of the Seventeenth Americas Conference on Information Systems, Detroit, Michigan, 2011, pp. 1-15.

[21] R. Giffinger, C. Fertner, H. Kramar, R. Kalasek, N. Pichler-Milanovic, and E. Meijers. (August, 2007) Smart cities: Ranking of European medium-sized cities. Smart cities. [Online]. Available:http://www.smartcities.eu/download/smart_cities_final_report.pdf.

[22] A. Griffin, and J. R. Hauser, The voice of the customer, Marketing science, vol. 12, no. 1, pp. 1-27, 1993.

[23] A. F. Hayes and K. Krippendorff, Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, vol. 1, no. 1, pp. 77-89, 2007.

[24] R. Hoegg, R. Martignoni, M. Meckel, and K. Stanoevska-Slabeva, Overview of business models for Web 2.0 communities, in Proceedings of GeNeMe, 2006, pp. 23-37.

[25] J. Howe, The rise of crowdsourcing, Wired magazine, vol. 14, no. 6, pp. 1-4, 2006.

[26] T. Ishida and K. Isbister, Digital Cities: Experiences, Technologies and Future Perspectives. Berlin: Springer-Verlag, 2000.

[27] R. Katila and G. Ahuja, Something old, something new: a longitudinal study of search behavior and new product introduction, Academy of management journal, vol. 45, no. 8, pp. 1183-1194, 2002.

[28] M. Krieger and Y. Wang, Ideas2ideas: encouraging constructive ideation. Monterey, California: Stanford University, 2010.

[29] K. Krippendorff, Content analysis: An introduction to its methodology. Thousand oaks, California: Sage Publications, 2004.

[30] P. Kristensson, A. Gustafsson, and T. Archer, Harnessing the creative potential among users, Journal of product Innovation Management, vol. 21, no. 1, pp. 4-14, 2004.

[31] O. Kwon and J. Kim, A methodology of identifying ubiquitous smart services for U-City development, In Ubiquitous Intelligence and Computing Proceedings (J.,Indulska, L. T. Yang, J. Cao, J. Ma, and T. Ungerer, Eds.). Berlin: Springer, 2007, pp.143-152.

[32] K. R. Lakhani and E. Von Hippel, How open source software works: "Free" user to user assistance, Research policy, vol. 32, no. 6, pp. 923-943, 2003.

[33] K. R. Lakhani, and R. Wolf, Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects, In Perspectives on Free and Open Source Software (J. Feller, B. Fitzgerald, S. Hissam, and K. Lakhani, Eds.). Cambridge, Mass: MIT Press, 2005, pp. 3-22.

[34] J. M. Leimeister, M. Huber, U. Bretschneider, and H. Krcmar, Leveraging crowdsourcing: activation-supporting components for IT-based ideas competition, Journal of management information systems, vol. 26, no. 1, pp. 197-224, 2009.

[35] D. Leonard and J. F. Rayport, Spark innovation through empathic design, Harvard business review, vol. 75, no. 6, pp. 102-115, 1997.

[36] C. Lettl, User involvement competence for radical innovation, Journal of engineering and technology management, vol. 24, no. 1-2, pp. 53-75, 2007.

[37] P. Lévy, L'intelligence collective: pour une anthropologie du cyberspace. Paris: La découverte Paris. 1994.

[38] L. A. Lievrouw, New media design and development: diffusion of innovations versus social shaping of technology, In The handbook of new media (L. A. Lievrouw and S. Livingstone Eds.). London: Sage Publications, 2006, pp. 246-265.

[39] J. Lorenz, H. Rauhut, F. Schweitzer, and D. Helbing, How social influence can undermine the wisdom of crowd effect, in Proceedings of the National Academy of Sciences, 2011, pp. 9020-9025.

[40] E. Loukis, Y. Charalabidis, and J. Scholl, Editorial of the Special Issue on Digital Cities, Telematics and Informatics, vol. 28, no. 3, pp. 144-147, 2011.

[41] P. Mechant, I. Stevens, T. Evens, and P. Verdegem, E-deliberation 2.0 for smart cities: a critical assessment of two idea generation cases, International Journal of Electronic Governance, to be published.

[42] J. Matthing, B. Sanden, and B. Edvardsson, New service development: learning from and with customers, International Journal of Service Industry Management, vol. 15, no. 5, pp. 479-498, 2004.

[43] C. A. Middleton and A. Bryne, An exploration of user-generated wireless broadband infrastructures in digital cities, Telematics and Informatics, vol. 28, no. 3, pp. 163-175, 2011.

[44] C. P. Moreau and D. W. Dahl, Designing the solution: The impact of constraints on consumers' creativity, The Journal of consumer research, vol. 32, no. 1, pp. 13-22, 2005.

[45] S. Murugesan, Understanding Web 2.0, IT Professional, vol. 9, no. 4, pp. 34, 2007.

[46] J. F. Noubel. (2004, November) Collective intelligence, the invisible revolution. TheTransitioner. Geraadpleeg. [Online]. Available:http://www.earthintelligence.net/dynamaster/file archive/070118/14da9d70ab635fb6f161a44f08dd75/Noubel%20on%20Collective%20Intelligence.pdf.

[47] S. Ogawa and F. T. Piller, Reducing the risks of new product development, MIT Sloan management review, vol.47, no. 2, pp. 65-71, 2006.

[48] T. Ojala, V. Valkama, H. Kukka, T. Heikkinen, T. Lindén, M. Jurmu, F. Kruger, and S. Hosio, UBI-hotspots: Sustainable ecosystem infrastructure for real world urban computing research and business. In Proceedings of the 2nd International Conference on Management of Emergent Digital EcoSystems, Bangkok, 2010, pp. 196-202.

[49] K. A. Paskaleva, The smart city: A nexus for open innovation?, Intelligent Buildings International, vol. 3, no. 3, pp. 153-171, 2011.

[50] P. B. Paulus and H. C. Yang, Idea generation in groups: A basis for creativity in organizations, Organizational behavior and human decision processes, vol. 82, no. 1, pp. 76-87, 2000.

[51] F. T. Piller and D. Walcher, Toolkits for idea competitions: a novel method to integrate users in new product development, R & D management, vol. 36, no. 3, pp. 307-318, 2006.

[52] M. K. Poetz and M. Schreier, The value of crowdsourcing: can users really compete with professionals in generating new product ideas?, Journal of Product Innovation Management, vol. 27, no. 2, pp. 245-256, 2012.

[53] E. Raymond, The cathedral and the bazaar. Knowledge, Technology & Policy, vol. 12, no. 3, pp. 23-49, 1999.

[54] H. Schaffers, N. Komninos, and M. Pallot, Smart Cities as Innovation Ecosystems Sustained by the Future Internet, FIREBALL White Paper, EU, 2012.

[55] A. Schulze and M. Hoegl, Organizational knowledge creation and the generation of new product ideas: A behavioral approach, Research policy, vol. 37, no. 10, pp. 1742-1750, 2008.

[56] D. Schuurman, L. De Marez, and K. De Moor. Identifying and assessing Lead User ideas in order to foster ICT-innovation: Case study digital TV, in Proceedings of International Society for Professional Innovation Management, 2010.

[57] D. Schuurman, D. Mahr, and L. De Marez, User characteristics for customer involvement in innovation processes: Deconstructing the Lead User-concept, in Proceedings of International Society for Professional Innovation Management, 2011.

[58] D. Schuurman, P. Mechant, and L. De Marez, Customer characteristics for open innovation: investigating motivations, nature and lead userness of users within incentivized idea contests, in Proceedings of the World conference on mass customization, personalization and co-creation: bridging mass customization & open innovation, University of California; Haas School of Business. 2011.

[59] D. H. Shin, Ubiquitous city: Urban technologies, urban infrastructure and urban informatics, Journal of Information Science, vol. 35, no. 5, pp. 515-526, 2009.

[60] J. Surowiecki, The wisdom of crowds. New York: Random House, 2005.

[61] United Nations, (2011, August) World Urbanization Prospects: The 2007 Revision Population Database. [Online]. Available: http://esa.un.org/unup/.

[62] E. Von Hippel, Lead users: A source of novel product concepts, Management science, vol. 32, no. 7, pp. 791-805, 1986.

[63] E. Von Hippel, "Sticky information" and the locus of problem solving: Implications for innovation, Management science, vol. 40, no. 4, pp. 429-439, 1994.

[64] E. Von Hippel, Innovation by user communities: Learning from open-source software, MIT Sloan management review, vol. 42, no. 4, pp. 82-86, 2001.

 

Received 1 June 2012; received in revised form 28 July 2012; accepted 9 August 2012