Alec Perkins Self-Org.

– Eglash

12.6.2007 Final Paper

Crowdsourcing, Evolutionary Design, Their Combination, and Implications Two recent trends, crowdsourcing and evolutionary design, offer many possibilities particularly with regards to design, and both have significant implications for end users and designers alike. Individually, crowdsourcing and evolutionary design are rapidly altering the way things are done. Together, these trends have the ability to offer many more choices for users, potentially completechoice. At the same time, they offer designers tools as replacements for focus groups and methods of rapid idea iteration. Like any tools, there are problems and shortcomings, however the complementary nature of these two trends overcomes many of these separate issues and leads to additional capability. With the increasing demand for custom, personalized designs, tools that allow users to feel a sense of involvement and give them an opportunity to express their desires beyond simply voting with their dollar greatly improve the value of the design to the users. I. Crowdsourcing As defined by Wikipedia, itself a prominent example of the concept, “crowdsourcing” is the outsourcing of a task, generally to a large group of people (the crowd), in the form of an open call.1 Contrasted to more traditional outsourcing which typically is more private, only intended for a specific group or company, crowdsourcingutilizes a public group which can consist of anyone, though Internet access is typically required. Prizes, rewards, or other forms of compensation are not uncommon as incentives to participate, however they are not required for a project to be considered as crowdsourced. Though the impetus for crowdsourced projects is not always self-organized, especially now that it is being used more and more in commercial projects, some projects, such as the stock

1

Wikipedia contributors, "Crowdsourcing," Wikipedia, The Free Encyclopedia, <http://en.wikipedia.org/w/index.php?title=Crowdsourcing&oldid=172251109> (19 November 2007).

1

photography and image site iStockPhoto, do get their start as the result of an at least somewhat self-organized process.2 John Howe, originator of the term crowdsourcing, likens it to distributed computing projects such as SETI@home, which uses the processing power of volunteered computers when not otherwise in use3. A more literal allusion to the similarity between crowdsourcing and distributed computing is the Amazon Mechanical Turk (AMT), a service created by online book (and now just about everything) seller Amazon.com, that provides a framework for both creating crowdsourcing tasks and finding tasks to participate in. Named after the name of a chess playing mechanical device, the AMT service allows jobs, or “Human Intelligence Tasks” (HITs), to include payment, making it possible to earn money by completing tasks as part of a crowd. Tapping into what Amazon refers to as “artificial artificial intelligence,” the service allows the HITs to take advantage of the analytical power of what is essentially a distributed computing system to accomplish tasks such as image content analysis that are still beyond the capabilities of other current computing systems, and it just happens to be using people as the processors instead of silicon. Like distributed computing, the problems that are being outsourced to the crowd generally need to be easily divided into fairly independent parts that can be worked on in parallel. Or, if the problem is not suitable for parallel operations, it is possible to present the problem in its entirety to each user. This has the particular advantage of generating potentially staggering numbers of solutions to the same problem which, with proper filtering, will yield high quality results. An example of this is the Netflix Prize, an ongoing competition initiated in October, 2006by the online DVD rental service Netflix to find an algorithm for DVD recommendations that surpasses the current algorithm in use by 10%, offering up to one million dollars to the finder of such a solution. As of December 5, 2007, the competition had enlisted the help of 23,912 teams, consisting of 29,297 contestants from 165 different countries. To date, there have been 20,091 submissions from just 2,744 teams; according to the competition website, more than 40
2

Howe, Jeff. “The Rise of Crowdsourcing.” Wired June 2006. <http://www.wired.com/wired/archive/14.06/crowds.html>. (19 November 2007).
3

Howe, 2006

2

submissions had been received just in the previous 24 hours. The top-ranking submission so far showed an improvement over the current algorithm by 8.5%, still short of the 10% goal, but enough to earn the submitting team a $50,000 “progress prize.”4 With extreme numbers of contributors, it is possible to use the crowd to handle both the distribution of a problem as well as redundant tasking. Two now famous HITs that were created on the AMT were searches for missing people using very recent satellite images and the ability of people to analyze images and understand them. The most recent, the search for adventurer Steve Fossett, who went missing in a 17,000 square mile region of Nevada, had more than 50,000 volunteers scouring satellite imagery for Fossett’s light aircraft. The tasks were distributed such that each image was analyzed by ten different users, improving the overall quality of the analysis – especially important in an application such as this.56 Crowdsourced tasks that do not offer any sort of direct compensation or incentive, such as the search for Fossett, have the advantage over other tasks in that those contributing their time are very likely doing it because they want to do it, and more often than not are fairly passionate about the task. Naturally, this generally results in higher quality contributions, or at the very least “power users” who contribute significant amounts to the project. One participant, Andy Chantrill,in the search for Fossett analyzed over 5,000 images of 278-square foot sections over a period of 30 hours, at one time working for 13 hours straight. In an interview with Steve Friess of Wired.com, self-described Fossett admirer Chantrill noted in an example of altruism that, "Whether they were [useful] or not, I don't know and will perhaps never know, but

4

“Netflix Prize Leaderboard.” Netflix 5 December 2007. <http://www.netflixprize.com/leaderboard>. (5 December 2007).
5

Friess, Steve. “50,000 Volunteers Join Distributed Search for Steve Fossett.” Wired. 11 September 2007. <http://www.wired.com/software/webservices/news/2007/09/distributed_search>. (5 December 2007).
6

I participated myself, analyzing a little over 1,000 images and flagging two. The analysis of the images did invoke a feeling of being part of a large computer, as well as a sense of futility as the number of available images kept increasing, since the images were more often than not nothing but a patch of brown or grey and the process became very tedious. However, the entire endeavor demonstrated the power of crowdsourcing, as the entire 17,000 square mile area was examined in its entirety at least once, 278 square feet at a time.

3

it's the group effort that counts. Collaborative efforts like these can be extremely powerful tools."7 While the concept of crowdsourcing is not at all new – the “longitude prize” created by the British Parliament in 1714 can be considered as such8 – the relatively recent development of the Internet has allowed it to explode, reaching effectively anywhere on the planet so long as that anywhere has a connection to the network. Crowdsourcing on the scale of the projects mentioned would be impossible without a rapid and widespread communication system unbound by distance like the Internet. Although it may be the most common use of crowdsourcing, the completion of simple, primarily analytical tasks such as image processing or audio track transcribing is just one side. The crowd can be a source for generative input, as websites like Threadless.com and Muji.net demonstrate. Threadless.com sells t-shirts designed and then voted on by users, with highly ranked shirts being put into production.9 In a similar vein, Muji.net, created by the Japanese design brand Muji, uses the crowd as a source for product ideas, which are then taken by the designers, developed, and put back onto the site for the crowd to then choose whether or not to put the product into production. Using a self-organizing decision process through the open market, Muji makes the product available for pre-order and, when receiving 300 orders, begins full manufacturing and eventually puts the products in its brick-and-mortar stores around the world in addition to being made available online.10 The amount and diversity of contributed solutions received, as in the case of the Netflix Prize, can make it impossible to actually get usable results from the mass of input. Fortunately for Netflix, the nature of their competition and goal allows for completely automated evaluation of the submissions, however many crowdsourced projects cannot be evaluated in the same manner because they are not suitable for
7

Friess (2007).

8

Betts, Jonathan. “John Harrison and the Longitude problem.” NMM <http://www.nmm.ac.uk/server/show/conWebDoc.355> (5 December 2007).
9

Threadless.com. <http://www.threadless.com/> (5 December 2007).

10

Boutin, Paul. "Crowdsourcing: Consumers as Creators." Business Week. 13 July 2006. <http://www.businessweek.com/innovate/content/jul2006/id20060713_755844.htm> (1 December 2007).

4

automated processes, the very reason they are being crowdsourced in the first place. Iterating the process can help to resolve this problem, using the crowd itself to then select the better contributions. Proper filters and community management, requiring understanding of social networks, can be expensive. Frameworks such as the AMT and InnoCentive help to simplify the implementation of crowdsourcing, however they also eliminate the feedback loop that systems like Muji.net greatly benefit from. Privacy is also next to impossible with crowdsourcing, as it is not feasible to get a crowd of essentially anonymous users to sign non-disclosure agreements. While website Terms of Service can attempt to impose secrecy on the contributors, the very nature of crowdsourcing conflicts with confidentiality.

II. Evolutionary Design An application of evolutionary computing, evolutionary design is the iterated evaluation of design solutions to a problem that are generated through pseudo-random or genetic processes, with each iteration using individual solutions based on those considered “most fit” from the previous generation. Mimicking biological evolution, evolutionary design relies on chance to generate permutations and variations within a population, which is then evaluated, though instead of environmental selection pressures, fitness functions are employed to yield a quantitative evaluation of the permutation’s success. Such a design process offers numerous advantages, such as the ability to solve problems that are computationally too complex and, given the number of variables, cannot have the correct solution simply calculated directly, and in many cases there is no correct solution, only a relatively most fit solution. Another advantage includes the ability to thoroughly explore a solution space, as the randomness involved means the permutation could end up anywhere within the possible solutions. 11 One famous implementation of evolutionary design was that of Karl Sims, who used a genetic algorithm to evolve locomotion using creatures that were assemblages of various sizes of blocks. The creatures, exemplary of evolutionary artificial life, included an evolved neural net for control, in addition to an evolved form. Through different
11

Parmee, Ian C. Evolutionary and Adaptive Computing in Engineering Design. London: SpringerVerlag, 2001.

5

environments that imposed varying selection pressures on the creatures, surprisingly sophisticated swimming, walking, jumping, and competing behaviors arose.12 Sims’ work highlights the major advantage of evolutionary design in that the simulations resulted in a significant variety of designs, many of which could be considered downright bizarre and would very likely never be considered by a human designer. In an ad hoc pseudo experiment, a genetic algorithm simulation environment was used to explore the capabilities of evolutionary simulations. Running as a screensaver, the breveCreatures simulation,13 inspired by the creatures evolved by Sims, used a genetic algorithm to evolve block creatures, with the goal of moving the farthest distance from an initial point in a set duration. Unlike Karl Sims’ project, however, the breveCreatures simulation does not include the evolution of a control system and instead uses motion between blocks based on the sine function, evident as the blocks would simply oscillate relative to each other, making the locomotion entirely dependent on block size, placement, and connection points. This emphasis on creature morphology had a significant impact on the results of the simulations. Locomotion was observed in a relatively short period of time, about 15 generations, as the creatures did not also have to learn how to move themselves, however there were equally rapid diminishing returns.

12 13

Bentley, Peter J. Evolutionary Design by Computers. San Francisco: Morgan Kaufmann, 1999.

“The breve Simulation Environment.” Spiderland.org. <http://www.spiderland.org/> (9 November 2007).

6

1 Simulation 1 reached a best distance of 43 2 Simulation 2 peaked at a best distance of 69, by generation 123. Screenshot from even after more than 800 additional simulation. generations. Screenshot from simulation.

Two independent simulations were run on separate systems, with no migration. One simulation (figure 1), run for only a short period of time, resulted in a fitness of 40 after about 100 generations, while the other simulation (figure 2) seemed to peak at a fitness of about 69 after approximately 1500 generations, not reaching any higher even after another 800 generations. One surprising result that seemed to contradict the variability expected to be present in the simulations was the fact that the dominant phenotype in both simulations was a three-legged creature, visible in the screenshots from each simulation. While they do have different specific morphologies, the overall configurations are strikingly similar, especially as there was no migration possible. A possible reason for this result is the fact that a three-legged morphology is inherently stablewithout any form of intelligent control. More sophisticated control systems, such as balancing, require feedback, which the sine function control method does not implement in any manner. This highlights the need to be aware of potential biases the implementation of the system may apply to the simulation. The experiment also demonstrated several other disadvantages or potential problems of evolutionary design systems, such as the requirement of a seed form or at least strict parameters, and the need for proper culling of low fitness permutations. The inherent randomness of the process resulted in extreme variability early on in the process that produced extremely ineffective permutations. A seed form allows for more rapid and effective results. While the goal of the breveCreatures simulations was completely autonomous evolution of locomotion and the configurations were relatively simple, involve a much
consisting of just a few blocks that had no significant fitness.

other projects that 3 "Deformed" creatures made appearances very late in the process, typically more complex design space need a seed form beyond the random assemblage of components in order to have results in a reasonable amount of time. Also, the simulations showed that proper limits are needed, as occasionally a creature consisting of either just a few or excessive numbers of blocks attached in an ineffective manner, or 7

even a completely useless individual consisting of a single block, was droppedrather late in the process. Just like biological evolution, these drastic changes among individuals in the population generally had an exceedingly negative effect on the fitness of the creature and were not selected for reproduction, though their continued repeated appearance suggested an occasionally heavy influence of randomness in the algorithm. An endeavor similar to that of Karl Sims, the GOLEM project undertaken by Hod Lipson and Jordan Pollack sought also to evolve both the forms and control systems of artificial creatures, except with the additional goal of automated manufacturing.14 This constraint required careful definition of the allowable components in order for the permutations to remain within the capability of the chosen manufacturing system, a fused deposition rapid prototyping machine. The project resulted in notably complex solutions to horizontal locomotion, but reached a point where “looking at individuals that are generated after many generations, we see highly coupled mechanisms that are difficult to improve on.” They also hypothesized that the simplicity of the environment, and limited ability of the evolved robots to interact with the environment (zero feedback), contributed to the stalling of evolution, 15 matching the results of the breveCreatures simulations. Even though the goal is often to automate a design process, evolutionary design does not have to eliminatehumans. One project that does involve people in an interactive selection process is an industrial design process utilizing genetic algorithms at the Advanced Computing Center, Arts and Design at Ohio State University.16 As computers currently do not understand aesthetics, the system was developed so that a designer could give a seed form, specify parameters, and then act as selector upon a population of
14

Lipson, Hod, and Jordan B. Pollack. "Automatic design and Manufacture of Robotic Lifeforms." Nature 406 (31 Aug. 2000), 974-978.
15

Lipson, Hod, and Jordan B. Pollack. “The Golem Project.” Brandeis University. <http://helen.csi.brandeis.edu/golem/next.html>. 11 October 2000. (4 December 2007).
16

Bezirtzis, Boris, Matthew Lewis, and Cara Christeson. "Interactive Evolution for Industrial Design." C&C’07 (13-15 June 2007). 183-192.

8

evolved permutations of the seed form.17 In this case, the evolution was intended to go in a very specific direction, as evidenced by the need for a designer given seed form. Similar interactive methods are also used to create evolutionary art, with the evolution simulations effectively being treated as a medium, in a style referred to as “evolutionism.”18 These projects are no different from those evolving locomotion, except that instead of selection being based on a quantitative fitness, the selective pressure is that of a qualitative decision being made by a human.

III.Combined As a tool for design, the combination of crowdsourcing and evolutionary design has the potential for exceptional power and influence. Each one is significantly complementary to the other. Whereas crowdsourcing is useful for gathering and filtering ideas, evolutionary design is useful for taking a concept and generating numerous effective permutations of solutions to the concept. As heavily emphasized in the design education of PDI students at RPI, substantial iteration, with as many different ways of doing something as possible, is necessary to make good design decisions. One part of the GOLEM project did combine evolutionary design with a somewhat crowdsourcing aspect. GOLEM@home involved the participation of volunteers the same way as SETI@home, with users contributing spare CPU cycles to run a screensaver version of the simulation environment. While it does loosely fit the definition of crowdsourcing, this component to the project did not take any actual input from the users, instead simply including their computers in the pool of processors running the simulation. With more and more companies using crowdsourcing methods to tailor products and services to users,19 a development system that can be automated while still be effective
17

Lewis, Matthew, and Keith Ruston. "Aesthetic Geometry Evolution in a Generic Interactive Evolutionary Design Framework." New Generation Computing 23.2 (2005): 171-179. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN=17188733&site=eh ost-live>.
18

Bentley, 1999. von Hippel, Eric. Democratizing Innovation. Cambridge: MIT Press, 2005.

19

9

will allow for complete mass customization, which adds significant value to the product,20though such methods are not without their problems. One major criticism of involving crowdsourcing in design, as made evident by many of the comments on an article discussing the crowdsourcing of graphic design, is that it devalues professional design.21 Mark Harmel, a professional photographer who made a living selling stock photographs, was suddenly forced to compete with amateurs at iStockPhoto who were delivering lesser quality work, but at an even more substantially lower price. This resulted in Harmel having to greatly increase his offering of images while charging less in order to maintain even a lower level of business.22 While there is a definite impact to the role of designers in a process that involves both crowdsourcing and evolutionary design, they are still necessary, especially as such frameworks, which are not themselves spontaneous, need careful planning.23 These frameworks are critical as they determine the agency afforded to the users. Too much freedom will be problematic as the products may not be socially responsible in any way, while too little freedom will not produce results any different than current methods. Also, complexity in evolving a manufacturable solution means the final development must still be made by a designer, though improvements in evolutionary methods and manufacturing processes will resolve this issue to some extent. One major pitfall of evolutionary processes is that they are almost exclusively additive and can easily end up including unnecessary, or vestigial, components, so long as those components do not significantly detract from the fitness. And, as evolution does not naturally backtrack through the solution tree, it is possible to be sidetracked on to an alternative branch without careful and specific selection pressures. This points toward
20

Schreier, Martin. "The value increment of mass-customized products." Vienna University of Economics and Business Administration.

21

Crowdsourcing graphic design. Springwise 24 May 2007. <http://www.springwise.com/style_design/crowdsourcing_graphic_design/>. (28 November 2007).
22

Howe, 2006

23

Piller, Frank T. et al. "Overcoming Mass Confusion: Collaborative Customer Co-Design in Online Communities." Journal of Computer-Mediated Communication 10 (4 Aug. 2005). Schreier, Martin. "The value increment of mass-customized products." Vienna University of Economics and Business Administration.

10

the need for a cyclical development process, even more so than is currently implemented, with the user being given the ability to select evolved designs, based on the supplied concept and parameters, and then allowing the evolution process to undergo additional iterations until a satisfactory result is achieved. In many ways, the cycle between crowd and simulation analogues the current product design method, where clients provide specifications for a design team, which then generates possible solutions, and comes back to the client for comments, then repeats, revising the concepts further until one can be picked. However, the more automated nature of evolutionary design means that the entire crowd can be made up of individual clients, which each participant having their own personal design team. As the trend of mass customization indicates, a more fully democratic process, with users able to select preferences beyond just color and extra features and potentially the entire function of a unique product, is inevitable. When combined with evolutionary design, crowdsourcing provides the conduit for the individual user to actively participate and provide feedback to the design process, resulting in entirely personalized products, self-organized product lines, and services.

11

Bibliography Books Bentley, Peter J. Evolutionary Design by Computers. San Francisco: Morgan Kaufmann, 1999. Eiben, A. E., and J. E. Smith. Introduction to Evolutionary Computing. New York: SpringerVerlag, 2003. Parmee, Ian C. Evolutionary and Adaptive Computing in Engineering Design. London: Springer-Verlag, 2001. Terzidis, Kostas. Algorithmic Architecture. Oxford: Architectural Press, 2006. Terzidis, Kostas. Expressive Form: A conceptual approach to computational design. New York: Spon Press, 2003. Tsui, Eugene. Evolutionary Architecture: Nature as a basis for design. New York: John Wiley, 1999. von Hippel, Eric. Democratizing Innovation. Cambridge: MIT Press, 2005.

Articles and papers Bezirtzis, Boris, Matthew Lewis, and Cara Christeson. "Interactive Evolution for Industrial Design." C&C’07 (13-15 June 2007). 183-192. Hempel, Jessi. "Crowdsourcing." Business Week (25 Sep. 2006): 38-39. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 1 December 2007. <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=22374029&site= ehost-live>. Janssen, Patrick. "A generative evolutionary design method." Digital Creativity 17.1 (Mar. 2006): 49-63. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 20379846&site=ehost-live>. Lewis, Matthew, and Keith Ruston. "Aesthetic Geometry Evolution in a Generic Interactive Evolutionary Design Framework." New Generation Computing 23.2 (2005): 171-179.

12

Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 17188733&site=ehost-live>. Lipson, Hod, and Jordan B. Pollack. "Automatic design and Manufacture of Robotic Lifeforms." Nature 406 (31 Aug. 2000), 974-978. Ogawa, Susumu, and Frank T. Piller. "Reducing the Risks of New Product Development." MIT Sloan Managment Review 47, no 2 (2006): 65-71. Piller, Frank T. et al. "Overcoming Mass Confusion: Collaborative Customer Co-Design in Online Communities." Journal of Computer-Mediated Communication 10 (4 Aug. 2005). Schreier, Martin. "The value increment of mass-customized products." Vienna University of Economics and Business Administration. von Hippel, Eric, and Ralph Katz. "Shifting Innovation to Users via Toolkits." Management Science 48 no. 7 (Jul 2002): 821-833. von Hippel, Eric. "Horizontal innovation networks - by and for users." MIT Sloan School of Management Working Paper No. 4366-02 (Jun. 2002).

Web sites Betts, Jonathan. “John Harrison and the Longitude problem.” NMM <http://www.nmm.ac.uk/server/show/conWebDoc.355> (5 December 2007). “The breve Simulation Environment.” Spiderland.org. <http://www.spiderland.org/> (9 November 2007). Boutin, Paul. "Crowdsourcing: Consumers as Creators." Business Week. 13 July 2006. <http://www.businessweek.com/innovate/content/jul2006/id20060713_755844.htm> (1 December 2007). Crowdsourcing graphic design. Springwise 24 May 2007. <http://www.springwise.com/style_design/crowdsourcing_graphic_design/>. (28 November 2007). Friess, Steve. “50,000 Volunteers Join Distributed Search for Steve Fossett.” Wired. 11 September 2007. <http://www.wired.com/software/webservices/news/2007/09/distributed_search>. (5 December 2007). Lipson, Hod, and Jordan B. Pollack. “The Golem Project.” Brandeis University. <http://helen.cs-i.brandeis.edu/golem/next.html>. 11 October 2000. (4 December 2007). Howe, Jeff. “The Rise of Crowdsourcing.” Wired June 2006. <http://www.wired.com/wired/archive/14.06/crowds.html>. (27 November 2007). “Netflix Prize Leaderboard.” Netflix 5 December 2007. <http://www.netflixprize.com/leaderboard>. (5 December 2007).

13

Prospero, Michael. "Designed By You." FC Now 10 May 2007. <http://blog.fastcompany.com/archives/2007/05/10/designed_by_you.html>. (28 November 2007). Threadless.com. <http://www.threadless.com/> (5 December 2007).

Background information Case, K., I. Graham, and R. Wood. "Shape modification using genetic algorithms." Proceedings of the Institution of Mechanical Engineers -- Part B -- Engineering Manufacture 218.7 (July 2004): 827-832. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 13605464&site=ehost-live>. Kicinger, Rafal, Tomasz Arciszewski,and Kenneth De Jong. "Evolutionary computation and structural design: A survey of the state-of-the-art." Computers & Structures 83.23/24 (Sep. 2005): 1943-1978. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 18126229&site=ehost-live>. Kicinger, Rafal, Tomasz Arciszewski,and Kenneth DeJong. "Evolutionary Design of Steel Structures in Tall Buildings." Journal of Computing in Civil Engineering 19.3 (July 2005): 223-238. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 17328622&site=ehost-live>. McAndrew, Patrick, et al. "The evolutionary design of a Knowledge Network to support knowledge management and sharing for lifelong learning." British Journal of Educational Technology 35.6 (Nov. 2004): 739-746. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 14860467&site=ehost-live>. Mehnen, Jörn, et al. "Multi-objective evolutionary design of mold temperature control using DACE for parameter optimization." International Journal of Applied Electromagnetics & Mechanics 25.1/4 (2007): 661-667. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 25215369&site=ehost-live>. Nariman-Zadeh, N., et al. "Evolutionary design of generalized polynomial neural networks for modelling and prediction of explosive forming process." Journal of Materials Processing Technology 164-165 (May 2005): 1561-1571. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 17795035&site=ehost-live>. Nilsson, Anna, and Frida Birath. "Topology Optimization Of A Stamping Die." AIP Conference Proceedings 908.1 (2007): 449-454. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 25209932&site=ehost-live>.

14

Snidal, Duncan, and Lora Viola. "The Evolutionary Design of International Institutions." Conference Papers -- International Studies Association (2007): 1. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 26960041&site=ehost-live>. Tanev, Ivan, Thomas Ray,and Andrzej Buller. "Automated Evolutionary Design, Robustness, and Adaptation of Sidewinding Locomotion of a Simulated Snake-Like Robot." IEEE Transactions on Robotics 21.4 (Aug. 2005): 632-645. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 17967788&site=ehost-live>. Vajna, Sándor, et al. "The Autogenetic Design Theory: An evolutionary view of the design process." Journal of Engineering Design 16.4 (Aug. 2005): 423-440. Academic Search Premier. EBSCO. Rensselaer Polytechnic Institute, Troy, NY. 27 November 2007. <http://search.ebscohost.com.libproxy.rpi.edu/login.aspx?direct=true&db=aph&AN= 18333788&site=ehost-live>. Wikipedia contributors, "Crowdsourcing," Wikipedia, The Free Encyclopedia, <http://en.wikipedia.org/w/index.php?title=Crowdsourcing&oldid=172251109> (19 November 2007). Wikipedia contributors, "Evolutionary computation," Wikipedia, The Free Encyclopedia, <http://en.wikipedia.org/w/index.php?title=Evolutionary_computation&oldid=167740 690> (17 November 2007).

15