Settlement Generation Challenge 2021

This is the overview page for the ongoing, fourth iteration of the GDMC AI Settlement Generation Challenge in Minecraft, running in 2021.

The Settlement Generation Challenge is about writing an algorithm that can create a settlement for a given, unknown Minecraft map. The challenge is to produce an algorithm that is adaptive towards the provided map, creates a settlement that satisfies a range of functional requirement - but also looks good and evokes an interesting narrative. The goal is to basically produce an algorithm that can rival the state of the art of what humans can produce. So far there have been 3 iterations of the challenge: once in 2018, and in 2019, and in 2020.

To participate, you have to write a program that can build a settlement for an unknown Minecraft map. After you submit your algorithm, we will run it on 3 previously unseen maps, and a panel of human judges will then evaluate the resulting settlement, based on a set of criteria focusses on Adaptation to the Environment and Terrain, Functionality from an Embodied Perspective, Narrative Integration and Visual Aesthetics.

gdmc-profile.png


Previous Years 2018, 2019, 2020
Conferences FDG
Submission Languages Python, Java, other
Submission Types
  • HTTP Interface via Forge Mod
  • MCEdit Filter
  • Participants Choice
Deadlines:
  • Submission: 31. of May, 2021
  • Results: August, 2021

Overview

To participate, you, or a team, have to write some code that can generate a Minecraft Settlement for an unseen Minecraft map. We accept submission in several frameworks, see below. You submit the code, together with some additional information to us before the deadline. We then apply this code to usually 3 competition maps selected by the organizers, which are not known before the deadline. The resulting settlements will then be sent to a range of expert and volunteer judges, who use Minecraft to look at and interact with the settlements, and then assign scores to each generator. We then announce the results - including who got the highest score and relating the settlements, code, and feedback publicly, both online and at a suitable scientific conference.

Rules

  • To participate you need to submit your code to our website before the deadline.
  • Team submissions are allowed.
  • Anyone is able to participate.
  • One submission per person / team only (We have considered exceptions for academic supervisors who led several teams - please contact us to discuss)
  • Unless you object specifically, we will publish the code and attached documentation after the evaluation period.
  • You keep ownership and rights to your own code and work - but unless you object we assume we can share it and use it in our material.
  • If you have any further questions about the rules, please aks them on our discord, so we can clarify: Join Discord

Winners will be announced during the 2021 Foundation of Digital Games Conference. At the same time we will also publish the detailed evaluation results online for each algorithm. Once the results have been announced we will also publish the competition maps, the settlements generated by the algorithms, and the participants description of their algorithm. We also encourage participants to publish their code, but this will not be mandatory. Unless participants object to sharing their code, we will make it available on our website with an appropriate open source license. If they decline, we will not publish the code. In either case, the organizers will not take exclusive ownership of the code. You can find previous years maps, code, resulting settlement, evaluation scores and other things on our website.

Evaluation

The submitted algorithms will be judged based on a set of 3, different, unknown maps. Example training maps will be provided. The competition maps will not be publicly available until after the competition. Every submitted algorithm will be run on the same 3 unknown maps, and the resulting maps with settlements will be anonymized and given to the judges. The judges will also be able to see the 3 maps in their natural state, before the algorithm was applied. We might change the number of maps, if submissions are getting too high.

The algorithms are given (dependent on the framework) a map and a 3d bounding box they should generate a settlement in. The algorithm should be able to run on maps of varying sizes - and should not run substantially longer than 10 minutes on a regular PC, and not exceed 2 Gbit of RAM for a single map. In the past we have usually used maps of around 256 by 256 blocks, that were generated by the standard Minecraft terrain generator. But the algorithm will be given the exact dimensions of the map when it is run and should be able to adapt to slightly bigger or smaller maps. For example, a competition map might be 200 by 300 in size.

The competition is judged by a set of human judges who are either members of the advisory board or delegates selected by the organization team. If a member of the advisory board or the organization team ends up submitting an algorithm themselves, they will be excluded from the evaluation process for that year, i.e. they will not be allowed to nominate judges or judge themselves.

The plan is to have all judges evaluate all algorithms and average their scores. If this is not possible, due to high number of submissions, there might be several evaluation rounds, where earlier rounds are used to reduce the number of submissions by evaluating a lower number of maps with only a subsection of the judges.

The judges will then evaluate all 3 maps for a given algorithm by spawning at the central point (for the x and z coordinate) of the map. While the judges are in creative mode, they are encouraged to walk through the settlement from a survival perspective. They are given a set of evaluation criteria (see Evaluation Criteria) and are asked to assign points based on the performance of the algorithm across all maps. The sum of the points is the score of the algorithm. The algorithm with the highest score wins.

New Evaluation Methods in 2021

For 2021 we plan to add two modifications to the evaluation in comparison to the 2020 maps.

First, we will include one larger map, possibly around 1000 by 1000 blocks. Here the challenge is to a.) make sure the algorithm still finished in the allotted 10 min. time window and b.) to figure out where to built on the map. Judges will spawn in the middle point of the map, and the generators should ensure in some form (sign, road, etc.) that judges are able to find the generated settlement on the map.

Secondly, since we want to move more towards co-creativity we will have one map that has already some structures build on them, similar to a player starting a settlement. We acknowledge that this is challenging, and are interested in how this can be dealt with. Participants are allowed to ignore, destroy, modify, or built over the existing structure, but we assume that judges will reward adaptivity to existing structures in their scoring.

Evaluation Criteria

We defined a set of evaluation criteria based on what we believe the challenges in moving towards more human-like settlement generation are. So, when evaluation the generated settlements judges are asked to score each generator in four categories: Adaptability, Functionality, Narrative and Aesthetics. For details, see the page on [[Evaluation Criteria]].

Chronicle Challenge

We are also still running the optional bonus challenge: Chronicle Generation. This is an optional challenge, that is evaluated separately from the regular competition. Contestants will have to indicate that their entry is competing in this challenge during submission.
For the chronicle challenge, the generator needs to produce a written book in Minecraft and places it inside the generated settlement – ideally in a place where it is easily found. The book should contain a text that is in some form about the settlement (in English) and contain some form of narrative related to the settlement or how it came about. The entries are evaluated based on how well the given chronicle fits the generated settlement and on overall quality. See the page [[Chronicle Challenge]] for more details on this bonus challenge.

Submission

See valid submission methods page to see all current methods.

Challenges

This section is about the specific challenges in generating a Minecraft settlement on a given map. We believe that these are some of the core obstacles that need to be overcome to move closer to human-level design. We should point out, though, that this list is not exhaustive. It is likely that there are other challenges and obstacles to get to human-like settlement generation. Furthermore, we consider the challenges outlined here ambitious goals to move towards, and do not expect them to be fully solved right away.
The evaluation criteria that will be given to the judges are based on these challenges, because we want participants to address these problems.

Adaptation to Environment and Terrain

Real-life human settlements are adapted towards their surrounding environment and terrain in multiple ways. On a per-building basis, houses and structures reflect the climate conditions they are built in. Furthermore, the environment of a settlement might also dictate the available resources for construction, and certain conditions might require specific types of buildings, such as irrigation, shelter, etc. A house in the cold climate of Edinburgh, for example, might be built sturdy, to protect against wind and cold and keep the warmth in. Likewise, the easy availability of stone might influence the choice of building material.

Real-life settlements are also adapted to the immediate terrain they are in. Natural features, such as mountains or lakes, play a role in the selection of where to settle. Some terrain features can be both positive and negative for a specific settlement, and settlements are often built to take advantage of the benefits and to compensate for the problems. A river, for example, might provide more food and added mobility via ship travel, but might also limit mobility by foot, and be a potential flooding hazard. A settlement might subsequently be built close to the river (or even more likely, at a good location to cross the river), to take advantage of the benefits. The same settlement might also build a dam to protect against flooding, and houses close to the water might sit on stilts. If the settlement dumps waste into the river there might also be a development of different districts, where better, more desirable houses are further up on the river.

On the other hand, humans change the terrain and environment in a limited fashion to better suit their needs. While it might be impossible to remove a mountain or an ocean, it might be possible to build a canal or tunnel.

In summary, human settlements are shaped by and in turn shape the terrain and environment around them. If we study human-built settlements in Minecraft, we can see that these principles are realized in some existing builds. AI settlement generators, on the other hand, struggle to adapt to environment and terrain. They often rely on flattening large areas and then building settlements that do not reflect the surrounding map at all. Similarly, the houses and structures built by them are often templates that get placed as a whole unit, and are in no way reflective of the environment they are in. As part of this challenge, we encourage participants to develop generators that produce different settlements for different kinds of maps, reflecting the available materials, surrounding terrain and environmental conditions. In particular, successful entries should generate settlement that fit into the topography of the map without changing it too much, and build structures that fit into the topography. We also encourage participants of the challenge to think of other ways in which the given maps influences the generated settlement, ideally in such a fashion that it is evident to a human observer what kind of adaptation happened. Ideally, different maps should lead to different settlements, and it should be clear in which way the settlements where shaped by the map. Also note, that there is some overlap with the other three criteria, as their particular solutions are also often ideally adapted to the underlying map.

Functionality from an Embodied Perspective

Minecraft is not just a creativity tool, but also a game. Subsequently, a Minecraft settlement is not just an aesthetic artifact; it also needs to fulfill certain functional roles. One big aspect of this is accessibility and mobility. Is it actually possible to walk through the settlement and reach its different parts? Are there doors into the buildings, are the bridges over the river? How easy is it to navigate the settlement? A second big function a settlement provides is protection from various dangers. Is the settlement well lit, or does it otherwise prohibit mobs from spawning? Does the settlement manage to keep dangerous mobs outside? Are there other forms of defenses and traps to protect the player? Finally, there is also a need to obtain food and process it - so is there an accessible way to produce food and not starve? This list of functional requirements is not exhaustive - it is just meant to give some examples.

This criterion does have a certain overlap with the challenge of adaptivity. Certain functionality only needs to be provided in some cases. For example, bridges compensate for reduced mobility introduced by obstacles. They are an adaptation to reduce the negative effects of the terrain. Other functionality takes advantage of given terrain features, such as river lock further enhancing the mobility of an existing waterway.

This should also be considered, not just from a player perspective, but also under the assumption that the NPC villagers living in the settlement have similar needs. There is crossover here with the next section, because some functionality displayed in typical human-build settlement is not functional in the strict sense. For example, in reality a lot of houses have pitched roof to ensure better rain water runoff. In Minecraft, this is not necessary (as rain does not pool on the ground), but players still build houses with pitched roofs. This narrative functionality alludes to a functionality we understand, yet is not strictly functional in Minecraft. We still encourage the inclusion of such functional elements. Ideally, these narrative functionalities should also be adaptive to the environment. So, pitched roofs should be more popular in a rainy swamp than in a dry dessert, etc.

In summary, the fact that there is an actual player in the world gives us an embodied perspective. The avatar can interact with the world in a variety of different ways, and a settlement can provide different affordances to the player. The challenge in this competition is to produce a generator that can ensure that the settlement provides a maximum of functionality and affordances for the player, while also satisfying the other criteria.

Believable and Evocative Narrative

Human settlements are not just collections of functional buildings, but they also tell a story about how they came about, who are the people living there, and how they see the world. When we look at human-created Minecraft settlements, we can often see how certain human or imagined cultures are reflected in the created buildings. There are cities that resemble ancient Rome, mythical Elven forest outposts, and modern US cities. Often, the buildings also reflect very clear narrative ideas, such as “this is a defensive mining outpost built in a harsh environment”, or “this is a capital city, built to impress foreign and domestic visitors alike”. These settlements are evocative, i.e. they manage to transport this story by looks alone. Relating to the earlier challenge of adaptivity, the better settlements also evoke narratives that work well with the terrain and biome they are in. The narrative they evoke should fit the terrain and environment of the given map, and ideally arise from it. Different maps should result in different narratives.

Human settlements also reflect how they came about. Medieval cores of modern cities, for example, tell a part of a settlement's origin story. This is something rarely seen, even in human-made Minecraft builds, as settlements here are created with the final state in mind. But procedural content generation does have a tradition of simulationist approaches, which would be a possibility for this challenge. An AI could simulate people living and building a settlement over several stages, rebuilding or replacing structures, or slowly modernizing them. This could solve several of the previous problems, as buildings and the overall settlement would be the actual result of an adaptive process, and the produce of an actual sequence of events forming the basis for a possible narrative.

There is also a certain overlap between narrative and functional requirements. How easy the player can obtain wood for building and further process it is a functional requirement. To satisfy this a settlement could have a nearby forest and a workbench nearby. But to make this work with a narrative of a settlement that relies on wood production it might also be good to build a structure that looks like a sawmill, and a street to transport the goods to town. This would not produce any additional functionality to the player (or the digital villagers), but it would tell a narrative that aligns with the functionality provided to the player. Similarly, there could also be elements that have purely narrative functionality, i.e. structures that have a believable fictional use that is not immediately reflected in the game mechanics. For example, a settlement could have an aqueduct to provide water, even though Minecraft characters do not need to drink.

To summarize, the particular challenge here is not just about narrative generation, but also about generating a narrative that fits with the generated settlement, and ensuring that this narrative is communicated to the player with the generated structures. Inspiration by existing, historical or imagined cultures can help to transport this narrative. The aim is to produce something that is both believable and evocative, while still being aligned with the previous criteria.

Visual Aesthetics

Aesthetics are arguably subjective, yet architects and city planners usually follow a range of principles when it comes to designing a settlement or a house. While untrained humans manage to intuitively realize these principles with some success, it is difficult to design an algorithm with automated aesthetic judgment. Existing Minecraft mods that add structures to the world circumvent this problem by hand-designing appropriate templates, and then building a settlement out of those templates. This solution has several problems: it allows for little variation between the buildings, as they all need to be pre-designed, and it also allows for little adaptation of the buildings to the surrounding buildings, the underlying terrain, etc. Buildings are sometimes parameterized, and hence reflect some specific environmental settings, such as available materials or the climate they are in. This makes them more adaptive, but there seems to be trade-off between controlling the exact look of a structure, and making it adaptable towards uncontrolled environmental factors. The challenge in our case is to ensure that buildings still follow basic design principles while being adaptive, functional and evocative.

There is also the further problem that the overall settlement itself should follow certain aesthetic rules. How buildings are aligned and what sight lines exist can play an important role for the overall feel of the settlement. Finally, there is also the question of how well the particular aesthetic expression chosen aligns with the other challenges. For example, a settlement which has a narrative of being a foreboding fortress, and is designed with that functionality, should also have an aesthetic that reflects this.

Bonus Challenge: Chronicle

While cities implicitly tell a story about their inhabitants, cities also often have explicit stories about their history. These chronicles contain notable events that affected and shaped the settlement, such as fires, wars of periods of vibrant trade. The can also provide a lot of additional context, such as what was the function or reason for the settlement during different time periods, or what ideas and philosophies have shaped the settlements development. They also often contain stories about important people that shaped the city. Basically, anything that could be written about a city and would be achieved in the city is appropriate.

For this challenge, your task is to create a chronicle that is currently stored in the city. It should be written as a Minecraft book, and should contain text relating to the generated city. This challenge relates back to several of our earlier criteria, such as adaptivity and narrative. The chronicle should be adapted to the actual generated city, and should the story of that specific city. The less generic this story is, the better. The chronicle challenge is optional, and entries will be evaluated based on the overall quality of the chronicle, and how well it fits with the different cities generated.

Examples

Here is a link of videos of good, human-created settlements that showcase some of the things we are looking for in a generator.

FAQ

Here is a short FAQ.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License