The exploration of immateriality as form is creating a new poetic field with which to narrate space and information.
Designers are increasingly faced with the problem of understanding and visualizing data-filled space and making it inhabitable.
The Process — Co-creation with AI
When it comes to studying a basic artisan technique humans are mostly motivated by curiosity, while the machine is driven by efficiency — an interesting tension worth exploring. With its underlying setup, this machine-learning process is related to the studies ancient sculptors undertook in the very beginning. Reminiscent of the helpers assisting them in the old workshops, the AI developed different strategies seeking constant improvement on its way to form a given object. By feeding it with different tools, rules and rewards through reinforcement learning, we were steering the process, but not the outcome.
The Training – Reinforcement Learning
With the given goal to sculpt a 3D model, the AI was trained through reinforcement learning based on rewards and punishments. The agent, which is defined as a certain machine-learning model, was programmed to seek maximum reward. In the voxel-based environment infinite data and a clear reward structure was provided for the agent to move through. The starting state of the environment was one big cube. Out of this, the agent needed to remove mass to get closer to a predefined target state. With each step, the agent could decide where to go and if to remove a mass of voxels around itself and how. To enable its learning, it was conditioned in a specific way: when extraneous mass was removed it was rewarded and it was given a penalty when mass that ought to be part of the final sculpture was removed. All the technical ML parts were implemented in Unity3D and Unity’s ml-agents library was used for implementing the reinforcement learning.
Detailed information about the setup and the technical pipeline of the training process can be found in this paper.
Trial & Error – Efficiency is the Way
This technical set-up defined the playground for the machine to learn sculpting. Through trial and error the agent created strategies to achieve the desired shape by removing mass from the original cube. By seeking maximum efficiency the sculpting AI was improving rapidly. One very striking outcome was the enormous parallelism in the training process: the AI was able to perform many training sets at the same time, which means a great variety of sculptural output. Observing the evolution of the learning curve including the strategies, behavior and visual output, we started to experiment with different parameters and predefined rules.
In a scenario like this, we collaborate with AI. The outcome is not a reproduction of our creativity, it is unforeseen. We receive unprecedented results, which re-inspire our way of creating, highlighting the importance of exploring the concept of co-creating with AI.
By studying the visual results along the process, our role as a designer became one of a curator: we selected a number of outcomes, which we defined as sculptures to present them as digital artworks. This is where human creativity came to play again: What would we consider as a presentable manipulation of the cube? How would we orchestrate the machine-made output? Which sculptures would we choose and what kind of environment would we create in order to exhibit them?
Visualizing the Process — Traces of Machine Learning
In the search for new ways of creative expression with AI, we experimented by implementing different tools for the agent to choose from. Through the reinforcement learning set-up, this allowed more complexity and resulted in a variety and depth of strategies as well as visual outcomes. We found the use of different tools was vital to the learning process. By regarding the traces of different tools on the surface of the emerging shapes, the sculpting process becomes visible.
Through the Eyes of the Machine — Changing Perspectives
To explore the concept of co-creation even further, we decided to take on the agent’s view. What did the process look like from the creator’s perspective? Through all steps of the process we found multiple vantage points that could be highlighted. In the attempt to interpret the agent’s decision-making we experimented with visualizing AI data such as confidence, or penalty and reward for individual steps within the 3D environment. By highlighting the agent’s path through the block, for example, we gained another perspective on the creation process itself.
The Conclusion — Co-creation with AI is infinite
What have we learned by observing the machine learning process? With AI as a creator, we have not only gained new inspiration, but also new perspectives on the relationship between man and machine in digital art. Above all, we have also questioned our own role as creators. Rather than leaving creation to AI, we need to find ways to integrate it into the creative process. We take technology as a starting point, a tool, a source of inspiration and a creative partner. The human aspect is quite clear in this: We choose the rules and define the approximate output. However, in the end it was the interplay between our human choices and the agent’s ability to find the best solutions and occasionally surprise us. This made the process rewarding to us and shows the true potential of an AI based co-creation process.
By understanding how the machine works, we can reinterpret it, rethink it, and develop new strategies or systems to implement it, enriching our creativity and the creation process on many levels. In the age of artificial intelligence, creative concepts and ideas become more relevant again as we determine and actively shape the environment and process and thus the impact of this AI evolution.
credits
Commissioned by: Weave Magazine
Production: Julia Laub
Creative Direction & Design: Cedric Kiefer
Code: Christopher Warnow