DATT4520 & DIGM5520: Generative Art in Mixed Reality / Spatial Computing in Responsive Environments

Spatial Computing in Responsive Environments

Also known as "Generative Art in Mixed Reality" (both titles are used in York U course catalogs)

2024 project

Synopsis: This practice-based course develops computationally literate art practices addressing 3D spaces of mixed reality as creative media, applying the interactive malleability of computation to understand, create and imagine new kinds of artistic, responsive, and generative worlds. That is, it addresses the spatialization of interactive computation, such that every part of peripersonal space around participants, in physical and virtual worlds, is rife with responsive behaviour.

Instructor: Graham Wakefield grrrwaaa at york u dot ca

mixed

Course Information

All course material is available at or linked from this website

Motivation

Aesthetic computation in 3D mixed reality is a knowledge area of increasing importance in the domain of Digital Media. We are moving into an age in which daily immersion within computation is increasingly spatialized, surrounded by ever-denser interactive arrays of sensors, displays, and networked computational devices.

Some say that end-user technology is about to pivot toward augmented visual experiences incorporated with our audio-visual and somatic experience to create a transparent natural human integration with technology that will become a disruptive and enabling transformation for society on par with the personal computer revolution of the 1980s and the Internet revolution of the 1990s. A diversity of industries have been investing heavily in each of these areas, yet acknowledge the necessity of new creative technologies, techniques, genres, and aesthetic practices to carry this work forward into the next generation.

The environments that surround us have a profound influence on shaping human potentials for access and interaction with the world we inhabit, with the virtual information spaces, and with each other. Immersive art is not new, but what differentiates computational immersive media from pre-digital immersive arts is the degree to which it is malleable, interactive, and affordable, allowing the immersion of a massive quantity of human perception and action within flexible, dynamic, and emergent systems.

The goal of the course is to engage with computationally literate art practices that apply the dynamic and interactive malleability of computation to deepen the aesthetic responsiveness of media environments surrounding us. The aim to understand, explore, and pioneer such developments from their arts and science foundations, in both theory and practice.

Format

Class hours are three per session, split between lectures and lab work. Lectures focus on the introduction of theoretical, aesthetic, contextual, and technical content of the course. Labs focus on practice in the form of instructor-led reconstructions, exercises/studies, and larger projects, and will include time for one-on-one/group meetings.

The course is hands-on, to learn about mixed reality world-making in a participatory way, through a collaborative development of a major project. The course is understood to be an experimental undertaking whose success very much depends on everybody's collective enthusiasm and active engagement. It calls for collaboration, diligence, curiosity, open-ness, and immediacy of creative thinking.

Theoretical and technical instruction enables students to develop projects, however the course is conceived as arts-focused and portfolio-centric, with emphasis primarily toward "compositional" and artistic explorations of 3D reality as a generative and responsive medium of expression (whether by producing artworks, or producing systems for creating art). We will critically engage with world makers ranging from the pre-digital immersive arts and early pioneers in artificial realities, to contemporary research-creation collaborations.

Prerequisites / Complements

For undergraduates: Prerequisite of EECS2030, DATT2050, or equivalent. This course complements existing engineering-focused courses such as EECS 4471 with an advanced practice-based focus blending art and technology.

For graduate students: This course complements existing engineering-focused courses such as GS/CSE 6335 Topics in Virtual Reality, GS/CSE 6329 Advanced Human-Computer Interaction and GS/CSE 5323 Computer Vision, with a technologically advanced practice-based creative focus. Within DMG it is complemented by DMG 5200 Experimental Telepresence, DMG 5510 3.0 Physical Computing III, and DMG 5950 3.0 Artificial Life, Generative Art and Creative Code.

Learning outcomes / objectives

At the completion of the course students will:

Evaluation

Undergraduate:

Graduate:

Graduate students are expected to achieve a higher calibre of work and depth of research underlying the realization of the assignments and project.

Practical assignments are assessed by the following criteria:

  1. Execution (20%): How well instructions were followed and conceptual goals of the assignment were met.
  2. Aesthetic qualities (30%): The clear and consistent articulation and composition of a creative whole, and the experiential and/or conceptual depth thereof, within the frame of the given assignment and context of the course.
  3. Technical completeness (30%): Functionality, accuracy, efficiency, creativity, and clear structure in the development and in the results.
  4. Novel contribution (20%): Ingenuity in response to unanticipated challenges, comprehension and creativity beyond what is demonstrated in labs, and vision in further extension.

Reporting & discussion

At the end of each week, each student should submit a report to directly account the development of practical work and learning progress. Reports are also examined for their awareness/understanding and creative extrapolation of issues relating to the specific goals of the project as well as to the broader concerns of the medium.

Participation

(Undergraduate students only - 20% of grade). This incorporates presence, contributions to in-class & online discussions, engagement with the material, and effort and advancement in practical work.

Final Project Presentation

In the final week we will have an open exhibit of the project and final presentation talks, with critical discussion that reflects on the results of the experience gained over duration of the course. The form of the presentations will be determined according to the project structure.

Final Report

All students (undergraduate & graduate) should complete a final report form here

Final Paper

Graduate students only (20% of grade) will provide a final report with the rigor and style of a technical academic paper.

VR Notes

Three.js Notes

WebAudio Notes

Node.js Notes

Git Notes

Class Project Research

Class 1 (Tue)

Class 2 (Wed)

Class 3 (Thu)

Class 4 (Tue)

Class 5 (Wed)

Class 6 (Thu)

Class 7 (Tue)

This is the half-way point through the course

Class 8 (Wed)

Class 9 (Thu)

Class 10 (Tue)

Class 11 (Wed)

Class 12 (Thu)

The paper will provide:

This paper should be formatted as a web page. You can use the gh-pages branch of the project repository to store this paper.

For reference, here are the links to the final paper from the class of 2022: