Thursdays, 1.30pm-4.20pm, Fall 2024, Fine Arts building room ACW 102
Instructor: Graham Wakefield g rrr w aaa a t yor ku do t ca
Online resources
Almost everything about this course will be linked from https://alicelab.world/digm5010 -- bookmark this URL -- or it will be in eClass
Synopsis: The purpose of Foundations is to equip and assist students to undertake graduate-level research in Digital Media.
DIGM5010 is foundational core course in the Digital Media Graduate Program (MA/MSc/PhD), co-organized by the Department of Computational Arts in the School of the Arts, Media, Performance & Design, and the department of Electrical Engineering and Computer Science in the Lassonde School of Engineering, York University.
"The Graduate Program in Digital Media provides highly qualified students with the opportunity to do specialised hybrid research work in a program that uniquely combines computational science and artistic practices. Work in digital media focuses on a broad range of current and emerging forms of digitally supported media, with applications that range from computer games to interactive art."
The Digital Media graduate program's academic objectives include promotion of an interdisciplinary approach to computational art-making and technology development, providing students with 21st century “real-world” skills in tandem with research acumen. To work and conduct research in this area means skillfully bridging literacies in art, science and engineering practices.
Developing these research literacies is what DIGM5010 is all about.
The goals of the Foundations course are therefore:
This means:
Establishing these foundations is ultimately evaluated through the potential to understand, transfer, and extend published research in these fields into new creative applications, recreating or mutating established research results, projects, or works to the specific interests of your research area(s) and creative domain(s).
(Reading, writing, making!)
Format and evaluation
Each weekly meeting will vary in format, but always at the heart is the discussion between all of us. This isn't a course with a static syllabus of material -- it is an adaptive living system.
The course essentially has two parallel threads.
In one thread, you will work through the stages of developing a research topic and question into a publishable research paper. This includes several milestone stages (some with course evaluation components for a total of 75% of the course grade).
In the past years, some students have also been successful in submitting their final papers to real academic conferences. For example, in 2024, all 12 students completed their papers, 8 of whom submitted them to the ISEA conference (a leading conference in our field), and 4 of these were accepted and presented at the conference!
In the other thread of the course, we will examine digital media research and research-creation with a focus on practice. We will often dive into collectively coding or algorithmically reconstructing and reinterpreting work drawn from the literature, nestling theory in practice (and practice in theory). The specific topics vary from year to year and are adapted where possible to the interests of the group (and the expertise of the instructor).
You are also expected to bring work to share, making things in response to our discussions and course discussions. You should be keeping your notes, studies and practices documented in a journal, which you will submit at the end of the course. Class participation, making and journalling make up 25% of the course grade.
Note: The grading scheme for graduate study at York is as follows: 90+ A+ (exceptional), 85+ A (excellent), 80+ A- (high), 75+ B+ (highly satisfactory), 70+ B (satisfactory), 60+ C (conditional), <60 F (fail) or I (incomplete).
Schedule
week 1: what is research
week 2: what is computation
| Date | Topic | Activity | Research Paper Workflow |
|---|---|---|---|
| 9/4 | What is Research? | Call for Papers & discussion | Identify motivations, areas, research communities |
| 9/11 | What is Computation? | Computational Sketching | Focused topic selection |
| 9/18 | Introductions | Live Coding | Initial reading list |
| 9/25 | Reproducing Research: Yellowtail | 1-on-1's | Problem/thesis statement |
| 10/2 | Annotated bibliography (outline literature review) | ||
| 10/9 | Paper 1-on-1's | Complete paper outline | |
| Reading Week | Complete first draft | ||
| 10/23 | Paper 1-on-1's | Proofreading, formatting & citations | |
| 10/30 | How to peer review | Peer review session | Revisions |
| 11/6 | Paper 1-on-1's | Revisions | |
| 11/13 | Final paper | ||
| 11/20 | Video/Tutorial | ||
| 11/27 | Final presentations | Post-mortem reflection |
Class recording
Classes are in-person. However, I usually prefer to open a Zoom session for my classes, and use share screen for the most part.
I normally also record sessions and share recordings with the course participants (so long as no participants have objections to this), as I have heard that it has been useful for many students to be able to review sessions after class hours.
Synchronous attendance is highly recommended given the importance of critical discussion on the development of your studies in the program and your research/research-creation; but if you are expecting challenges in attending all times please do not worry. Having the sessions recorded alleviates some inevitable challenges of the graduate situation, and moreover allows some time to review any material if it seemed to fly too fast. You are all coming from different backgrounds so this is also inevitable.
Sep 4, 2023 Class Recording
Hello and welcome!
We are in a program of digital media, co-run by computational arts and computer science. Art & science as "two cultures", meeting through a cybernetic medium of computation.
But perhaps "meeting" is too gentle a word. Sometimes it feels like a tectonic collision. To discover something new, we cannot follow a single path. Collisions between trajectories derail and open into new possibilities, that wouldn't be found if you only followed trend lines or deductive trails.
So, be comfortable to get out of your comfort zone. Learn from each other. It's OK to be awkward. Children are the most awkward, and the fastest learners, so let us be children again from time to time -- so long as we work hard at it!
So for example, what does "computation" mean to you?
What does "research" mean to you?
Deep dive presentation & discussion
Take some time to think about this:
Through this course you will write a research paper and take it through the process of literature review and development, drafting a paper in response to a call for research, and peer review and completion thereof, modeled on real processes of academic research publication. The call for papers we will use is adapted from calls for the SIGGRAPH Art Papers and the ISEA (International Symposium on Electronic Arts).
(Broad terms of call, with theme(s) and topics)
The "DIGM Computational Art/Sci Symposium" acts as a bridge between art and technology to rethink and explore our future. The dream is to bring art and engineering communities together, as has been so fruitful in history. We understand art in its broadest sense, encompassing different fields from fine art to design and architecture. Submissions exploring how computer science and interactive techniques — especially those linked to recent developments — that relate to questions of the future are particularly encouraged. We encourage submissions that discuss and explore within the fields of electronic arts, creative technology, digital culture and all manners of art-science-technology collaboration not yet born. We would like you to engage in the Renaissance of the 21st century!
Topics of interest include, but are not limited by:
(Submission categories)
Prospective authors may consider one of the following categories as they prepare their work for submission.
(Paper requirements)
We invite paper submissions for original and substantial research, both traditional and non-traditional, that has not been previously published. Papers will follow full academic practices, through an extensive literature review, argumentation, and observations or evaluations of impact. All papers must follow academic standards and will undergo blind peer review based on quality, relevance, originality, and impact.
(See notes on the Final Paper submission for details about formatting.)
Submissions will be uploaded online. We also ask you to add a very short text explaining why your submission is important for the academic and wider community, from raising critical issues to opening perspectives or generating solutions. In doing so, we aim to enhance the impact of your cultural contributions.
Authors will be required to present their papers with a duration of 10 minutes.
Sep 11, 2023 Class Recording
This week's paper workflow
Here are some recent conference calls (I have also been sharing these on the DMgrad email list):
Survey responses
From the responses of the survey last week, I can see that some of you have some advanced experience in some programming languages and creative software environments, with Javascript especially as well as Max, Unity being the most frequent ones, and Python, C, GLSL, TouchDesigner, Godot, and Unreal are also mentioned. Some of you have only beginner level experience with these.
Topics of interest mentioned (repeated mentions marked with asterisks):
This is more than we can realistically cover in one semester, especially in a course in which at least half of our time is spent on research writing, but I can certainly do some deep-dives into some of these, and I will try to weave through the topics with the technical platforms mentioned!
I also compiled a list of conferences and journals for some of these topics -- which you might use in your search for related and exemplar papers for the writing:
Goals that you mentioned include:
These are great!
A futorologist said to me: to understand the future, we must go much further into the past; to see patterns that recur and trajectories behind the present. However this doesn't mean entering the future through a rear-view mirror (McLuhan's warning); it is to understand the language and concepts we see the world in today, and see their gaps and limitations.
Let's step back a moment, and understand computation from its genealogical emergence, and conceptual foundations:
What is computation? - History, theory, implementation; programs as data
It can be helful sometimes to step into the shoes of those that have gone before, to see how we ended up here, and what we may have lost or missed along the way.
Take John Maeda, the designer & MIT Media Lab professor, who pioneered reactive graphics in the era of the Macintosh and CD-ROM.
Design By Numbers (John Maeda, 1999)
"Drawing by hand, using pencil on paper, is undisputedly the most natural means for visual expression. When moving on to the world of digital expression, however, the most natural means is not pencil and paper, but rather, computation. Today, many people strive to combine the traditional arts with the computer, and while they may succeed at producing a digitally empowered version of their art, they are not producing true digital art. True digital art embodies the core characteristics of the digital medium, which cannot be replicated in any other."Computation is intrinsically different from existing media because it is the only medium where the material and the process for shaping the material coexiist in the same entity: numbers. The only other medium where a similar phenomenon occurs is pure thought. It naturally follows that computational media could eventually present the rare opportunity to express a conceptual art that is not polluted by textual or other visual representation. This exciting future is still at least a decade or two away. For the moment, we are forced to settle with society's current search for true meaning in an enhanced, interactive version of the art that we have always known."
Maeda studied with Muriel Cooper and Paul Rand, and redefined the use of electronic media as a tool for expression by combining computer programming with traditional artistic technique, which helped lay the groundwork for interactive motion graphics as seen on the web today. (This itself is part of a longer genealogical history that traces back to a movement of thought in the 1960's regarding how computers can augment intelligence, the nature of creativity -- with implications for AI development today)
Other key insights from Maeda's interactive graphics explorations:
Maeda's courses and research in the Aesthetics & Computation group at MIT inspired a whole generation of creative coders. He taught Casey Reas and Ben Fry, and his Design By Numbers software was the precursor of their Processing (which led to P5.js).
Maeda's courses challenged students to rethink the medium from its most basic elements. A typical assignment:
Given a mobile point in space over a finite rectangular area, create a parametric drawing that illustrates repetition, variety, or rhythm. MAS 964 P
Golan Levin was one of Maeda's students, and went on to focus specifically on the creation of audiovisual instruments, responding directly to Maeda's project. He is now a professor at Carnegie Mellon University, and a key figure in the Art & Code community. There’s a lot to draw from his Master's thesis, both theoretically and practically.
(Also note the document structure, as an example of a thesis in our field.)
For example, look at Curly and Yellowtail -- perhaps we can try to recreate this as our first example of "reproducing research".
Let's start with a sketching application -- but let's think about how we can use computation to augment or transform our gestures in some way.
Today's code sketching progress:
Examples from 2024's class
Homework
Continue with your topic research for the paper, beginning to build your reading list.
Please add 2-4 slides to introduce yourself in The Google Slide deck here: https://docs.google.com/presentation/d/1y77v3C8q-2MuFzcvDynw2mqwjO-ng7peo9vcHvXhQtU/
Have a good read of Golan Levin's Master thesis, and note down your thoughts and questions about it for our next coding session!
How would you respond to the example challenge, "Given a mobile point in space over a finite rectangular area, create a parametric drawing that illustrates repetition, variety, or rhythm."?
Sep 18, 2023 Class Recording
Introduction Presentations -- let's get to know each other, and our diverse backgrounds. And let's hear about your research topic areas and questions!
As the first output in the paper writing process, you will prepare a collection of papers related to your research topic/question.
Literature review part 1: Reading List
In this course you will investigate a topic within a sub-area of the digital media realm that intersects with something of value to your own research goals in the program.
In the first phase you will build up a reading list on your chosen topic. Please create an online document that you can share (e.g. a github page, a google doc, etc.) to collect your notes and references as you develop this reading list.
As we saw in class, good research needs a good research question, but that doesn't always become apparent at first. Start wider until you are ready to go deep & narrow. Seek out key papers, conferences, and other key resources for the topic.
Some investigative tips:
cited by and related articles trails to find new leads. Aim to find the most significant (qualitatively & quantitatively) papers. You should be aiming for a collection of 25-50 potentially interesting papers on the topic at this point.
Let's continue with the sketching.
First a quick note -- what we are doing looks a bit like p5.js. In fact, if we remember to refactor code that we will re-use into re-usable functions, then it might start to look even more like p5.js -- maybe we will have
line()andbackground()etc. That's good: we are in the stage of reproducing research. And if we find there are moments where we want to do things a little differently, because of the needs of our project, that's good too -- we aren't limited to what's already given because we know how to remake it, and maybe we'll have a discovery that can advance research!
We saw how we can draw in response to mouse/touch movements, and add generative variation to them.
We talked about Maeda's comment "the most interesting pixels are the mouse" and this represents not just space but also time. How can we use the timing of a drawing gesture to modify the result? Can you think of ways to use speeds, rhythms, echoes, ?
A more complex example, inspired by Paul Haberli's Dynadraw:
Example script from a previous class
With these steps, we should be in a position to attempt to reconstruct Golan Levin's Curly/Yellowtail, for example.
This is an example of reproducing research. First we should sketch out what is required based on the source material, and work from there to refine from a sketch through pseudo-code and implementation of components until we have the final result.
"Yellowtail repeats a user's strokes end-over-end, enabling simultaneous specification of a line's shape and quality of movement. Each line repeats according to its own period, producing an ever-changing and responsive display of lively, worm-like textures."
Detailed description from page 73 of the thesis:
"a user’s linear marks transform into an animated display of lively, worm-like lines. After the user deposited a mark, the system would then procedurally displace that mark end-over-end, making possible the simultaneous specification of both a line’s shape as well as its quality of movement. Straight marks would move along the direction of their own principal axes, while circular marks would chase their own tails. Marks with more irregular shapes would move in similarly irregular, but nonetheless rhythmic patterns."
The " screen space obeyed periodic (toroidal-topology) boundary conditions, such that marks which crossed the edge of the screen would reëmerge on the screen’s opposite side, rather than disappearing altogether."
Notice also the self-observation and critique, see p79. Although this project does not achieve the goal of the thesis, these observations inform the progress that follows. This is a positive research path.
OK so let's start by pseudo-coding Yellowtail!
Here's what we ended up with as pseudo-code in class, before we started coding:
there is a canvas
state:
mouse: x, y, buttonstate
time
currentpath = null
list of finished paths
start position
list of segments (dx, dy change vectors)
pointerdown:
create a new currentpath object, with start position at mouse x,y & t
pointerup:
if currentpath
add my currentpath to the list of finished paths
currentpath = null again
pointermove:
if currentpath exists
add mouse dx,dy & t to currentpath's list of segments
animate:
for each path of finished paths
remove 1st segment (shift)
(something about coordinates)
stick it onto the end (push)
wrap around canvas width/height
e.g. if x > width; x -= width, etc. for 4 boundaries
drawpath:
begin position at path's start position
for each segment of the path
line from last position to new position by adding segment change
(path, moveto, lineto, stroke)
draw:
clear screen
for each path of finished paths
drawpath(line)
if currentpath exists
drawpath(currentpath)And here's the final code we ended up with (with a couple of little refinements added):
Please continue working on extending and mutating this into a new direction! We will share each other's codepens in the next class.
Some rules of thumb while coding:
Use the simplest limits you can -- e.g. limiting yourself to drawing only black lines. More colour, shape and style variations can always be added later. Let's focus on behaviour first.
Break a problem down into sub-problems. Approach the problem from a simpler approximation first -- the simplest version. E.g. make it work in a static way before a dynamic way, or make it work for one, before making it work for many, etc.
Try to work out a problem in pseudocode first -- just write it in commments, draw it on paper, etc, any form that is concrete will help to see the problem more clearly, and diving head first into code isn't often the right thing to do. Once the method becomes clearer, start converting pseudocode into "minimum viable" code.
Use event handlers (draw-frame, mouse, keyboard, timers, ...) to animate and interact with things.
Figure out working conditions logically from basic requirements. E.g. for anything to animate we're going to need to clear the screen on each frame, which means we're going to have to redraw everything every time, which means...
Use state (variables for numbers, strings, objects, arrays) to make things exist over time. Once captured, data can be replayed, modified, etc. Often you can represent state in a few different ways, and the choice will make some processes easier than others.
Test often. Each time you add one minor element, make sure it works for all likely input.
Handle special cases: starting values, boundary cases, error handling...
Don't worry about trying to make anything optimal -- make the most naive way that works, then refine from there.
Use abstractions (functions, objects) to encapsulate and structure ideas more easily & clearly. Any time you feel like you are writing the same code several times, replace it with a function or loop. Separate out reusable "support routines" from the code that represents the main ideas.
Comment the code and use good variable names -- you'll thank yourself in the future when you come back to it! (And anyone else looking at the code will thank you more -- remember research is about sharing!)
Take notes as you go. At any time you might have an idea of a different direction to explore -- you can only do one at once, so write them down! Even if they are just comments in the code.
Make many copies, saving a version (in Codepen you can do this via a Fork) for each minor milestone. If it goes wrong but in an interesting way (a happy accident), save a version of that too.
Sep 25, 2023 Class Recording
Oct 2, 2023 Class Recording
Research paper work: Begin developing your readling list into an annotated bibliography
It was mentioned earlier that, as an alternative to a paper, one option for the final submission is an "explorable explanation". What does that mean?
The term is borrowed from a website collection of Explorable Explanations, which took the term from a 2011 article by Bret Victor. Here's a couple of quotes that are highly relevant to our goals in this course:
"What does it mean to be an active reader? An active reader asks questions, considers alternatives, questions assumptions, and even questions the trustworthiness of the author. An active reader tries to generalize specific examples, and devise specific examples for generalities. An active reader doesn't passively sponge up information, but uses the author's argument as a springboard for critical thought and deep understanding."
This is great advice for a researcher, and great things to do while annotating a bibliography!
"A typical reading tool, such as a book or website, displays the author's argument, and nothing else. The reader's line of thought remains internal and invisible, vague and speculative. We form questions, but can't answer them. We consider alternatives, but can't explore them. We question assumptions, but can't verify them. And so, in the end, we blindly trust, or blindly don't, and we miss the deep understanding that comes from dialogue and exploration."
Against this he suggests creating "Explorable Explanations":
"The goal is to change people's relationship with text. People currently think of text as information to be consumed. I want text to be used as an environment to think in."
He shows a few examples of how we can embed reactive elements and interactive simulations within a document. This isn't just a novelty:
"It's tempting to be impressed by the novelty of an interactive widget such as this, but the interactivity itself is not really the point. The primary point of this example -- the reason I call it an "explorable explanation" -- is the subtlety with which the explorable is integrated with the explanation."
By interacting with these elements we can verify statements, develop intuition, make discoveries, and explore new questions about the topic.
A simple example is his model-driven debate -- drag the green text!
Many examples of Explorable Explanations
Another collection, focused on Complex Systems
An academic journal based on this format is Distill
A platform for "repdroducible research" is Nextjournal -- see examples
A platform for building observable data apps at Observable -- see examples
Oct 9 Class Recording
Oct 23, 2023 Class Recording
Oct 30, 2023 Class Recording
What qualifies as good research? One way of knowing this is to look at how research is reviewed. Again, most grant appplications, but also journals and conference review bodies often publish guidelines for reviewers. These are the criteria by which your work will be evaluated.
Guidelines from
Here are key points:
"The purpose of peer review is to improve the quality of the manuscript under review, and of the material that is eventually published. Conscientious peer review is a time-consuming task but is essential to assure the quality of scientific journals."
Reviews should be conducted fairly and objectively. Personal criticism of the author is inappropriate. If the research reported in the manuscript is flawed, criticize the science, not the scientist. Criticisms should be objective, not merely differences of opinion, and intended to help the author improve his or her paper.
Comments should be constructive and designed to enhance the manuscript. You should consider yourself the authors’ mentor. Make your comments as complete and detailed as possible. Express your views clearly with supporting arguments and references as necessary. Include clear opinions about the strengths, weaknesses and relevance of the manuscript, its originality and its importance to the field. Specific comments that cite line numbers are most helpful.
Begin by identifying the major contributions of the paper. What are its major strengths and weaknesses, and its suitability for publication? Please include both general and specific comments bearing on these questions, and emphasize your most significant points. Support your general comments, positive or negative, with specific evidence.
Is the aim clearly stated? Do the title, abstract, key words, introduction, and conclusions accurately and consistently reflect the major point(s) of the paper? Is the writing concise, easy to follow, and interesting, without repetition?
Are the methods appropriate, sound, current, and described clearly enough that the work could be repeated by someone else? Is the research ethical and have the appropriate approvals/consent been obtained? Are appropriate analyses used? Are they sufficiently justified and explained? Are statements of significance justified? Are results supported by data? Are any of . the results counterintuitive? Are the conclusions supported by the data presented?
Are the references cited the most appropriate to support the manuscript? Are citations provided for all assertions of fact not supported by the data in this paper? Are any key citations missing?
Should any portions of the paper should be expanded, condensed, combined, or deleted?
Do not upload any part of a submitted paper to a cloud service, such as a grammar checker or AI tool; nor should you share it with anyone else.
These are general guidelines, but practices and cultures of value can differ very greatly between different research communities -- and we are often transdiscplinary...
We are running an 'internal review' process, emulating what is frequently done in conference submission review processes. Typically this means:
For our purposes, you will act as both author and reviewer: each of you will act as reviewers for the other students' submissions.
As a structure, our review is based on materials as used by the SIGGRAPH Art Papers review body. Each of you will be randomly assigned up to 3 papers to review.
Nov 6, 2023 Class Recording
If you want to submit your paper to ISEA2025, the deadline is currently Nov 15th at 11am EST; I don't know if that will be extended yet.
For the purposes of the course, please submit your final papers by December 8th, thank you!!
Nov 13, 2023 Class Recording
Research is about sharing. Sometimes, that requires sharing how.
Nov 20, 2023 Class Recording
Nov 27, 2023 Class Recording
We have about 10 minutes per presentation, plus 5 minutes for questions & discussion!
Thank you everyone for a wonderful semester!
Recordings of the weekly sessions will be here: