Thursdays, 1.30pm-4.30pm, Fall 2024, Fine Arts building room ACW 102
Instructor: Graham Wakefield grrrwaaa a t york u do t ca
Hello and welcome!
Synopsis: The purpose of Foundations is to equip and assist students to undertake graduate-level research in Digital Media.
DIGM5010 is foundational core course in the Digital Media Graduate Program (MA/MSc/PhD), co-organized by the Department of Computational Arts in the School of the Arts, Media, Performance & Design, and the department of Electrical Engineering and Computer Science in the Lassonde School of Engineering, York University.
"The Graduate Program in Digital Media provides highly qualified students with the opportunity to do specialised hybrid research work in a program that uniquely combines computational science and artistic practices. Work in digital media focuses on a broad range of current and emerging forms of digitally supported media, with applications that range from computer games to interactive art."
The Digital Media graduate program's academic objectives include promotion of an interdisciplinary approach to computational art-making and technology development, providing students with 21st century “real-world” skills in tandem with research acumen. To work and conduct research in this area means skillfully bridging literacies in art, science and engineering practices.
Developing these research literacies is what DIGM5010 is all about.
The goals of the Foundations course are therefore:
This means:
Establishing these foundations is ultimately evaluated through the potential to understand, transfer, and extend published research in these fields into new creative applications, recreating or mutating established research results, projects, or works to the specific interests of your research area(s) and creative domain(s).
(Reading, writing, making!)
Format and evaluation
Each weekly meeting will vary in format, but always at the heart is the discussion between all of us. This isn't a course with a static syllabus of material -- it is an adaptive living system.
In the early part of the course the instructor may bring lecture material or assign readings or research tasks; and we will often dive into collectively coding or algorithmically reconstructing and reinterpreting work drawn from the literature, nestling theory in practice. You are also expected to bring work to share, making things in response to our discussions and course topic. You should be keeping your notes, studies and practices documented in a public journal. (Class participation, making and journalling make up the first 25% of the course grade.)
You can submit codepen sketches through this form
As this goes on, each of you will develop a focus on a specific topic to research in depth, leading to a project documented in a research paper. The project's goal is to functionally reproduce and/or interestingly mutate work from the research community and literature.
We will go through draft processes and a simulated conference submission and review procedure later in the course to help understand academic research structures and refine your work. In the past years, some students have also been successful in submitting their final papers to real academic conferences.
Online resources
Almost everything about this course will be linked from https://alicelab.world/digm5010 -- bookmark this URL!
Previously we have also used a Microsoft Teams account for sharing file materials etc., but I have found it somewhat unreliable/unstable and I am open to alternatives if there are good suggestions.
Remote participation
Regardless whether we are meet in person or remotely, I usually prefer to open a Zoom session for my classes, and use share screen for the most part.
I normally also record all sessions and share recordings with the course participants (so long as no participants have objections to this), as I have heard that it has been useful for many students to be able to review sessions after class hours, including when, due to factors outside their control, they could not attend in class time. Synchronous attendance is highly recommended given the importance of critical discussion on the development of your studies in the program and your research/research-creation; but if you are expecting challenges in attending all times please do not worry. Having the sessions recorded alleviates some inevitable challenges of the graduate situation, and moreover allows some time to review any material if it seemed to fly too fast. You are all coming from different backgrounds so this is also inevitable.
Note: The grading scheme for graduate study at York is as follows: 90+ A+ (exceptional), 85+ A (excellent), 80+ A- (high), 75+ B+ (highly satisfactory), 70+ B (satisfactory), 60+ C (conditional), <60 F (fail) or I (incomplete).
Sep 5, 2023
Hello and welcome!
We are in a program of digital media, co-run by computational arts and computer science. Art & science as "two cultures", meeting through a cybernetic medium of computation.
But perhaps "meeting" is too gentle a word. Sometimes it feels like a tectonic collision. To discover something new, we cannot follow a single path. Collisions between trajectories derail and open into new possibilities, that wouldn't be found if you only followed trend lines or deductive trails.
So, be comfortable to get out of your comfort zone. Learn from each other. It's OK to be awkward. Children are the most awkward, and the fastest learners, so let us be children again from time to time -- so long as we work hard at it!
So for example, what does "computation" mean to you?
Deep dive: What is computation? - History, theory, implementation; programs as data
Take some time to think about this:
Please add 2-4 slides to introduce yourself in The Google Slide deck here: https://docs.google.com/presentation/d/1lvAaIH7lXHz0sc02L7JXVsM_QySOY4SPPdc5tNI5HC4/edit
I said in class we should look at others’ Masters & PhD theses/dissertations for inspiration. Here’s the Masters thesis of Golan Levin, from when he was a student at MIT. He is now a professor at Carnegie Mellon University, and a key figure in the Art & Code community. There’s a lot to draw from this thesis, both theoretically and practically. Have a good read and bring some thoughts and questions for next week: http://www.flong.com/archive/texts/publications/thesis/index.html
Sep 12, 2023
A futorologist said to me: to understand the future, we must go much further into the past; to see patterns that recur and trajectories behind the present. However this doesn't mean entering the future through a rear-view mirror (McLuhan's warning); it is to understand the language and concepts we see the world in today, and see their gaps and limitations. So it can be helful sometimes to step into the shoes of those that have gone before, to see how we ended up here, and what we may have lost or missed along the way.
Take John Maeda, the designer & MIT Media Lab professor, who pioneered reactive graphics in the era of the Macintosh and CD-ROM.
Design By Numbers (John Maeda, 1999)
"Drawing by hand, using pencil on paper, is undisputedly the most natural means for visual expression. When moving on to the world of digital expression, however, the most natural means is not pencil and paper, but rather, computation. Today, many people strive to combine the traditional arts with the computer, and while they may succeed at producing a digitally empowered version of their art, they are not producing true digital art. True digital art embodies the core characteristics of the digital medium, which cannot be replicated in any other."Computation is intrinsically different from existing media because it is the only medium where the material and the process for shaping the material coexiist in the same entity: numbers. The only other medium where a similar phenomenon occurs is pure thought. It naturally follows that computational media could eventually present the rare opportunity to express a conceptual art that is not polluted by textual or other visual representation. This exciting future is still at least a decade or two away. For the moment, we are forced to settle with society's current search for true meaning in an enhanced, interactive version of the art that we have always known."
Maeda studied with Muriel Cooper and Paul Rand, and redefined the use of electronic media as a tool for expression by combining computer programming with traditional artistic technique, which helped lay the groundwork for interactive motion graphics as seen on the web today.
Other key insights from Maeda's interactive graphics explorations:
Maeda's courses and research in the Aesthetics & Computation group at MIT inspired a whole generation of creative coders. He taught Casey Reas and Ben Fry, and his Design By Numbers software was the precursor of their Processing (and thence P5.js).
Maeda's courses challenged students to rethink the medium from its most basic elements. A typical assignment:
Given a point in space over a finite rectangular area, create a parametric drawing that illustrates repetition, variety, or rhythm. MAS 964 P
Golan Levin was one of Maeda's students, and went on to focus specifically on the creation of audiovisual instruments, responding directly to Maeda's project. (For example, look at Curly and Yellowtail.) He is now a professor at Carnegie Mellon University, and a key figure in the Art & Code community. There’s a lot to draw from this thesis, both theoretically and practically. http://www.flong.com/archive/texts/publications/thesis/index.html -- what were your thoughts and questions after reading this? (Also note the document structure.)
Can you continue by drawing an example from Golan Levin's research-creation thesis?
Sep 19, 2023
Deep dive presentation & discussion
Through this course you will write a research paper and take it through the process of literature review and development, drafting a paper in response to a call for research, and peer review and completion thereof, modeled on real processes of academic research publication. The call for papers we will use is adapted from calls for the SIGGRAPH Art Papers and the ISEA (International Symposium on Electronic Arts).
(Broad terms of call, with theme(s) and topics)
The "DIGM Computational Art Symposium" acts as a bridge between art and technology to rethink and explore our future. The dream is to bring art and engineering communities together, as has been so fruitful in history. We understand art in its broadest sense, encompassing different fields from fine art to design and architecture. Submissions exploring how computer science and interactive techniques — especially those linked to recent developments — that relate to questions of the future are particularly encouraged. We encourage submissions that discuss and explore within the fields of electronic arts, creative technology, digital culture and all manners of art-science-technology collaboration not yet born. We would like you to engage in the Renaissance of the 21st century!
Topics of interest include, but are not limited by:
(Submission categories)
Prospective authors may consider one of the following categories as they prepare their work for submission.
(Paper requirements)
We invite paper submissions for original and substantial research, both traditional and non-traditional, that has not been previously published. Papers will follow full academic practices, through an extensive literature review, argumentation, and observations or evaluations of impact. All papers must follow academic standards and will undergo blind peer review based on quality, relevance, originality, and impact.
(See notes on the Final Paper submission for details about formatting.)
Submissions will be uploaded online. We also ask you to add a very short text explaining why your submission is important for the academic and wider community, from raising critical issues to opening perspectives or generating solutions. In doing so, we aim to enhance the impact of your cultural contributions.
Authors will be required to present their papers with a duration of 10 minutes.
As the first step in this process, you will begin with preparing a collection of papers related to your research topic/question.
Literature review part 1: Reading List
In this course you will investigate a topic within a sub-area of the digital media realm that intersects with something of value to your own research goals in the program.
In the first phase you will build up a reading list on your chosen topic. Please create an online document that you can share (e.g. a github page, a google doc, etc.) to collect your notes and references as you develop this reading list.
As we saw in class, good research needs a good research question, but that doesn't always become apparent at first. Start wider until you are ready to go deep & narrow. Seek out key papers, conferences, and other key resources for the topic.
Some investigative tips:
cited by and related articles trails to find new leads. Aim to find the most significant (qualitatively & quantitatively) papers. You should be aiming for a collection of 25-50 potentially interesting papers on the topic at this point. Let's continue from last week's drawing app.
First a quick note -- what we are doing looks a bit like p5.js. In fact, if we remember to refactor code that we will re-use into re-usable functions, then it might start to look even more like p5.js -- maybe we will have
line()andbackground()etc. That's good: we are in the stage of reproducing research. And if we find there are moments where we want to do things a little differently, because of the needs of our project, that's good too -- we aren't limited to what's already given because we know how to remake it, and maybe we'll have a discovery that can advance research!
We saw how we can draw in response to mouse/touch movements, and add generative variation to them.
We talked about Maeda's comment "the most interesting pixels are the mouse" and this represents not just space but also time. How can we use the timing of a drawing gesture to modify the result? Can you think of ways to use speeds, rhythms, echoes, ?
A more complex example, inspired by Paul Haberli's Dynadraw:
Sep 26, 2023
With these steps, we should be in a position to attempt to reconstruct Golan Levin's Curly/Yellowtail, for example.
This is an example of reproducing research. First we should sketch out what is required based on the source material, and work from there to refine from a sketch through pseudo-code and implementation of components until we have the final result.
"Yellowtail repeats a user's strokes end-over-end, enabling simultaneous specification of a line's shape and quality of movement. Each line repeats according to its own period, producing an ever-changing and responsive display of lively, worm-like textures."
Detailed description from page 73 of the thesis:
"a user’s linear marks transform into an animated display of lively, worm-like lines. After the user deposited a mark, the system would then procedurally displace that mark end-over-end, making possible the simultaneous specification of both a line’s shape as well as its quality of movement. Straight marks would move along the direction of their own principal axes, while circular marks would chase their own tails. Marks with more irregular shapes would move in similarly irregular, but nonetheless rhythmic patterns."
The " screen space obeyed periodic (toroidal-topology) boundary conditions, such that marks which crossed the edge of the screen would reëmerge on the screen’s opposite side, rather than disappearing altogether."
Notice also the self-observation and critique, see p79. Although this project does not achieve the goal of the thesis, these observations inform the progress that follows. This is a positive research path.
OK so let's start by pseudo-coding Yellowtail!
Here's what we ended up with as pseudo-code in class, before we started coding:
there is a canvas
state:
mouse: x, y, buttonstate
time
currentpath = null
list of finished paths
start position
list of segments (dx, dy change vectors)
pointerdown:
create a new currentpath object, with start position at mouse x,y & t
pointerup:
if currentpath
add my currentpath to the list of finished paths
currentpath = null again
pointermove:
if currentpath exists
add mouse dx,dy & t to currentpath's list of segments
animate:
for each path of finished paths
remove 1st segment (shift)
(something about coordinates)
stick it onto the end (push)
wrap around canvas width/height
e.g. if x > width; x -= width, etc. for 4 boundaries
drawpath:
begin position at path's start position
for each segment of the path
line from last position to new position by adding segment change
(path, moveto, lineto, stroke)
draw:
clear screen
for each path of finished paths
drawpath(line)
if currentpath exists
drawpath(currentpath)And here's the final code we ended up with (with a couple of little refinements added):
Please continue working on extending and mutating this into a new direction! We will share each other's codepens in the next class.
Some rules of thumb while coding:
Use the simplest limits you can -- e.g. limiting yourself to drawing only black lines. More colour, shape and style variations can always be added later. Let's focus on behaviour first.
Break a problem down into sub-problems. Approach the problem from a simpler approximation first -- the simplest version. E.g. make it work in a static way before a dynamic way, or make it work for one, before making it work for many, etc.
Try to work out a problem in pseudocode first -- just write it in commments, draw it on paper, etc, any form that is concrete will help to see the problem more clearly, and diving head first into code isn't often the right thing to do. Once the method becomes clearer, start converting pseudocode into "minimum viable" code.
Use event handlers (draw-frame, mouse, keyboard, timers, ...) to animate and interact with things.
Figure out working conditions logically from basic requirements. E.g. for anything to animate we're going to need to clear the screen on each frame, which means we're going to have to redraw everything every time, which means...
Use state (variables for numbers, strings, objects, arrays) to make things exist over time. Once captured, data can be replayed, modified, etc. Often you can represent state in a few different ways, and the choice will make some processes easier than others.
Test often. Each time you add one minor element, make sure it works for all likely input.
Handle special cases: starting values, boundary cases, error handling...
Don't worry about trying to make anything optimal -- make the most naive way that works, then refine from there.
Use abstractions (functions, objects) to encapsulate and structure ideas more easily & clearly. Any time you feel like you are writing the same code several times, replace it with a function or loop. Separate out reusable "support routines" from the code that represents the main ideas.
Comment the code and use good variable names -- you'll thank yourself in the future when you come back to it! (And anyone else looking at the code will thank you more -- remember research is about sharing!)
Take notes as you go. At any time you might have an idea of a different direction to explore -- you can only do one at once, so write them down! Even if they are just comments in the code.
Make many copies, saving a version (in Codepen you can do this via a Fork) for each minor milestone. If it goes wrong but in an interesting way (a happy accident), save a version of that too.
Oct 3, 2023
Announcement about an exhibition next week:
Entangled Dimensions: Art in the Age of Neural Media at Illuminarium, Toronto
This immersive exhibition explores the intricate connections between human and non-human intelligences through digital and neural media art. Curated by Ryan Kelln, we invite you to experience the work of Toronto-based artists and dive into a world of cutting-edge art featuring AI-driven creations, EEG/brainwave-powered pieces, 3D-printed marvels, and other pioneering forms of digital expression.
Featured Artworks / Artists:
Presented by the collaborative initiative of @thecreativeschl and @cityofto Creative Technology Office - Future Makers Collective and the Human Feedback Foundation.
Reproducing & sharing research through making: discussion & sharing of your codepens!
Research paper work: Begin developing your readling list into an annotated bibliography
It was mentioned earlier that, as an alternative to a paper, one option for the final submission is an "explorable explanation". What does that mean?
The term is borrowed from a website collection of Explorable Explanations, which took the term from a 2011 article by Bret Victor. Here's a couple of quotes that are highly relevant to our goals in this course:
"What does it mean to be an active reader? An active reader asks questions, considers alternatives, questions assumptions, and even questions the trustworthiness of the author. An active reader tries to generalize specific examples, and devise specific examples for generalities. An active reader doesn't passively sponge up information, but uses the author's argument as a springboard for critical thought and deep understanding."
This is great advice for a researcher, and great things to do while annotating a bibliography!
"A typical reading tool, such as a book or website, displays the author's argument, and nothing else. The reader's line of thought remains internal and invisible, vague and speculative. We form questions, but can't answer them. We consider alternatives, but can't explore them. We question assumptions, but can't verify them. And so, in the end, we blindly trust, or blindly don't, and we miss the deep understanding that comes from dialogue and exploration."
Against this he suggests creating "Explorable Explanations":
"The goal is to change people's relationship with text. People currently think of text as information to be consumed. I want text to be used as an environment to think in."
He shows a few examples of how we can embed reactive elements and interactive simulations within a document. This isn't just a novelty:
"It's tempting to be impressed by the novelty of an interactive widget such as this, but the interactivity itself is not really the point. The primary point of this example -- the reason I call it an "explorable explanation" -- is the subtlety with which the explorable is integrated with the explanation."
By interacting with these elements we can verify statements, develop intuition, make discoveries, and explore new questions about the topic.
A simple example is his model-driven debate -- drag the green text!
Many examples of Explorable Explanations
An academic journal based on this format is Distill
A platform for "repdroducible research" is Nextjournal -- see examples
A platform for building observable data apps at Observable -- see examples
One of the most timely topics of focus today is that of machine learning and artificial intelligence, and particularly now the cohabitation of human/biological and machine agents and intelligences in the world.
For instance, York University recently established The Center for AI & Society organized research unit, and was awarded $80M from federal funding for the Connected Minds program. See the announcement here. There will be many multidisciplinary projects funded through this, for which you might be hired as research assistants.
There are also scholarships for graduate students (and post-doctroral research appointments) through this, to which you may apply:
What are the research questions, and especially, what can our field contribute? Where does this overlap or intersect with your interests? How will you articulate this?
There's lots of different ways we could approach this subject. For the purposes of our DIGM5010 course: what are some of the foundations of our field that may inform this?
Let's follow back through a selective history.
Oct 10
Review of schedule for remainder of semester
Complete your annotated bibliography
Annotated bibliographies due Sunday Oct 13th
Next week is reading week -- a perfect time for you work on your paper's first draft!
Oct 24, 2023
For next week, complete your paper draft. It must be submitted before next week's class - submit the paper with this form
First a quick reminder of what we looked at back in week 4:
Braitenberg, Valentino. Vehicles. 1984 - Can we reproduce a Vehicle world?
Craig Reynolds. Steering Behaviors For Autonomous Characters, Game Developers Conference. 1999 - Can we reproduce a flocking simulator?
Steering behaviours for vehicles and flocks
Our script so far:
Oct 31, 2023
Make sure your paper is submitted (by email to me) before this class!!
What qualifies as good research? One way of knowing this is to look at how research is reviewed. Again, most grant appplications, but also journals and conference review bodies often publish guidelines for reviewers. These are the criteria by which your work will be evaluated.
Guidelines from
Here are key points:
"The purpose of peer review is to improve the quality of the manuscript under review, and of the material that is eventually published. Conscientious peer review is a time-consuming task but is essential to assure the quality of scientific journals."
Reviews should be conducted fairly and objectively. Personal criticism of the author is inappropriate. If the research reported in the manuscript is flawed, criticize the science, not the scientist. Criticisms should be objective, not merely differences of opinion, and intended to help the author improve his or her paper.
Comments should be constructive and designed to enhance the manuscript. You should consider yourself the authors’ mentor. Make your comments as complete and detailed as possible. Express your views clearly with supporting arguments and references as necessary. Include clear opinions about the strengths, weaknesses and relevance of the manuscript, its originality and its importance to the field. Specific comments that cite line numbers are most helpful.
Begin by identifying the major contributions of the paper. What are its major strengths and weaknesses, and its suitability for publication? Please include both general and specific comments bearing on these questions, and emphasize your most significant points. Support your general comments, positive or negative, with specific evidence.
Is the aim clearly stated? Do the title, abstract, key words, introduction, and conclusions accurately and consistently reflect the major point(s) of the paper? Is the writing concise, easy to follow, and interesting, without repetition?
Are the methods appropriate, sound, current, and described clearly enough that the work could be repeated by someone else? Is the research ethical and have the appropriate approvals/consent been obtained? Are appropriate analyses used? Are they sufficiently justified and explained? Are statements of significance justified? Are results supported by data? Are any of . the results counterintuitive? Are the conclusions supported by the data presented?
Are the references cited the most appropriate to support the manuscript? Are citations provided for all assertions of fact not supported by the data in this paper? Are any key citations missing?
Should any portions of the paper should be expanded, condensed, combined, or deleted?
Do not upload any part of a submitted paper to a cloud service, such as a grammar checker or AI tool; nor should you share it with anyone else.
These are general guidelines, but practices and cultures of value can differ very greatly between different research communities -- and we are often transdiscplinary...
We are running an 'internal review' process, emulating what is frequently done in conference submission review processes. Typically this means:
For our purposes, you will act as both author and reviewer: each of you will act as reviewers for the other students' submissions.
As a structure, our review is based on materials as used by the SIGGRAPH Art Papers review body. Each of you will be randomly assigned up to 3 papers to review.
Nov 7, 2023
Paper process discussion -- one week to ISEA deadline: https://isea2025.isea-international.org/submission/submission-guideline/
Continuing with reproducing research: steering behaviours for vehicles and flocks part II. Today's code:
If you want to submit your paper to ISEA2025, the deadline is currently Nov 15th at 11am EST; I don't know if that will be extended yet.
For the purposes of the course, please submit your final papers by December 8th, thank you!!
Nov 14, 2023
ISEA DEADLINE EXTENDED to Nov 30 at 10am EST!
A quick introduction to Node.js
wss.clients.forEach(other => {})other == client)? Or do?state.paths.push(JSON.parse(msg.data.toString()));{ cmd:"add", path: path }For next week:
Research is about sharing. Sometimes, that requires sharing how.
Nov 21, 2023
1 on 1 paper sessions
A quick introduction to Git
Video tutorials due
Nov 28, 2023
We have about 10 minutes per presentation, plus 5 minutes for questions & discussion!
You can submit all of your codepen sketches as we go along through this form.