Week 1: September 9

Discussion of Syllabus
Introduction to Realtime Video
  • Realtime processing
  • Responsive media
  • Examples: Slitscan, Timespace, broken TV, Particles & Fluids
  • Limitations, Strengths


Introduction to Cycling74’s Max
  • How to Install Max and Documentation
  • The Working Environment
  • Max: Objects and Connections
  • MAX BASICS...
  • Sliders and other forms of input
  • Mathematics Operators, Comparisons
  • Inlets, Outlets, Flow of execution, Bang
  • Routing, Gates and Switches
  • Time: Metro
  • Iteration: Uzi
  • Class exercise: pseudo-code, incrementing counter
  • Style: AVOID iterators, indexing into arrays, data structures; USE flow, time-based media

Useful Links:
  • New in Max6 Usability Features
  • Organized Patching with Trigger
  • Functional list of Max objects by type, with links to individual object documentation.

Assignment: 
Tutorial text and patches are embedded within Max6 environment but you can also find the patches inside the your Max6 folder: 
~/Applications/Max6/patches/docs/tutorial-patchers


Week 2: September 16

Max intro continued
  • Arguments
  • Sub Patches
  • lists, zl
  • Presets
  • Style: modularity, commenting, README, automation, robustness
Introduction to Jitter
  • Video Matrices, Layers
  • What is a Matrix?
  • Getting video in: jit.qt.movie, jit.qt.grab
  • jit.matrix, jit.unpack, jit.pack
  • Displaying video: jit.pwindow
  • Jitter Attributes
  • Processing example: feedback Jitter Tutorial 18
Assignment: 
Lab:



Week 3: September 23

Intro to Open Sound Control (OSC)
  • Enhances your code's organization, documentation, and flexibility
  • Open-ended, dynamic, URL-style symbolic naming scheme
  • Pattern-matching language to specify multiple recipients of a single message
  • High resolution time tags, and high-resolution data

  • "Bundles" of messages whose affect must occur simultaneously
  • Query system to dynamically determine the capabilities of an OSC server and get documentation
  • Biggest problem: no standardization of OSC address patterns, interconnectivity of devices
OSC Links / Readings:
  • User level overview of OSC in this 2003 Paper
  • More in depth paper on latest specifications here


Introduction to Jamoma
  • VIdeo i/o
  • Basic video operations
  • Effects


Class Exercise:
  • Editing and handling OSC messages (prepend, append, sprintf, etc)

Lab: 
Define Etude 1: Modify  Live | Recorded Video by combining or by applying an effect, provide  control parameter(s)
Assignment: Jitter Tutorials 3 - 5



Week 4: September 30

Jamoma
Filters, Effects

Lab: Work on Etude 1: Modify  Live | Recorded Video
Assignment: Jitter Tutorials 6 - 10



Week 5: October 7

Advanced Video Processing
  • Submatrix
  • Scissors and Glue
  • Downsampling
  • Repositioning
  • jit.op
  • jit.expr
  • example: bfg
Temporal Processing
  • jit.slide
  • jit.matrixset
Lab
  • Present Etude 1: Modify  Live | Recorded Video, with parameter
  • Live camera in physical space (EV 7.725), Lighting
  • Define Etude 2: Gesturally modulate video


Readings:

 


Week 6: October 21

Extracting Statistics
Tracking value
  • Using color space information
  • Example: creating a light Theremin
  • Value tracking: Jit.findbounds
  • Masking techniques
Tracking change
  • Frame Differencing
  • Background subtraction

Lab:
  • Work on Etude 2: Gesturally modulate video


Related Articles:




Week 7: October 28

Introduction to the Computer Vision Library: cv.jit
  • Installation
  • Feature extraction: motion
  • Feature extraction: shape (morphology): blob, edge, shape
  • Learning (Hidden Markov Models), strengths and limitations
Lab: 
  • Present Etude 2: Gesturally modulate video
  • Extend Etude 2's using cv.jit etc.
  • Compare tracking by colour, luminosity, total motion
  • Compare tracking by recognizing shape
  • Define Etude 3: Modulate one video | sound  stream by features from a second video | sound stream
  • Example: convolution ripple, particle ripple (Julian Stein)



Week 8: November 4
Theory
  • Audio-vision
  • Visual-music
  • Transmodality 

Introduction to MSP
  • DAC~ ADC~
  • DSP Status: DSP, CPU, GPU and scheduling optimization
  • Gain and Monitoring
  • Synthesis: ie. Cycle~
  • Play~ Buffer~ Sfplay~ Groove~
  • real-time Audio Processing
  • Granular Synthesis
  • Mapping and modulation
Class exercise: Scrubbing sound files
Class exercise: Feedback machine


Week 9: November 11

Class: 
  • Present Final Project: Groups should be ready to pitch their project concept including their strategies for implementation.

Lab: One of the following: Physics or Graphics
Special Topic: Physics
  • Newtonian dynamics F = mA, Gravity, Conservation Law
  • jit.phys
Special Topic: Open GL
  • Introduction to 3d objects
  • Creating simple forms
  • The render object
  • Importing more complex objects
  • Drawing Points and Lines
  • Mapping video to a plane
  • The slab object
  • Compositing in OpenGL


Lab: 
  • Creating a sound ‘instrument’ based on video
  • Work on Etude 3: Modulate one stream by features from a second stream


Week 10: November 18

Lab:
Special Topic: 
Special Topic: Optimization
  • Scheduling 
  • Hardware: CPU, GPU, DSP
  • Low-level language: Open GL
  • Extend via external: compiling C via XCode
  • Extend via Gen
Work Session:
  • Work on Etude 3


Week 11: November 25
  • Present Etude 3
Lab: 
  • Optimization
  • Open GL
  • Physics


In-class work period. Each group must meet with the instructors to discuss their final project.
Bench test.  Debug.

Tutorials and Recipes:


Week 12: December 2

Lab: In-class work period. Each group must meet with the instructors to discuss their final project.
Rehearse on site.  Debug.  Optimize.

Week 13:  December 9

Final project presentations and critique