Our senses never worked in isolation. The tools we build for them mostly do.
If there are fundamental connections between light and sound, then an effect
that alters one should have a natural analog in the other. Six pairings
follow, each one animated so you can watch the claim rather than read it.
None of this is an attempt at a strict definition. We form associations
constantly — some learned, some perceived, some built deliberately out of a
theory — and from the inside they all feel the same. A sound simply is that
shape, and the certainty arrives before any reason for it does. Humans do
this, animals do it, and it is not a flaw in the equipment. It is most of
what makes the equipment interesting.
The six below are the ones that held. Each survived contact with other people
well enough to build on, and the mixer at the foot of this page is made from
two of them.
Two men, one table, 1911 is the other end of the same
question — what an association looks like when it does not generalise.
Scriabin and Rimsky-Korsakov on the colour of a musical key: twelve keys, two
agreements. Nobody could ship that, and the mixer here pointedly does not try
to. Colour marks which channel you are dragging and claims nothing else.
These come out of work I did between 2007 and 2012 — an MFA thesis on
cross-modal perception, then a company built on it. They were on a page the
old site served, which is now archived. The code below is that code, moved
rather than rewritten.
How to read it
These are arguments, not measurements. Nothing here is a yield or a
published figure. Each pairing claims that an audio effect and a visual effect
share a principle, and you can disagree with any of them.
They are not all the same kind of claim. Pan and horizontal position
is close to literal. Reverb and a soft-box holds in physics. Compression and
density is a metaphor that earns its place by being useful. The page says
which is which rather than flattening them.
Everything moves. Each pairing is a canvas, not a picture, because
the argument is about change over time and a still frame cannot make it.
Pan / LocalizationPosition in stereo space corresponds directly to visual position along the horizontal axis — the same way we locate objects in physical space using both eyes and both ears.Volume / HeightHeight on the Y axis represents volume. “Up in the mix” is already idiomatic — a balloon rising to its peak in the visual world.Reverb / Light diffusionReverberation — many scattered reflections diffusing in space — is identical in principle to how a soft-box diffuses light: the same source, scattered in many directions, softening harsh edges.Compression / DensityCompression raises low signals and lowers high ones — squeezing dynamic range. Visually, a compressed object is denser: the same mass, visibly more compact. A squeezed beach ball contains the same air.Phase / Light interferenceWhen two identical frequencies arrive simultaneously, they cancel out (phase cancellation). When light passes through two pinholes and projects onto a screen, the same pulsation appears — coherence.Delay / Visual afterimageAudio delay is a repetition of sound at diminished volume. The Pulfrich effect produces the visual equivalent: a dimmed eye perceives a pendulum as lagging behind its true position — an afterimage in time.
The mixer they were built for
Each sound is a shape in physical space, and each shape is a sound.
Horizontal position is stereo pan. Vertical position is volume. Physics —
gravity, spring repulsion, collision — governs movement, so a mix is
something you arrange rather than something you dial in.
This is a working prototype of that interaction, not a video of one. Drag the
shapes. Turn the sound on if you can.
Drag · left and right is pan · up and down is volume · double-click to mute
What came of it
Shapemix raised $2M, shipped four iOS apps including a remixing contest
co-branded with SPIN, and earned a patent. More than any of that, it proved
the proposition: cross-modal interaction is not a novelty, it is a more
natural way to work with sound.
What this cannot tell you
These pairings are arguments, not measurements.
Each one claims an audio effect and a visual effect share a principle. Pan and horizontal position is close to literal; reverb and a soft-box is an analogy that holds in physics; compression and density is a metaphor that works because it is useful, not because it is measured. They are not all the same kind of claim and the page should not pretend they are.
The Pulfrich effect is real; the delay analogy is a reading of it.
A dimmed eye does lag. Whether that is the same thing as an audio delay tap is an argument.