Email to Steffen Werner, October 10, 2006
Is a sense of time—as opposed to a sense of duration*—primarily generated from self-motion of the viewer through an environment rather than motion that takes place apart from the viewer?
*Need working definition of “sense of time” vs. “sense of duration”:
Duration = how long something takes?
Time = sequential ordering of events?
To test this idea we need to be able to separate self-motion from motion that takes place within the viewer’s environment. Following your suggestion, one way would be to introduce a temporal ambiguity between the two and then measure which type of movement was more potent in eliciting a sense of “before” or “after” from the viewer.
Based on my understanding of the literature summarized below, this seems feasible. From those studies I notice at least 3 main variables that could be manipulated but there may be other, more nuanced variables, that I have overlooked. I would appreciate your insight on this.
1. Temporal trajectories: Faster vs. slower
2. Rigidity: Perspective transformations vs. non-perspective cues
3. Motion type: Cues from motion within the environment vs. self-motion
Johansson (1986): Optic flow and vector analysis.
Giachritsis & Harris (2000) — one of many studies indicating that:
Schrater, Knill, & Simoncelli (2001)
Bertin & Israël (2005)
Domini, Caudek, & Skirko (2003)
Entire abstract: (Could use your help with this one.)
Future movement extrapolation (caused by 100 ms latency between proximal stimulus and percept) can explain various illusions, namely that observers most familiar with a “carpentered world” routinely:
Part of an email from Jayme to Steffen Werner, Oct 7, 2006
Changizi’s article started me thinking about Hubbard’s stimuli in his original RM in depth study where he used “approaching” and “receding” squares. When I went back to the original study I noticed that there was something wrong with stimuli measurements. I don’t know why but it kept nagging at me so I came home from work yesterday and started plotting out Hubbard's stimuli—and there is definitely something wonky between his probes and his inducing stimuli. Of course his sloppiness isn't what's interesting but rather that he seems to have inadvertently created two different temporal trajectories with the same, or nearly the same stimuli, so that there are different temporal depths.
And I remembered what you suggested in our meeting that we could set up an experiment with incongruities. I thought that was a great idea but I couldn't think of what to manipulate (and after 2 or 3 days of thinking I only came up with a couple of invertible word pairs). I began to wonder if there wasn't a way to exploit different temporal depths, which led me to start playing around with time/space plots (see below). If you keep time constant, at regularly spaced intervals, but vary the rate at which the object grows—imagine these are Hubbard's approaching squares seen from the side —then you get these different temporal depths. I used a rate of 1.10, 1.15, and 1.25 in these graphs—Hubbard used 1.5.
If you do the opposite and hold space constant, you get exactly the same depth.
Then I decided to invert the graphs and you get this weird visual similarity between space/time inverted and time/space inverted.
And you're probably wondering what on earth this has to do with our experiment and the answer is I'm not sure but I keep wondering if there isn't something useful in these relationships that we could exploit for incongruities.
Email to Steffen Werner, October 1, 2006
Here are some of my thoughts and search results based on the conversation we had a few weeks ago about a research project investigating the link between linear perspective and time perception:
As I indicated earlier, there are references in art theory about certain uses of one-point perspective to denote time; that the placement of the vanishing point can have temporal significance; and that different depths in a picture can be used to represent sequential events.
But I want to be very careful here not to suggest that this is some sort of widely held theory, that art historians universally acknowledge or even give much thought to the relationship of the vanishing point to temporality. However, it does show up and I think that if we go ahead with some sort of study that I can draw on a respectable number of references to make the argument. I was relieved to see it mentioned by Arnheim (see below) but even he never draws the link to optic flow (of course it was only 1954) though he does use the word “pointed flow.”
“Finally, it should be observed that central perspective locates infinity in a specific direction. This makes space appear as a pointed flow, entering the picture from the near sides and converging toward a mouth at the distance. The result is a transformation of the simultaneity of space into a happening in time—that is, an irreversible sequence of events. The traditional world of being is redefined as a process of happening. In this way central perspective foreshadows and initiates a fundamental development in the Western conception of nature.” (Art &Visual Perception, pp. 287-288)
The relationship of the vanishing point to the focus of expansion/(contraction)
Unlike perspective as a metaphor for time, there is such an abundance of sources about the VP and the FOE that it’s hard to narrow down the choicest citations. There are many studies in machine vision, AI, robotics, virtual reality, etc., that are concerned with “extraction of the vanishing point” in order to calculate the focus of expansion. There are endless proposals for new algorithms, improvements to the Hough Transform, and modifications to Gaussian sphere calculations. I haven’t gone very deeply into this but in terms of making the case that the VP is related to the FOE, I think we’d have the argument fairly nailed.
I found one possibly interesting paper that was being used in a computer vision class at Brown. It’s by Antonio Criminisi at Microsoft and he concentrates on algorithms and methods for extracting the VP, mostly for 3D models. But he’s done a lot of work with Renaissance art that’s interesting and maybe it would fit in here. Maybe not.
The main point is that VP = FOE shouldn’t be hard to make.
The relationship between optic flow (self-motion) and time perception
The most compelling reason for taking an interest in this topic comes from animal studies of hippocampal place cells.
At the end of my U of O trip I had a chance to talk to a couple of the other candidates at the airport because the Seattle flight was delayed. One of them had been working with Rizzo who I knew had done a lot of motion processing research with Nawrot. We got into a lively discussion about motion processing and I don’t know how but I started to talk about this idea of perspective use in art and optic flow as possibly integral to time perception. The other guy who was there got kind of serious; he was interested in the idea because his research was with rats and hippocampal place cells and it had been found that animals in linear environments could temporally code their positions using theta oscillations. He stressed that the phenomenon only showed up in linear environments.
I made a note of this on my checkbook cover.
I checked into this after my conversation with you and it seems that over the years there have been all sorts of speculation and research about the role of theta waves. But in 1993 there was a landmark study demonstrating that the firing spikes of place cells shift systematically in relationship to theta oscillations. The relationship between the timing of the spikes and the theta phase was called “phase precession” and it was hypothesized that temporal and sequential information was being encoded by theta oscillations within the spatial framework of the place cells. Again, the phenomenon is most robust in linear environments.
OK, so now you’re thinking, “1993?” But the research has continued in that direction and there’s a very nice review paper by Buzsáki from 2005 that outlines the current thinking on the differences between spatial mapping and the temporal encoding of events.
But the jackpot came when I discovered that the September 2006 issue of Hippocampus was devoted entirely to place fields and theta waves as the basis for episodic memory. Episodic memory? It’s is a pretty big stretch for researchers working with animals. After all episodic memory is only a hypothesis and it requires “autonoetic consciousness” to perform “mental time travel.” But the authors in the special edition of Hippocampus are arguing that episodic memory is context dependent and that spatial context must be considered as at least a subunit of general context—if not the other way around. They are even making a case for spatial mapping—where the animal creates a map by repeated spatial overlappings during exploration—as being similar to semantic memory because there is memory for the environment without any strong temporal or sequential information.
I keep thinking when I read these articles that for animal researchers to go out on a limb like this they must be fairly confident that they’re onto something. Of course, I’m already predisposed but it just seems like it ought to be fertile ground for some cognitive studies.
Related psychophysical and cognitive studies
I didn’t find a lot of relevant information. Most of the studies were on optic flow and timing. There were a couple of interesting studies on linear perspective and vergence (in art). And there is the one that Jennifer Freyd wrote me about: static pictures depicting motion excite MT+ in the same way that animated pictures do but there is a 100 msec lag for the static images suggesting that there is a feedback loop to MT+ once the scene has been recognized—it might be relevant, I don’t know.
Possible experimental designs
My first idea was to do something Freyd-like with before/after comparison pictures but I couldn’t think of anything and I’m a little gun shy about the method after the thesis.
My second idea was to use some sort of visual search task that would be prompted by temporally situated questions. Viewers would see a scene—not as complex as Where’s Waldo, just sufficiently complex—after they had been asked something like, “Will the boy with the glasses pass the woman carrying the briefcase?” or, “Did the girl with the baseball mitt walk by the fire hydrant?” We’d look to see if there were any temporally dependent search patterns like viewers scanning the foreground first and moving backwards or visa versa and if those patterns changed when there wasn’t much perspective depth to the picture. Of course this would require some sort of eye tracking system and I couldn’t remember if the lab had one…
But then I thought, “Why not just go back to the original question?” Do certain kinds of images evoke more of a sense of time than others? Are those images the ones in which optic flow might be inferred by the placement of the vanishing point? Are there other kinds of compositions that are likely to evoke the possibility of optic flow and thus “time”?
It seems like it would be possible to put together a set of images that participants could rate on say a 5 point scale as to how much they conveyed a sense of time. The images could be a variety of drawings, paintings and photographs. I know that doesn’t sound very exciting but it would be a nice starting point and I can think of a few “layers” of experimental questions so that we could run 2 or 3 different experiments and hopefully generate a nice paper that would be the basis for later experiments.
I just want to say that I’m not wedded to any of these ideas and was even trying to keep things fairly sketchy so that I didn’t latch onto some sort of experiment and then have a hard time giving it up. I would, however, like to do something with this general topic and am open to any ideas you might have about the best way to start.
I’ll bring in all of the articles I have for tomorrow’s meeting and we can go from there. Results of the lit search follow though the list might not be entirely up-to-date.
See you tomorrow-
Arnheim, R. (1971). Art and Visual Perception. Berkeley: University of California Press.
Bertin, R. J. V., & Israël, I. (2004). Optic-flow-based perception of two-dimensional trajectories and the effects of a single landmark. Perception, 34, 453-475.
Burr, D., & Santoro, L. (2001). Temporal integration of optic flow, measured by contrast and coherence thresholds. Vision Research, 41, 1891-1899.
Buzsáki, G. (2005). Theta rhythm of navigation: link between path integration and landmark navigation, episodic and semantic memory. Hippocampus, 15, 827-840.
Cantoni, V., Lombardi, L., Ports, M., & Sicard, N. (2001) Vanishing point detection: representation analysis and new approaches. 11th International Conference on Image Analysis and Processing
Criminsi, A. (2002). Single-view metrology: algorithms and applications. Invited paper, Microsoft research.
De Hoz, L., & Wood, E. R. (2006). Dissociating the past from present in the activity of place cells. Hippocampus, 16, 704-715.
Domini, F., Caudek, C., & Skirko, P. (2003). Temporal integration of motion and stereo cues to depth. Perception & Psychophysics, 65, 48-57.
Entright, J. T. (1987). Art and the oculomotor system: perspective illustrations evoke vergence changes. Perception, 16, 731-746.
Entright, J. T. (1987). Perspective vrgence; oculomotor responses to line drawings. Vision Research, 27, 1513-1526.
Giachritsis, C. D., & Harris, M. G. (2005). Global versus local image expansion in estimating time-to-contact from complex optic flow. Perception, 34, 577-585.
Johannson, G. (1986). Relational invariance and visual space perception: on perceptual vector analysis of the optic flow. Acta Psychologia, 63, 89-101.
Kalkofen, H. (2003). Irreconcilable Views. In H. Hecht, R Schwartz, & M. Atherton (eds.) Looking into Pictures: An Interdisciplinary Approach to Pictorial Space (pp. 355-378). Cambridge, MA: MIT Press.
Lorteije, J. A. M., Kenemans, J. L., Jellema, T., van der Lubbe, R. H. J., de Heer, F., & van Wezel, R. J. A. (2006). Delayed response to animate implied motion in human motion processing areas. Journal of Cognitive Neuroscience, 18, 158-68.
Mausfeld, R. (2003). Conjoint representation and the mental capacity for multiple simultaneous perspectives. In H. Hecht, R Schwartz, & M. Atherton (eds.) Looking into Pictures: An Interdisciplinary Approach to Pictorial Space (pp. 17-60). Cambridge, MA: MIT Press.
Mizumori, S. J. Y. (2006). Hippocampal place fields; a neural code for episodic memory? Hippocampus, 16, 685-690.
O’Keefe, J., & Recce, M. (1993). Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus, 3, 317-330.
Schrater, P. R., Knill, D. C., & Simoncelli, E. P. (2001). Perceiving visual expansion without optic flow. Nature, 410, 816-819.
Tulving, E. (2002). Episodic memory: from mind to brain. Annual Review of Psychology, 53, 1-25.
Willats., J. (1997). Art and Representation: New Principals in the Analysis of Pictures. Princeton. NJ: Princeton University Press.
In all of my ePortfolio work I talk about using the blog as an engine for collecting evidence and tagging that evidence so that during the selection process it is easy to filter. But I never follow my own advice. So as of today (1/15/13) I am going to begin dumping things in here and see if I can train myself to get in the habit of posting something at least once a week.